The Quality of Your LLM Evaluator Depends on What You Let It See — Not How Smart It Is

Three evaluation frameworks, three paradigms, one clear finding: trace-based judging beats text-only judging not because of a better model, but because it gets better inputs. Enterprise agents finally get a testing methodology.

Imagine you're grading a student's math exam. One proctor sees only the final answer written on the page. Another proctor sees the student's entire scratch work — every crossed-out attempt, every intermediate calculation. Same exam, same grading rubric, but the second proctor catches errors the first one misses and — crucially — doesn't flag correct-but-messy work as wrong. That's the core mechanism of this paper: evaluation quality is determined by what information you feed the evaluator, not by how clever the evaluator is. The committed claim: for enterprise analytics agents running multi-tool workflows, a trace-based LLM judge (one that sees the full execution trace of tool calls, intermediate outputs, and agent reasoning) achieves F1=0.90 on known-risk detection, while a text-only LLM judge matches on recall but generates five times more false alarms. Meanwhile, deterministic heuristics — hand-coded rules — catch the obvious stuff but miss subtle semantic failures entirely. Three paradigms, tested head-to-head, on the same frozen execution traces. The study runs in three phases, each answering a different question. Phase 1 asks: are these evaluation harnesses even stable? If you run the same evaluator on the same trace ten times, do you get the same score? The answer is yes — 97% of configurations meet their stability threshold, and stability patterns don't depend on which agent you're evaluating. This is table-stakes work, but it's rarely done, and the fact that they did it first gives the subsequent phases credibility. Phase 2 is the precision-recall shootout across 72 ground-truth test cases. The trace-based framework wins on balanced accuracy; the text-only framework catches failures at a comparable rate but cries wolf far too often. Phase 3 is the most interesting: unknown risk discovery. They turned all three frameworks loose on traces without pre-labeled ground truth and found 24 novel failures — with zero overlap between the functional testing failures and the security red-teaming failures. Architecturally, this is evaluation-framework benchmarking, not model development. The three frameworks represent distinct paradigms: trace-based LLM judging (the evaluator sees structured execution traces), text-only LLM judging augmented with adversarial red-teaming prompts, and deterministic heuristic rules. The agents under evaluation are analytics agents — tools that take natural language queries, decompose them into SQL or visualization steps, and return answers. The experimental design is frozen-trace: all agents ran once, traces were captured, and evaluators scored the same static artifacts. This eliminates a major confounder (agent non-determinism) and makes the comparison clean. The integrity setup is solid for an industry paper. Frozen traces are the right choice — they turn a noisy evaluation problem into a deterministic one. The 72 ground-truth test cases are hand-labeled, and the stability protocol (repeated measurements) catches the evaluator-variance problem that plagues LLM-as-judge work. The weakness is scale: 72 test cases and 24 discovered failures are enough to demonstrate the methodology but not enough to claim statistical robustness across enterprise domains. The agents are internal prototypes, not production systems, and the evaluation frameworks are described in terms of paradigms rather than named, reproducible tools. The zero-overlap finding between functional and security evaluation is the result with legs. It's not that one framework is better — it's that they see different failure surfaces entirely. Functional testing catches wrong answers, stale data references, hallucinated columns. Security red-teaming catches prompt injection, data exfiltration, privilege escalation. No single framework caught both. This is the paper's real contribution to the field: a methodological argument that you need at minimum two evaluation frameworks with different information access and different adversarial postures. The obvious next experiment is scaling this to production traces with real user queries and real failure distributions, rather than controlled research traces from internal prototypes. The authors almost certainly didn't run this because enterprise compliance and data governance make it hard to use real customer interactions as evaluation data — and because the controlled setting was the right first step for a methodology paper. But until someone runs this on messy, real-world traces at scale, the 5x false-alarm ratio and the zero-overlap claim remain findings about a research sandbox, not about production reality.