PALATE MATCHES YOURS
+90%
ONE-TIME COST
$0.17
PER TOPIC
$0.49
Imagine you hire an expensive consultant to taste-test every dish at a restaurant chain. They're good — their palate matches yours 90% of the time. But you're paying per plate, forever. Now imagine instead you ask that consultant to write down their decision rules: 'If the sauce has X and the texture hits Y, it's a pass.' Once written, a line cook can run the checklist at zero marginal cost. That's ExecJudges: instead of calling a frontier LLM to judge every document-query pair individually, you ask the LLM once to write a program — using lexical matching, dependency parsers, and pool statistics — that encodes its judgment logic. Then you run the program. The committed claim: a frontier LLM can compile its own relevance-assessment behavior into readable, executable code that matches the LLM's per-document judgments when compared against trained NIST assessors, across seven TREC collections. The ExecJudges-L variant, which also ingests a small number of human labels, exceeds per-topic classifiers on all seven collections and comes within ~0.05 Krippendorff's alpha of LLM judges given the same labels. The ladder here is well-constructed. The authors compare against three serious baselines: trained NIST human assessors (the gold standard in IR evaluation), Claude Sonnet 3.5 as the LLM judge, and recently proposed per-topic classifiers. On TREC Deep Learning 2019–2021, ExecJudges matches Claude's agreement with NIST assessors. The system-ranking correlation (Kendall's tau 0.87–0.94 against official TREC runs) is the number that matters most for practical use — if your cheap judge ranks systems the same way as the expensive one, the cheap judge wins. The honest gap: ExecJudges-L doesn't always beat the LLM judge; it sometimes trails by about 0.05 alpha. The authors don't hide this. Architecturally, this is deliberately anti-neural. The generated programs use only lexical matching, standard NLP parsers, and distributional statistics from the document pool. No embeddings, no model inference at grading time. The LLM's role is code generation, not runtime inference — a one-shot compilation step. This design choice is the paper's core bet: that the structure of relevance judgment for a given topic is simple enough to be captured in a few hundred lines of deterministic code. It's a strong claim about the nature of relevance itself. Integrity is solid for an IR evaluation paper. TREC collections are the community's standard benchmark — these aren't self-selected datasets. Seven collections provide breadth. Comparison against NIST assessors is as close to ground truth as the field gets. The key limitation: the authors are grading their own system against baselines they selected, with no pre-registration and no independent replication yet. The per-topic cost numbers ($0.17–$0.49) are reported but the full compute accounting for program generation isn't detailed. The milestone that matters: ExecJudges works on TREC-scale pools (thousands of documents per topic). The next unlock is whether this approach transfers to open-domain, continuously updated collections where topic drift and document freshness matter — the conditions of real production search evaluation. The authors tested on static TREC collections; production IR evaluation is a moving target. The obvious experiment not run: applying ExecJudges to a non-English collection or a domain-specific corpus (legal, medical, patent). The programs rely on lexical matching and English-language parsers. Cross-lingual or domain-specific evaluation would stress-test whether the compilation step generalizes or whether the programs are brittle English-IR artifacts. My read: they're saving this for the next paper — it's the natural extension and the grant-renewal pitch.
The analogy
Like hiring an expensive consultant to taste-test every dish, then asking them to write down their decision rules so a line cook can run the checklist at zero marginal cost. The expertise gets compiled into a checklist, not invoked per plate.
Recipe checklist
Expert writes decision rules once; line cook executes them repeatedly at zero marginal cost per dish
ExecJudges
Frontier LLM writes relevance programs once; deterministic code executes them at zero marginal cost per document
Honest assessment
A well-executed paper with a genuinely interesting conceptual reframing: LLMs as compilers of their own judgment, not permanent runtime infrastructure. The TREC evaluation is thorough and the baselines are honest. The load-bearing assumption — that relevance per topic is lexically compressible — is demonstrated but not proven to generalize beyond English TREC collections. If you run IR evaluation at scale, this is worth reading today. If you're watching the LLM-as-judge trend, this is the first serious argument that the judges should be writing themselves out of a job.
What Is CLAIMS?
73
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
4/5
Architecture
4/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine reframing: compiling LLM judgment into deterministic programs is a real conceptual move, not just a speedup trick. But the idea that relevance is compressible isn't proven — it's demonstrated on TREC.
The field fight
Are LLM judges sustainable infrastructure for large-scale relevance evaluation, or an expensive stopgap?
Side A
LLM judges are the best scalable proxy for human judgment; cost will decrease with model commoditization
Held by: Thomas et al. 2024, Faggioli et al. 2023, most LLM-as-judge papers
Side B
LLM judgment logic is extractable into cheaper, inspectable, deterministic code — the LLM is a compiler, not infrastructure
Held by: ExecJudges (this paper)
Milestone tracker
Unlocks: Continuous, near-free relevance evaluation for production search systems without per-query LLM costs
Current
7
TREC collections
static pool evaluation
Next
1
live eval track
production-scale with drift
~1-2 years if adopted by a TREC or CLEF evaluation campaign
Verified Facts (6)
ExecJudges agrees with NIST assessors at the level of the Claude Sonnet 5.5 LLM judges on TREC DL 2019-2021
ExecJudges-L variant agrees with NIST more than per-topic classifiers with the same labels on all seven collections
ExecJudges-L either exceeds the agreement of LLM judges provided the same labels or is within about 0.05 of it in Krippendorff's alpha
Labels rank the official TREC DL runs with Kendall's tau of 0.87 to 0.94
Writing the programs incurs a one-time cost of $0.17 to $0.49 per topic, with no additional costs for grading
Programs rely only on lexical matching, standard NLP libraries like parsers, and pool statistics
Practical Implications (4)
IR teams paying per-query LLM evaluation costs can potentially reduce ongoing costs to near zero after a one-time program generation step
Evaluation becomes reproducible and inspectable — you can read the program and understand why a document was judged relevant
TREC-style evaluation campaigns could adopt ExecJudges to scale assessor pools without scaling assessor budgets
The approach creates an audit trail — unlike LLM judges, the decision logic is visible and debuggable
More latest news
Read all