A Standardized Test for AI Peer Reviewers Reveals That Model Choice Matters More Than Fancy Tooling

First rigorous benchmark for LLM-based peer review: inject known errors into real papers, measure who catches what. Model choice swings detection by 50 percentage points; tools help only where models are already weak.

Imagine you're a high-school teacher who suspects students are using the same essay-grading app to check each other's homework. You want to know: does the app actually catch mistakes, or does it just sound confident? So you plant known errors — a wrong date, a botched calculation, a made-up citation — into clean essays and see which graders find them. That's exactly what this paper does to AI peer reviewers, except the essays are real arXiv preprints and the graders are frontier LLMs. The committed claim: we now have a reusable, open benchmark that quantifies AI peer-review reliability by error type (general, math, numeric, citation), severity (three levels), and false-positive rate — and the first system (Skepthical) that retrieves cited papers, runs computer algebra, and executes code to verify claims. Applied to 30 real papers across astrophysics and ML with 357 injected errors, twelve reviewer configurations are ranked head-to-head. The headline finding is brutal in its simplicity: swapping the underlying LLM changes total detection rate by roughly 50 percentage points, dwarfing the effect of prompt engineering or agentic scaffolding. Three frontier models on identical prompts produced wildly different detection curves. If you're deploying AI review at scale and you picked the wrong model, no amount of tool-calling saves you. Tools and multi-agent scaffolding do help — but only where the base model is weak. Claude Code improved over a single Claude call specifically on math and numeric errors in astrophysics, where the model lacks domain-specific calculation ability. Skepthical's enforced citation retrieval crushed every other system on citation-error detection in astrophysics. But on ML papers — where frontier models already know the literature — Skepthical's citation advantage vanished, statistically indistinguishable from the best single-LLM reviewer. The tool provides what the model can't supply on its own, and nothing more. False-positive rates stay below 10% for most systems, but math flags are the primary source of false alarms. This is the integrity bottleneck: an AI reviewer that flags correct math as wrong is worse than one that misses an error, because it erodes trust and wastes author time. The paper is honest about this, disaggregating false positives by error category. The benchmark design is the real contribution. Error injection is localisable (you know exactly where the error is), objective (adjudication logs are released), and covers the four failure modes that matter in peer review. The evaluation pipeline, injected errors, and all adjudication logs are open. This means other groups can run any new model or system through the same gauntlet without re-creating the infrastructure — a genuine public good for the field. What this doesn't tell you is whether AI reviewers catch the errors that actually matter in the wild — subtle conceptual flaws, motivated reasoning, or claims that are technically correct but misleading. Injected errors are by definition known-category; the unknown-unknowns problem remains open. Still, as a first standardized measurement instrument for a capability that's already deployed at production scale, this is the benchmark the field needed and didn't have.