AI Peer Review Catches 71% of Planted Errors — But Only When You Know What You're Looking For

A controlled benchmark plants 100 known errors into psychology papers and stress-tests frontier LLMs and commercial review tools. Best single system: 71/100. Ensemble of all systems: 93/100. The 7 uncatchable errors were all omissions.

Imagine you're a teacher grading spelling tests, but instead of giving students a word list, you hand them a finished essay and ask them to find the misspellings you secretly introduced. That's what this paper does to AI review systems — except the 'misspellings' are statistical errors, misreported numbers, logical contradictions, and missing information planted into real psychology papers. The core mechanism is a known-answer test: you control the ground truth, so you can measure recall precisely. The committed claim: this is the first controlled, public benchmark for AI peer review, built from 100 deliberately planted errors across 10 open-access psychology papers. The best single system (not named as a frontier model — it's the commercial tool Refine.ink's output combined with frontier model runs) caught 71 errors. The worst caught 30. Pool every system together and you get 93. The seven errors no system caught were all omissions — information deleted from the paper rather than mistakes injected into it. That distinction matters enormously: AI review tools are pattern-matchers looking for things that are wrong, not detectives noticing what's missing. The ladder here is interesting because there's almost no prior art in controlled benchmarks for AI review. Most evaluations of AI peer review have been subjective (do human reviewers agree with AI comments?) or uncontrolled (did AI catch errors that were later found by humans?). This paper creates a ground-truth setup that didn't exist before. The comparison is internal — system against system — rather than against a human-reviewer baseline. No human reviewers were tested on the same 100 errors, which is the glaring absence. Architecturally, this is a prompt-and-observe evaluation study, not an algorithmic contribution. Frontier LLMs (the specific models aren't named in the abstract but the full outputs are public) and two commercial AI review platforms — Refine.ink and at least one other — were given modified papers and asked to review them. The key finding is that model outputs are only partly correlated: different systems catch different errors, which means ensembling is the big lever. Refine.ink contributes the most unique catches (errors only it found), but at higher cost per error. The integrity picture is mixed in a specific way. The planted-error methodology is strong — you know the ground truth, so recall is measurable without ambiguity. But false positives were explicitly not measured, which is a serious gap. A system that flags everything as an error would score 100/100 on recall. The error distribution is also acknowledged as artificial: 100 hand-planted errors in psychology papers may not reflect the distribution of real errors in real submissions. The papers, errors, model outputs, and full experiment log are public, which is the right move and makes replication straightforward. The milestone that matters is extending this to false-positive measurement and cross-disciplinary coverage. Right now we know recall on a specific error distribution in psychology. We don't know precision, we don't know how the error types map to real-world frequencies, and we don't know if results generalize to other fields. The 93% ensemble ceiling and the omission-blindness finding are the two numbers to track — future systems should be benchmarked against both. The obvious next experiment: run human expert reviewers on the same 100 errors and compare. This would establish a human baseline and let you calculate whether AI review is complementary, redundant, or inferior. The honest read on why it wasn't run: this is a 5-page workshop paper by a single author. Recruiting and coordinating expert human reviewers is expensive and slow. It's the natural follow-up, and the public dataset makes it easy for someone else to do.