
STATING
+30%
INTERVALS BETWEEN
+95%
INSPECTOR PASSES
+70%
Imagine you hire two inspectors from the same firm to grade a building. Inspector A passes 70% of the units. Three months later, Inspector B — same firm, same badge, same name on the door — fails nearly every one. No unit got worse; the firm just quietly replaced the inspector. That is what this paper documents happening in LLM-based peer review, and the numbers are not subtle. Jizhou Guo audited twenty frontier models from OpenAI, Google, and Anthropic against ICLR 2027 submissions while those papers were still under review, with tests registered before decisions were announced. The headline result: between an earlier and later version of Claude Opus and of Gemini, the share of papers placed on the accept side drops by 52 and 53 percentage points respectively, across 183 and 60 randomly drawn submissions. In six further paired comparisons the drop ranges from 43 to 79 points. Zero papers moved upward in any comparison. When the prompt states a 30% acceptance rate, the gap between two Opus versions widens to 79 points. This is not noise — it is a regime change wearing the same brand name. The paper's most diagnostic move is a critique-swap experiment. Score-stripped written critiques from the later model version are fed to the earlier version's scoring function (and vice versa). The later critiques alone lower the accept share by 35 points for Opus and 51 for Gemini. Even under a third-party scorer, the Opus drop holds at 35 points. Even after rewriting critiques to preserve every technical point while removing evaluative tone, the drop is still 24 points. The drift lives in the substance of the critique, not just calibration of the score. Calibration compression is universal: all twenty models assign a score of 2 or 3 in 473 of 475 reviews, versus 60 of 72 for human reviewers. The models agree with each other on rank order more than you would expect — Opus 4.6 and 5.5 percentiles correlate at 0.73 — but their thresholds sit in completely different places. Three models' self-estimated percentiles correlate with human reviewers more strongly than a third human reviewer does (point estimates 0.17–0.21, 95% CIs from −0.04 to +0.42), suggesting the ranking signal is real even as the accept/reject boundary is arbitrary. A contamination check adds a sharp edge. Claude Opus 5.5 names the true first author of 27 decided ICLR 2026 papers from titles alone with zero wrong names. Claude Opus 4.6 accepted all 11 decided papers it said it recognized versus 5 expected by chance. Retrospective benchmarking on decided papers — the standard practice in the field — therefore cannot distinguish review quality from memory. The practical upshot is blunt: any venue or benchmark that thresholds on LLM-reviewer scores must validate that threshold against human decisions for each model version separately. A calibration established in March is meaningless in June if the provider ships a new checkpoint. The authors have registered tests against ICLR 2027 decisions due on 16 December 2026, which will close the loop on whether version-specific calibration can be made to work at all. This paper does not propose a fix. It proposes a measurement regime and demonstrates that the problem is far larger than the community has acknowledged. The field fight over whether LLMs can substitute for human reviewers has been conducted mostly on stale benchmarks with decided papers; this is the first audit conducted prospectively on live submissions, with pre-registered tests, and the version-drift signal it surfaces is devastating for anyone treating LLM review scores as stable measurements.
The analogy
You hire two building inspectors from the same firm, same credentials, same name on the badge. Inspector A passes 70% of units. Three months later Inspector B — same firm, silently swapped in — fails nearly all of them. Nothing about the buildings changed; the firm just shipped a new inspector without telling anyone. That is LLM version drift applied to peer review: the measurement instrument changes underneath you, mid-experiment, with no notice.
Building inspector
Same firm replaces inspector silently; pass rates swing wildly with no change in buildings
LLM reviewer
Same model name, new checkpoint; accept rates drop 52-79pp with no change in papers
Honest assessment
This is a measurement paper that documents a devastating problem rather than solving it. The version-drift signal is enormous (52-79pp), the experimental design is unusually disciplined for this subfield (prospective, pre-registered, with clean ablations), and the contamination finding alone should give pause to anyone running retrospective LLM review benchmarks. The main limitation is that the registered tests against actual decisions are pending — the paper is explicitly incomplete. If the December 2026 results confirm the pattern, this becomes a foundational citation for anyone arguing that LLM review scores require per-version calibration. If they don't, the paper is still a strong audit of instability.
What Is CLAIMS?
80
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
5/5
Ladder
4/5
Architecture
3/5
Integrity
4/5
Milestone
4/5
Successor
4/5
What is this paper actually committing to?
Genuine first: prospective, pre-registered audit of LLM review version drift on live submissions. The 52-79pp accept-rate swings are not an incremental finding — they invalidate current practice.
The field fight
Can LLM-generated reviews substitute for human peer review at scale?
Side A
LLMs correlate with human reviewers and can at least triage or supplement; deploy now, calibrate later
Held by: ReviewerGPT, MARG, venues experimenting with LLM review
Side B
LLM scores are not stable measurements; version drift and contamination make current deployment invalid without per-version calibration
Held by: This paper
Milestone tracker
Unlocks: Whether per-version calibration against human decisions produces usable thresholds
Current
52-79
pp drift
version gap documented
Next
16 Dec 2026
date
registered test vs ICLR decisions
~2 months from paper submission
Verified Facts (9)
Between an earlier and a later version of Claude Opus and of Gemini, the share of papers placed on the accept side falls by 52 and 53 percentage points on 183 and 60 randomly drawn submissions
By 43 to 79 points in six further paired comparisons, with no paper moving up in any of them
Stating a 30% acceptance rate in the prompt leaves a gap of 79 points between the two Opus versions
The later critiques alone lower the accept share by 35 points for Opus and 51 for Gemini
All twenty models give a 2 or a 3 in 473 of their 475 reviews, against 60 of 72 human reviews
Claude Opus 5.5 names the true first author of 27 decided ICLR 2026 papers from their titles alone, with no wrong name
Claude Opus 4.6 accepted all 11 decided papers it said it recognized, against 5 expected
The percentiles of Opus 4.6 and 5.5 correlate at 0.73 while their accept shares differ by 44 points
Self-estimated percentiles of three models correlate with two human reviewers more strongly than a third reviewer does (point estimates 0.17 to 0.21, 95% intervals between -0.04 and +0.42)
Practical Implications (4)
Any venue using LLM-generated review scores must validate thresholds against human decisions separately for each model version — calibration does not transfer across updates
Retrospective benchmarks on decided papers are contaminated by model memory; Opus 5.5 can name first authors from titles alone
All 20 tested frontier models compress scores to 2-3 on a wider scale, making threshold-based decisions extremely sensitive to small calibration shifts
Version-pinning is necessary but not sufficient — the critique content itself drifts, not just the scoring calibration
More latest news
Read all