Imagine you hire three building inspectors to independently check a house. The whole point is that Inspector A catches the bad wiring, Inspector B catches the foundation crack, and Inspector C catches the roof leak. Now imagine all three inspectors start consulting the same AI checklist before writing their reports. Suddenly they all flag the wiring and nobody mentions the roof. You've paid for three inspections and gotten one, photocopied. That's the core mechanism this paper measures in peer review. Ilya Gusev tracks "critique overlap" — the fraction of specific problems identified by one reviewer that another reviewer of the same paper also identifies — across ICLR, NeurIPS, ICML, and two journal platforms (F1000 and PLOS). At ICLR, this overlap rose from 12.0% in 2026 to 21.3% in 2027, comparing reviews from the same stage of the process. A third review added only 5.46 new problems to the first two in 2027, down from 6.12 the year before. The marginal value of each additional reviewer is declining. The causal bridge to LLMs comes via MELD, a detector of LLM-generated text. Within the same paper, two reviews that MELD flags as LLM-generated overlap more than two unflagged reviews: by 3.0 percentage points at ICLR in 2024-2025, rising to 4.5 points in 2026-2027, and 5.4 points across five journal sources. This is the crux: the effect is not just that overlap is growing — it's that LLM-flagged reviews are measurably more similar to each other than human reviews of the same paper. The homogenization has a detectable signature. The architectural choice here is straightforward but well-executed. The paper defines critique overlap at the claim level — individual problems or suggestions — rather than at the document level (e.g., BLEU score between full review texts). This granularity matters because two reviews can use completely different phrasing and still raise the same objection. Matching was done by extracting atomic claims from reviews and computing pairwise overlap. MELD serves as the LLM detection layer, applied to all reviews, creating a natural experiment: flagged vs. unflagged reviews of the same paper, same venue, same year. The integrity picture is mixed but honest. The paper uses real conference data across multiple venues and years, which is strong. The year-on-year trend at ICLR is the cleanest signal (same venue, same review stage, consecutive years). But MELD is not ground truth — it's a classifier with its own false-positive and false-negative rates, and misclassification would attenuate or inflate the gap. The paper acknowledges this. There is no counterfactual: we cannot know what these reviewers would have written without LLM assistance. The correlation between MELD flags and higher overlap is suggestive, not causal proof. The practical implications are sobering. If LLM-assisted reviews converge on the same critiques, then adding more reviewers no longer buys proportionally more coverage. The marginal information gain per reviewer drops. Conference program committees that rely on reviewer count as a proxy for review quality are spending more for less. The 20-year version of this trajectory is a peer review system that is cheaper to produce but functionally monocular — a single synthetic perspective replicated across multiple review slots. What the paper does not do is propose a fix, nor does it model what level of overlap would be catastrophic versus merely concerning. The 12% to 21.3% jump is striking, but we have no established baseline for what "healthy" overlap looks like in peer review. That's the obvious next study: establishing a theoretical or empirical floor for how much overlap is expected when independent, competent humans review the same paper.
The analogy
Three building inspectors who all consult the same AI checklist before writing their reports: you've paid for three inspections but gotten one perspective photocopied three times. The redundancy that made multi-reviewer peer review valuable is exactly what's being eroded.
Building inspectors
Three independent inspectors each catch different defects — wiring, foundation, roof — because they look with different eyes
Peer reviewers
Multiple reviewers of one paper should catch different flaws, but LLM-assisted reviews converge on the same critiques, reducing marginal coverage
Honest assessment
This paper identifies a real and growing problem — LLM-assisted peer reviews are converging on the same critiques — and measures it rigorously across multiple venues. The measurement is solid; the causal link via MELD is suggestive but not airtight. What's missing is a threshold model (how much overlap is too much?) and any intervention proposal. Practitioners running program committees should care today; the broader community should care in 2 years when overlap may cross 30%.
What Is CLAIMS?
63
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
3/5
Integrity
3/5
Milestone
2/5
Successor
4/5
What is this paper actually committing to?
Genuine new measurement: claim-level critique overlap linked to LLM detection, across multiple venues and years. Not the first paper to flag LLM use in reviews, but the first to measure functional homogenization.
The field fight
Is LLM-assisted peer review a net positive (scales review capacity) or a net negative (erodes review diversity)?
Side A
LLMs help under-resourced reviewers produce competent reviews faster; the capacity gain outweighs homogenization risk
Held by: Conference organizers facing reviewer shortages, OpenReview advocates
Side B
LLM-assisted reviews converge on the same critiques, destroying the independent-perspective redundancy that makes multi-reviewer systems work
Held by: This paper, Liang et al. 2024, review-integrity researchers
Milestone tracker
Unlocks: Policy intervention threshold — conferences would need to structurally change review assignment or mandate LLM-free reviews
Current
21.3
% overlap
ICLR 2027
Next
~30
% overlap
Functional review failure?
1-2 years at current trajectory
Verified Facts (4)
At ICLR, measured overlap rose from 12.0% in 2026 to 21.3% in 2027
A third ICLR review of a paper added 5.46 new problems to the other two in 2027, against 6.12 in 2026
Two MELD-flagged reviews overlap more than two unflagged reviews: by 3.0 points at ICLR in 2024-2025, by 4.5 points in 2026-2027, and by 5.4 points across five journal sources
MELD is used as the detector of LLM-generated text
Practical Implications (3)
Conference program committees should stop treating reviewer count as a proxy for coverage diversity — three LLM-assisted reviews may provide less marginal coverage than two human reviews
Venues should consider requiring reviewers to disclose LLM use and may need to structurally diversify review assignments to compensate for convergence
Claim-level overlap could become a quality metric for review pools, flagging papers that received functionally redundant reviews before acceptance decisions
More latest news
Read all