Imagine you're polling a neighborhood about a new park. You knock on one door and ask the same household five times. Each family member's opinion is correlated — they share a kitchen table, a property line, a worldview. You can learn a lot about THAT household, but you can't claim you've polled the neighborhood. The standard self-consistency certification framework for LLMs has exactly this problem: it asks the same prompt many times, treats each rollout as independent evidence, and declares confidence. This paper proves the rollouts aren't independent, and that changes which question you're answering. The committed claim: when LLM reasoning rollouts are positively correlated (through shared prefixes or prompt-level heterogeneity), the standard betting e-process — the sequential test that stops early when it's confident — is valid only for a 'realized null' (did THIS draw of a prompt-belief produce a correct majority?) and NOT for the 'population null' (does the model generally get this right across prompts?). These two nulls coincide only when correlation ρ = 0. In practice, ρ is nowhere near zero. The mechanism is clean and elegant. The authors model dependence via an intraclass-correlation structure — a random-effects model where each prompt carries a latent state that induces equi-correlated rollout increments. This is textbook mixed-effects modeling applied to a novel target. The key result: the effective sample size of within-prompt rollouts saturates at 1/ρ. If ρ = 0.45 (as measured on GSM8K with Claude Haiku 4.5), you get roughly 2.2 effective samples no matter how many rollouts you generate. Extra rollouts are wasted compute that cannot recover population validity. The empirical confirmation is well-chosen. On GSM8K (ρ̂ = 0.435, θ̂ = +0.96), the model is so accurate that the realized-vs-population distinction is dormant — both tests agree because the answer is overwhelmingly correct. On AIME (ρ̂ = 0.480, θ̂ = −0.15), where the model is near chance, the gap explodes: a single-unit e-process crosses the confidence threshold on 24% of prompts at a nominal α = 0.05, while the prompt-level procedure correctly declines at 0.00%. This is the kind of controlled contrast that makes a theoretical result land. The fix is structurally simple but practically expensive: use a prompt-level e-process, allocating one rollout per prompt across many prompts, rather than many rollouts per prompt. Under a pure rollout budget, this is provably optimal. The tradeoff is obvious — you need diverse prompts, not repeated sampling, which changes the economics of certification pipelines. The integrity profile is mixed. The theoretical results are mathematical derivations from the random-effects model, not empirical claims, so they stand or fall on the model's adequacy. The simulation + real-rollout validation on two benchmarks (GSM8K, AIME) is well-executed but narrow — two benchmarks, one model (Claude Haiku 4.5). The paper does not test whether the intraclass-correlation structure actually fits across model families, benchmark difficulties, or chain-of-thought strategies. No code release is mentioned. This paper matters because self-consistency is quietly becoming load-bearing infrastructure. As LLM certification moves from academic benchmarking to deployment gating — deciding whether a model is safe to ship — the distinction between 'this prompt worked' and 'this model works' is not academic. If your certification pipeline is running 100 rollouts on 10 prompts instead of 10 rollouts on 100 prompts, you may be systematically overclaiming reliability. The effective-sample-size saturation at 1/ρ is the kind of result that should change how people allocate compute budgets for evaluation.
The analogy
You're polling a neighborhood about a new park by knocking on one door and asking the same household five times. Each answer is correlated — shared kitchen table, shared worldview. You learn about THAT household with high confidence, but you haven't polled the neighborhood. Self-consistency certification with repeated rollouts on one prompt has the same structural flaw: correlated evidence that answers a narrower question than you think.
Household poll
Asking the same family 5 times yields correlated responses that characterize one household, not the neighborhood
Prompt rollouts
Sampling the same prompt N times yields correlated reasoning that certifies one prompt-belief, not the model's general reliability
Honest assessment
A clean theoretical contribution that identifies a real and practically consequential gap in how LLM self-consistency certification works. The math is solid, the empirical illustration is well-chosen but narrow (one model, two benchmarks). This matters most to teams building certification pipelines for deployment gating — if you're allocating rollout budgets, the 1/ρ effective-sample-size cap should immediately change your spreadsheet. It's a corrective paper, not a methods breakthrough, so don't expect a new algorithm — expect a reason to redesign your existing one.
What Is CLAIMS?
73
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
4/5
Integrity
3/5
Milestone
4/5
Successor
4/5
What is this paper actually committing to?
Genuine clarification: the standard self-consistency e-process certifies a different hypothesis than practitioners assume when rollouts are correlated. Not a new method — a corrective that reframes what existing methods actually prove.
The field fight
Can you certify LLM reliability by repeatedly sampling the same prompt, or must certification be prompt-diverse?
Side A
Repeated rollouts on a prompt are independent enough — the e-process framework gives valid finite-sample guarantees
Held by: Standard self-consistency certification, majority-vote confidence, prior e-process constructions
Side B
Rollouts are correlated through shared prefixes/prompt-state, so within-prompt repetition certifies a narrower claim than assumed; prompt diversity is mandatory
Held by: This paper
Milestone tracker
Unlocks: Prescriptive guidance for certification pipeline design: how many prompts vs rollouts for a given reliability claim
Current
2
benchmarks × 1 model
this paper
Next
10+
benchmarks × 5+ models
ρ universality map
~6-12 months of API-budget engineering
Verified Facts (8)
GSM8K: ρ̂ = 0.435, θ̂ = +0.96
AIME: ρ̂ = 0.480, θ̂ = −0.15
Single-unit e-process crosses on 24% of AIME prompts at nominal ε = 0.05
Prompt-level procedure declines at 0.00% on AIME
Effective sample size of within-unit mean saturates at 1/ρ
Optimal allocation under pure rollout budget: one rollout per prompt
Model used: Claude Haiku 4.5
Asymptotic crossing probability at least π = P(μ(Z) > 0)
Practical Implications (4)
If you run self-consistency certification, switch from many-rollouts-per-prompt to one-rollout-per-many-prompts — the effective sample size saturates at 1/ρ ≈ 2 within a single prompt
Measure intraclass correlation ρ on your specific model and task before trusting any e-process certificate as a population reliability claim
Certification pipelines that gate deployment on e-process crossing should budget for prompt diversity, not rollout depth — fundamentally different compute allocation
The 24% false-crossing rate at nominal 5% on AIME-difficulty tasks means current self-consistency certificates may be systematically overclaiming reliability on hard problems
More latest news
Read all