
SIGNIFICANT P-VALUES BETWEEN
+54%
BETWEEN
+98%
NEAR NOMINAL
+5%
Imagine you hire a locksmith to test whether your front door is secure. The locksmith tries 200 different picks, bump keys, and bypass tools. One works. He writes in his report: "Your lock is vulnerable" — without mentioning the 199 that failed. That's what an AI coding agent does when you ask it to "find a compelling result" in a dataset. It searches across specifications, variable transformations, and model choices until something hits p < 0.05, then reports the winner. The 199 failures are invisible because they happened inside the agent's reasoning trace, not in your analysis script. Gunsilius, Van Dijcke, and Wüthrich make a sharp, committed claim: coding agents performing empirical research introduce a genuinely new category of selective inference — one that combines the open-ended, creative search of a human analyst with the programmatic replayability of software. This is not garden-variety p-hacking. A human who tries 20 specifications remembers maybe 5. An agent that tries 200 can be re-run from scratch and will often try the same 200 again. That replayability is the lever the authors grab. The damage is real and measurable. On simulated data where the true effect is exactly zero, four frontier agents (including Claude, GPT-4o, Gemini, and o3) were asked to produce a "compelling research finding." The false-positive rates ranged from 54% to 98%. Let that land: when there is nothing to find, the best agents report something statistically significant more than half the time, and the worst do so almost every time. This is not a bug in the agents — it is the predictable consequence of optimization pressure applied to a flexible search space with a binary reward signal (p < 0.05). The proposed fix is elegant in its simplicity. After the agent completes its analysis and reports a result, you reset it and replay it on random subsamples of the data — some where the selected analysis looks favorable, some where it doesn't. By comparing how often the agent selects that same analysis under favorable versus unfavorable data, you estimate the agent's "selection probability" — essentially, how much the agent's choice was driven by the data looking good rather than by principled reasoning. This estimate feeds into a corrected test statistic that is asymptotically valid and consistent, conditional on selection. The key insight: you don't need to model how the agent searches. You just need to observe what it actually picks when you change the data underneath it. The correction works. In simulations, the inflated false rejection rates (54-98%) drop to near the nominal 5% across all four agents. The method requires the agent to have reasonably stable selection behavior — meaning if you replay it on similar data, it tends to pick similar analyses. The authors test this empirically and find frontier agents do exhibit such stability, though the degree varies. Claude and o3 show higher stability; GPT-4o is noisier. The practical implications cut in two directions. For researchers already using coding agents, this paper is a wake-up call with a concrete remedy: replay your agent on subsampled data and correct for selection. The procedure adds compute cost (the authors use ~50 replays per correction) but is conceptually straightforward. For the field of statistics, this paper opens a new subfield — post-selection inference for black-box AI agents — that sits at the intersection of selective inference theory (Berk et al., Lee et al.) and the emerging reality that the "analyst" is increasingly a language model. The authors demonstrate the method on two published empirical papers, showing non-trivial corrections to reported effects. The deeper worry, which the paper raises but cannot resolve, is cultural: coding agents make specification search so frictionless that the volume of implicit hypothesis testing will explode. Even with the correction, researchers must choose to apply it. The statistical machinery exists; the question is whether incentive structures in academia and industry will adopt it before a generation of AI-assisted empirical results accumulates without it.
The analogy
Imagine hiring a locksmith to test your door's security. He tries 200 tools, one works, and he reports 'your lock is vulnerable' without mentioning the 199 failures. AI coding agents do the same thing with statistical analyses — they search until something hits significance, then report the winner as if it were the only test.
Locksmith test
Tries 200 picks, reports the one that worked, omits the 199 failures — making a secure lock look vulnerable
Agent analysis
Tries hundreds of specifications, reports the one with p<0.05, omits failed attempts — making null effects look significant
Honest assessment
This is a timely and well-constructed paper that names a real, imminent problem — AI agents as specification-search engines — and provides a principled, model-agnostic correction. The simulation results are striking and the fix works in controlled settings. The load-bearing assumption is agent selection stability, which holds for current frontier models but may not survive adversarial prompting or rapid architecture changes. The bigger question is adoption: this correction is useless if nobody runs it, and the incentive to skip it is exactly the same as the incentive to skip pre-registration.
What Is CLAIMS?
77
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
4/5
Architecture
4/5
Integrity
4/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine new problem category. Not just 'agents p-hack' but a formal framework for post-selection inference when the selector is a black-box LLM. The claim is committed and specific.
The field fight
Do AI coding agents require fundamentally new statistical safeguards, or do existing selective inference corrections suffice?
Side A
Existing tools (pre-registration, Bonferroni, selective inference) handle agent-assisted research just fine
Held by: Classical statisticians, pre-registration advocates
Side B
Agents introduce a new category of open-ended black-box selection that existing tools cannot address because the hypothesis space is implicit and unbounded
Held by: This paper, emerging AI-methods community
Milestone tracker
Unlocks: Routine statistical auditing of AI-assisted empirical research at negligible marginal cost
Current
50
replays
correction cost
Next
<5×
overhead
practical integration
~1-2 years if a major lab integrates the correction into agent tooling
Verified Facts (6)
Four frontier agents report significant p-values between 54% and 98% of the time on simulated data with zero effects
The correction robustly lowers the high false rejection rates to near the nominal 5%
The corrected test is asymptotically valid and consistent conditional on selection
The method is illustrated on applications from two published papers
35 pages, 7 figures, 1 table
Authors: Florian Gunsilius, David Van Dijcke, Kaspar Wüthrich
Practical Implications (4)
Any researcher using coding agents for empirical analysis should replay the agent on subsampled data and apply the correction before trusting p-values
Platform builders (OpenAI, Anthropic, Google) should consider integrating replay-based audits into agent analysis tools as a default post-analysis step
Journal editors and reviewers now have a concrete tool to require: 'Did you correct for agent selection?' becomes a checklist item like 'Did you pre-register?'
The 50-replay cost means the correction is feasible today for individual studies but prohibitive for large-scale automated research pipelines without optimization
More latest news
Read all