Read the paper
SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents
on GitHub
MODELS REPORT
+95%
CONTAINING TRUTH
-62%
GPT-6 ASTRA ACHIEVES
+94%
FABLE ACHIEVES
+99%
Imagine you're a home cook following a recipe that calls for a meat thermometer, but yours reads 20°F too low. A bad cook serves raw chicken and swears it was 165°F. A good cook notices the steak feels wrong, adjusts, and serves safe food. A great cook runs a boiling-water calibration before trusting the thermometer at all. SciResearch-Eval hands AI agents a broken thermometer — a physics simulator with missing or wrong physics — and watches what happens. The committed claim: this is the first benchmark that simultaneously tests agent-designed experiments, deliberately flawed simulation tools, and proper scoring of calibrated uncertainty intervals against exact ground truth. Prior agent benchmarks hand the agent a task and grade the answer. This one hands the agent a task, a lying instrument, and grades both the answer and the agent's honesty about how sure it is. The scoring regime matters: agents must report 95% confidence intervals, and the benchmark checks whether those intervals actually contain the truth 95% of the time. The results are stark. When the simulator is physically wrong, most models confidently report wrong values — their stated 95% intervals contain the truth only 20–62% of the time. They are systematically overconfident. GPT-6 Astra hits 94% coverage; Claude Fable 5.2 hits 99%. The mechanism for Astra is particularly interesting: rather than explicitly diagnosing the missing physics, it designs gentle experiments (slow heating) that keep the flawed simulator in its accurate regime. It routes around the flaw without naming it. Claude 5.5 Opus only diagnoses the missing physics 35% of the time at default effort, versus 66% for Astra. The effort ablation is the most provocative finding. Claude 5.5 Opus at maximum reasoning effort improves more than tenfold and matches Astra at default — but requires 13–31× more reasoning tokens. The correction is latent in the model; it just needs to be forced to think harder. Older and open-weight models gain nothing from extra effort. This suggests a sharp capability threshold: either the knowledge is in there waiting to be activated, or it isn't, and no amount of chain-of-thought will conjure it. Meanwhile, telling the agent the right strategy (stating what approach to use) improves interval scores 1.6–2.5×, replicated on a pre-registered fresh split — but actual coverage barely moves. The agents can follow instructions to get closer answers but still can't calibrate their uncertainty. Fable 5.2 is the accuracy champion at default effort, with interval scores 1.9× better than Astra's, but it writes 6× as many tokens to get there. At Astra's token budget, Fable falls short. This is a clean cost-performance tradeoff the benchmark surfaces but doesn't resolve — different deployment contexts will weight accuracy versus compute differently. The integrity design is unusually strong for a benchmark paper. The main results use a held-out split of 93 tasks across 9 models totaling 2,511 episodes. The effort ablation and strategy prompting results were pre-registered on a fresh split before execution. The environment, graders, and harness are released. The weakness is that this is still a single-domain benchmark — thermal property measurement via heat conduction — and it's unclear whether the scientific judgment being tested transfers to chemistry, biology, or other experimental domains. The bigger question this paper participates in: can LLMs do science, or just mimic it? The benchmark's answer is nuanced — most models mimic, a few genuinely reason, and the gap between mimicry and reasoning shows up precisely when the tool is lying. That's a diagnostic the field has been missing.
The analogy
You hand someone a meat thermometer that reads 20°F low and ask them to cook a safe steak. A bad cook trusts the number and serves raw meat. A good cook notices something's off and compensates. A great cook runs a calibration check first. SciResearch-Eval is this test for AI agents — a deliberately broken instrument that reveals whether the agent is reading numbers or doing science.
Broken thermometer
A measurement tool with a known-to-the-tester but unknown-to-the-cook systematic bias, revealing whether the user trusts instruments or trusts judgment
Flawed simulator
A physics simulator with missing or wrong physics, revealing whether the agent trusts outputs or reasons about model validity and uncertainty
Honest assessment
SciResearch-Eval is a genuinely novel evaluation instrument that tests something prior benchmarks didn't: whether AI agents can recognize when their tools are lying and report honest uncertainty. The integrity design — pre-registered splits, open harness, 2,511 episodes — is stronger than most benchmark papers. The limitation is real: single-domain scope means we don't know if 'scientific judgment' here is thermal-physics-specific pattern matching or a generalizable capability. The finding that most models are confidently wrong (20–62% coverage on stated 95% intervals) while two frontier models succeed via different mechanisms (Astra routes around flaws, Fable brute-forces accuracy) is the kind of result the agent-evaluation community should build on.
What Is CLAIMS?
70
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
3/5
Integrity
4/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine first in combining agent-designed experiments, adversarial simulator flaws, and calibrated interval scoring. Not just another agent benchmark — it tests epistemic honesty, a dimension prior benchmarks ignored.
The field fight
Do LLM agents actually reason scientifically, or do they execute learned procedures that collapse when tools are unreliable?
Side A
Frontier models are approaching genuine scientific reasoning; scaling and prompting unlock it
Held by: GPT-6 Astra (94% coverage), Claude Fable 5.2 (99% coverage)
Side B
Models are pattern-matching, not reasoning; adversarial conditions expose brittle overconfidence
Held by: Claude 5.5 Opus/Sonnet (20–62% coverage), older/open-weight models
Milestone tracker
Unlocks: Evidence that scientific judgment benchmarking generalizes beyond a single physics problem
Current
1
domain (thermal)
this paper
Next
5
domains
multi-domain generalization
~1-2 years if community adopts the open harness
Verified Facts (9)
93 tasks in held-out split, 9 models, 2,511 episodes
Models report 95% intervals containing truth in 20-62% of cases when simulator has missing physics
GPT-6 Astra achieves 94% coverage; Claude Fable 5.2 achieves 99%
Claude 5.5 Opus diagnoses missing physics in 35% of episodes vs 66% for Astra
Strategy prompting improves interval scores 1.6-2.5-fold, replicated on pre-registered fresh split
Maximum effort Opus improves more than tenfold but requires 13-31× more reasoning tokens
Fable 5.2 interval score 1.9× better than Astra but writes 6× as many tokens
Older and open-weight models gain nothing from extra effort
Coverage barely moves with wider intervals or strategy prompting
Practical Implications (4)
Teams deploying LLM agents for scientific workflows should not trust uncertainty estimates from most current models — stated 95% intervals contain truth only 20-62% of the time when tools have systematic errors
Explicitly stating the experimental strategy in prompts improves accuracy 1.6-2.5× but does NOT fix calibration — better answers, still overconfident
The effort-scaling finding suggests a practical diagnostic: if giving your agent 10-30× more reasoning tokens doesn't improve results, the capability isn't latent — switch models, don't scale compute
Benchmark harness is open-source and extensible — teams building scientific agent evaluations can use this as infrastructure rather than starting from scratch
More latest news
Read all