
Read the paper
Openhagen Worlds: Calibrated Environments for Measuring and Training Scientific Discovery Agents
on Openhagen (hosted eval)
Imagine you're a driving instructor and you need to know if your student can actually drive — not just recite the manual. You could hand them a written test, but that measures recall. You could put them in a simulator, but if the simulator always starts them on the same road, they memorize the route. What you really want is a procedurally generated city where the roads change every time, with a clear pass/fail rubric, and a difficulty dial calibrated so that a good driver passes and a lucky guesser doesn't. That's what Openhagen Worlds builds for scientific discovery agents. The committed claim: a benchmark suite of 27 environment families (178 difficulty levels) where every level is admission-gated by a triple calibration — a scripted oracle must succeed ≥75% of the time, the strongest plausible non-deductive policy must fail ≥75% of the time, and known shortcut strategies must fail ≥65% of the time. This is not a dataset of questions. It's an interactive protocol where agents run experiments under a budget and commit answers checked by a deterministic verifier against hidden ground truth. The architecture choice is deliberate and consequential. Rather than the now-standard static QA benchmark (GPQA, MMLU, ARC), Openhagen treats evaluation as a sequential decision problem: the agent observes, acts (runs an experiment), observes the result, and iterates under a fixed budget before committing. The environments span classic experimental-reasoning puzzles through simulator-grounded physics and chemistry, with 92 core levels for training small models and 86 frontier rungs that extend into multi-stage campaigns of 3, 5, and 8+ committed stages on a single hidden system. The multi-stage campaigns are the sharpest innovation — they force sustained hypothesis revision, not one-shot pattern matching. The ladder numbers are revealing. DeepSeek V4.1 Flash at default reasoning effort (16,384 tokens per call) solves 32 of 84 withheld frontier instances. At maximum effort (393,216 tokens per call — a 24× token budget increase), it climbs to 57 of 84. The oracle ceiling is 79. The largest gap sits in eight-stage campaigns, which is exactly where you'd expect recall and single-shot reasoning to break down. This tells us something important: current frontier LLMs are mediocre scientific reasoners even when given enormous compute budgets, and the difficulty gradient is calibrated tightly enough to expose that. The training signal result is modest but honest. Expert iteration on a 9B model lifts core-tier success on trained families from 0.090 to 0.148 — a 64% relative improvement that is still far below the oracle. Crucially, the authors check for GPQA Diamond regression and find none, meaning the training didn't just overfit to the benchmark format at the cost of general reasoning. The gain appears partly attributable to better action formatting rather than deeper scientific reasoning, which the authors acknowledge directly. Integrity is the strongest dimension here. The triple calibration gate (oracle ≥0.75, non-deductive ≤0.25, shortcut ≤0.35) is applied per-level on 64 development instances before admission — this is essentially a pre-registration of difficulty. The verifier is deterministic. The scoring rules are stated in every observation. Withheld test instances are served through a hosted evaluation service, preventing overfitting to specific instances. This is markedly better integrity infrastructure than most AI benchmarks ship with. The obvious next experiment is scaling expert iteration to a 70B+ model and running it on the frontier multi-stage campaigns. The authors ran 9B only, and the gains were modest. The honest read: this is compute-limited. Training RL-style on interactive environments is expensive, and a solo-author project almost certainly ran out of budget. The gap between 9B and frontier-scale models on this benchmark is the single most interesting unexplored question the paper leaves open.
The analogy
Like a driving test where the roads are procedurally generated each time, the pass/fail rubric is posted on the dashboard, and the test is calibrated so good drivers pass and lucky guessers reliably fail — Openhagen builds the same thing for AI scientific reasoning.
Driving test
Procedurally varied road layouts with posted rubric, calibrated so skill passes and luck fails
Openhagen Worlds
Procedurally generated experiment environments with stated scoring rules, triple-calibrated so reasoning passes and shortcuts fail
Honest assessment
Openhagen Worlds is a benchmark contribution, not a methods breakthrough — and it knows this. The calibration infrastructure (triple-gated admission, deterministic verifier, hosted evaluation) is genuinely rigorous and sets a new standard for how AI scientific reasoning should be measured. The training results are modest and honest. The main risk is adoption: benchmarks only matter if labs use them, and this is a solo-author project competing for attention against well-resourced alternatives. If you work on AI agents for science, this is the evaluation infrastructure you should be using or arguing against.
What Is CLAIMS?
77
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
4/5
Architecture
3/5
Integrity
5/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine first in calibrated interactive benchmarking for scientific discovery. The triple-gated admission protocol is novel infrastructure, not just another question set. This is a tool contribution, not a methods paper.
The field fight
Can AI scientific reasoning be measured without contamination from memorization, shortcuts, or unstated rules?
Side A
Static QA benchmarks (GPQA, MMLU-Pro) are sufficient proxies for scientific reasoning if questions are hard enough
Held by: GPQA, MMLU, ScienceQA, ARC-AGI
Side B
Interactive environments with calibrated difficulty and hidden truths are necessary to separate reasoning from recall and shortcuts
Held by: Openhagen Worlds, Discovery World, ScienceAgent benchmarks
Milestone tracker
Unlocks: Evidence that LLMs can sustain multi-step hypothesis revision — the core skill for autonomous scientific discovery
Current
57/84
frontier instances
DeepSeek V4.1 Flash max effort
Next
70/84
frontier instances
8-stage campaign proficiency
Depends on lab adoption; 1-2 years if major labs engage
Verified Facts (6)
27 procedurally generated environments with 178 difficulty levels, 92 core levels and 86 frontier rungs
Calibration: scripted oracle succeeds on at least 0.75, strongest plausible non-deductive policy on at most 0.25, each gated shortcut on at most 0.35, on 64 development instances
DeepSeek V4.1 Flash solves 32 of 84 withheld frontier instances at default reasoning effort (16,384 tokens per call) and 57 at maximum effort (up to 393,216 tokens per call); oracle: 79
9B model expert iteration: core-tier success on trained families rises from 0.090 to 0.148 with no detectable loss on GPQA Diamond
Frontier campaigns extend 15 discovery worlds with three, five and eight or more committed stages
Hosted evaluation at https://eval.openhagen.ai
Practical Implications (4)
Teams building scientific discovery agents now have a calibrated benchmark that separates reasoning from memorization — use it before claiming your agent 'does science'
The 92 core levels are sized for small-model training, making this accessible to academic labs without frontier compute
The hosted evaluation service at eval.openhagen.ai provides reproducible comparison without test-set contamination
Multi-stage campaign results expose a specific weakness in current LLMs: sustained hypothesis revision over 8+ steps, which is the actual bottleneck for autonomous scientific discovery
More latest news
Read all