Read the paper
TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs
on Project Site
Imagine you're training a new chef. First test: follow a recipe. Second test: pick the best dish from a menu for a customer. Third test: invent the dish yourself, source the ingredients, and cook it — in someone else's kitchen, using only their equipment. Most AI benchmarks test the first two. TwinLab-Bench tests the third, and the results are sobering. The core claim is straightforward: existing benchmarks for AI scientists evaluate code execution and next-step selection from predefined candidates, but they cannot evaluate open-ended experimental design because there is no reliable way to run an agent-designed experiment and get readings grounded in reality. TwinLab-Bench solves this by building "TwinLab agents" — digital twins of 52 real laboratories spanning physics, chemistry, wet-lab biology, materials science, and engineering. Each twin is constructed from actual lab records, with calculators calibrated to reproduce real experimental readings. When the tested AI agent designs an experiment, the TwinLab agent executes it in the digital twin and returns readings that are validated against ground truth from the original lab records. The benchmark supports over 100 experimental action types across its 52 tasks — this is not a narrow sandbox. The scoring regime compares the agent's final report against a hidden answer drawn from real laboratory records. Critically, the authors run closed-book probes to control for answers the model might have memorized from training data, and they show that the TwinLab agent nearly eliminates two failure modes: readings that deviate from real records, and inconsistent answers to repeated identical requests. This is the infrastructure contribution — not a new model, but a new testing regime that makes open-ended experimental design evaluable. Thirteen agents were evaluated. Claude Fable 5.2 leads at 45.0, Opus 5.5 follows at 40.2, GPT-6.1-sol scores 31.7, with DeepSeek, Qwen, Kimi, and GLM trailing further. The best frontier model fails on more than half the tasks. The authors conduct an empirical failure analysis of Claude Fable 5.2's errors, which is arguably the most useful part of the paper — it surfaces where open-ended scientific reasoning actually breaks down, rather than just reporting a leaderboard number. The architectural choice is agent-as-environment: a TwinLab agent wraps each lab's raw materials and calculators into an interactive environment that accepts natural-language experiment descriptions and returns structured readings. This is a simulation-based evaluation regime, not a physical experiment, but the calibration against real lab records is what gives it teeth. The key compute property is that each TwinLab agent must be built per-lab from that lab's specific records, which limits scalability but maximizes fidelity. The integrity picture is decent but not airtight. Ground truth comes from real lab records (strong), closed-book probes control for memorization (strong), and consistency checks control for stochastic noise (strong). But the benchmark is author-constructed, not community-adopted yet, and there is no independent replication. The 52 tasks were selected by the authors — we don't know what was tried and excluded. Code and tasks are released publicly, which is the right move for a benchmark paper. The milestone that matters: 45.0 is the current ceiling. The number to watch is 75 — roughly the threshold where an AI agent would be reliable enough that a human scientist would trust it to run an overnight experiment without supervision. At current improvement rates, we are likely 2-4 years from that number on this benchmark, but the gap between 45 and 75 may be nonlinear — the easy tasks are already solved, and the hard ones may resist scaling.
The analogy
Think of flight simulators. For decades we tested pilots by asking them to fly pre-programmed routes (code execution) or pick the best heading from a dropdown (next-step selection). TwinLab-Bench is the first sim that says: here's a cockpit and weather conditions — file your own flight plan, fly it, and we'll grade the landing against what a real pilot achieved in this same weather. The sim is built from black-box recordings of real flights, not from physics equations, so its readings are grounded in reality rather than theory.
Flight simulator
Cockpit built from black-box recordings of real flights; pilot files own flight plan and sim grades the landing against the real outcome
TwinLab-Bench
Digital twin built from real lab records; AI agent designs own experiment and twin grades the report against hidden ground-truth from the actual lab
Honest assessment
TwinLab-Bench is an infrastructure contribution, not a model contribution — and it fills a real gap. The 45.0 ceiling for the best frontier model on self-designed experiments is a useful calibration point for anyone betting on AI scientists. The benchmark's credibility rests on its calibration against real lab records and its memorization controls, both of which are well-executed. The main limitation is that it's author-constructed and not yet community-adopted; until other groups build tasks on this framework or replicate the results, the 52-task set is an existence proof, not a standard. Worth watching closely, not yet citing as gospel.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
4/5
Integrity
3/5
Milestone
3/5
Successor
3/5
What is this paper actually committing to?
Genuine first: no prior benchmark evaluates self-designed experiments grounded in real lab records across 52 diverse tasks. This is an evaluation-regime contribution, not a model contribution — and the field needed it.
The field fight
Can AI agent capability be meaningfully evaluated with closed benchmarks (predefined candidates), or does open-ended experimental design require a fundamentally different evaluation regime?
Side A
Closed benchmarks with predefined action spaces are sufficient proxies for scientific capability; real-lab grounding is unnecessary overhead
Held by: ScienceAgentBench, MLAgentBench, code-execution benchmarks
Side B
Open-ended experimental design is a qualitatively different ability that existing benchmarks cannot measure; grounding in real lab data is essential
Held by: TwinLab-Bench, this paper
Milestone tracker
Unlocks: AI agents trusted to design and run experiments overnight without human supervision — a step-change in lab throughput
Current
45.0
score
Claude Fable 5.2
Next
75
score
unsupervised-trust threshold
~2-4 years if frontier models continue improving, but difficulty curve likely steepens past 50
Verified Facts (9)
52 tasks from laboratories in the physical sciences, the life sciences, and engineering
More than 100 experimental action types supported
Claude Fable 5.2 averages a score of only 45.0
Opus 5.5 scores 40.2, GPT-6.1-sol scores 31.7
13 agents evaluated including DeepSeek, Qwen, Kimi, GLM
33 pages, 6 figures, 14 tables
Code and tasks released at twinlab-bench.github.io
TwinLab agent almost eliminates readings that deviate from real records and inconsistent answers to repeated requests
Closed-book probes control for answers recalled from memory
Practical Implications (4)
Lab managers evaluating AI assistants for experimental design should expect ~45% reliability at best on novel tasks — plan for human review of every AI-designed experiment
Benchmark builders now have a reusable pattern: build digital twins from real lab records + TwinLab agents as interactive environments, rather than relying on predefined action spaces
Frontier model developers have a concrete new leaderboard for scientific reasoning that cannot be gamed by memorization or code execution alone
The 100+ action types across 52 labs provide a taxonomy of experimental operations that could standardize how labs describe their workflows for digital-twin construction
More latest news
Read all