No Frontier LLM Can Pick the Right Clinical Trial Design — and Now There's a Benchmark to Prove It

617 oncology dose-finding trials, each with an expert-validated design. Five frontier LLMs fail to match. The gap traces to missing latent variables and lack of decision-tree reasoning.

Imagine you're a chess engine, but half the pieces are invisible. You can see the board — the cancer type, the drug class, the patient population — but critical pieces like the sponsor's risk tolerance, whether the drug has a known steep dose-toxicity curve, or how many dose levels are practically manufacturable are simply not written down anywhere. That's the problem facing any LLM asked to recommend a Phase-I clinical trial design: the recorded trial description is an incomplete specification of the problem that produced the design choice. TrialDesignBench is the first benchmark that forces this question into the open. The authors curated 617 Phase-I and Phase-I/II oncology dose-finding trials from ClinicalTrials.gov, each mapped to one of nine recognized statistical design families — from the classic 3+3 rule-based approach to model-assisted designs like BOIN and Bayesian model-based designs like CRM. Crucially, every trial's reference label was validated against current FDA Project Optimus guidance, which since ~2022 has pushed the field away from maximum-tolerated-dose thinking toward optimal biological dose paradigms. This means the benchmark isn't grading against historical practice — it's grading against the contemporary expert standard. Five frontier LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.1 405B, DeepSeek-V2.5) were tested with chain-of-thought prompting. None matched the expert reference. The headline metric is class-wise balanced accuracy — meaning performance averaged across all nine design categories, not dominated by the most common ones. All models underperformed, and the failure modes were systematic, not random. The ablation study is the load-bearing part of the paper. Two interventions moved the needle most: supplying the latent design determinants that aren't recorded in trial registrations (things like whether the drug has a narrow therapeutic window, or whether combination-agent constraints apply) boosted balanced accuracy by +0.22. Separately, structured decision-tree reasoning — walking the model through the branching logic an experienced statistician uses — added +0.24. These aren't small effects on a balanced-accuracy scale. They point directly at the two capabilities LLMs lack: inferring unstated context from sparse clinical signals, and executing multi-step conditional reasoning over design taxonomies. The benchmark's integrity rests on the formalization of expert knowledge into a reproducible decision process. The authors essentially reverse-engineered how a senior biostatistician picks a trial design, encoded that logic, and used it to produce reference labels. This is a strength — it makes the benchmark reproducible and auditable — but also a limitation: it assumes one correct answer per trial, when in practice reasonable experts might disagree on borderline cases. The paper acknowledges this but doesn't quantify inter-expert agreement. The practical implication is clear: if you're building AI tools for clinical trial design, you now have a benchmark that reflects post-Project Optimus reality. The field can stop evaluating LLMs against outdated design conventions. But the gap between current LLM performance and expert-level design selection remains large, and the ablation results suggest the fix isn't just more pretraining data — it's structured reasoning over latent clinical variables that don't appear in any training corpus. TrialDesignBench is a diagnostic tool, not a solution. It tells you exactly where LLMs break on this task and why. The next step — building systems that can infer those latent determinants and reason through the design decision tree without hand-holding — is the hard part, and this paper doesn't attempt it. That's honest scoping, not a gap.