Imagine you can recite every step of a soufflé recipe from memory — timing, temperatures, the fold technique — but when you stand at the counter, you can't tell which bowl is the mixing bowl, can't judge whether the oven is at temperature from its indicator light, and keep reaching for the knife handle-first. That's the gap BEAKER exposes in today's multimodal AI models: fluent procedural narration, catastrophic embodied execution. The benchmark itself is a 1,000-question exam across 11 task types, built entirely by chemistry and embodied-AI experts with zero AI-assisted generation. Tasks span affordance grounding (where exactly on a piece of equipment should a robotic arm grab?), trajectory planning (what collision-free path moves a flask from station A to station B?), fine-grained apparatus and state recognition, safety-norm identification, and temporal procedure reasoning. The coverage is deliberately execution-oriented — not 'what is this molecule?' but 'which region of this fume hood is the correct insertion point for this flask, and in what order do the next four physical actions happen?' Twenty-eight models were evaluated in a single-turn, offline setting: general-purpose heavyweights (GPT-4o, Gemini, Claude), science-oriented models, chemistry-specialized systems, and embodied MLLMs. The headline result is a clean separation between what models can and cannot do. Coarse procedural understanding — watching a video and summarizing the experimental steps — is within reach for frontier models. But affordance grounding and trajectory planning scores collapse. Models that can narrate 'now transfer the solution' cannot reliably identify the operable grasp region on the beaker or plan a feasible path that avoids the adjacent apparatus. The diagnostic value here is in the task-type decomposition. BEAKER doesn't produce a single leaderboard number that conflates reading comprehension with spatial reasoning. Instead, it separates 11 capabilities and shows where performance craters. Fine-grained apparatus recognition is surprisingly weak — models confuse volumetric flasks with Erlenmeyer flasks, misidentify safety equipment, and fail to distinguish clean from contaminated states. Mid-length procedure ordering (5-10 steps) is another failure mode: models handle 3-step sequences and very long protocols (where they can rely on memorized templates) but stumble on the middle ground where genuine temporal reasoning is required. The expert-curation pipeline is the load-bearing structural choice. Every question and annotation was hand-created by domain experts, which eliminates the circular problem that plagues AI-generated benchmarks (you can't diagnose AI weaknesses using AI-generated test items). The tradeoff is scale: 1,000 samples is enough for diagnostic work but not for training. This is an exam, not a curriculum. For the self-driving lab community, the practical takeaway is sobering: no current MLLM is close to serving as a reliable 'embodied brain' for autonomous chemical experimentation. The bottleneck isn't language understanding or even visual recognition of common objects — it's the precise, action-grounded spatial reasoning that connects perception to physical manipulation. Closing this gap likely requires architectures that fuse language models with dedicated spatial-reasoning and physics-simulation modules, not just bigger vision-language pretraining. BEAKER is accepted as a Spotlight at NeurIPS 2026 on the Evaluations and Datasets track, which signals the community considers this a significant diagnostic contribution. The benchmark's longevity depends on whether the 11 task types remain discriminating as models improve — if affordance grounding scores jump from 30% to 90% within a year, BEAKER will need harder items. But given how far current models are from ceiling, that's unlikely to be a near-term problem.
