Imagine you're dropped into a country where traffic lights mean the opposite of what you've learned your entire life. Green means stop. Red means go. Yellow means reverse. You have a badly translated phrasebook — some entries are right, some are wrong, and you don't know which. Your only option is to drive, observe what happens, adjust your mental model, and try again. That's the core mechanism of ExplorationBench: force AI systems into environments where their pre-trained knowledge is actively misleading, hand them a flawed manual, and measure whether they can discover the real rules through iterative experimentation. The committed claim: ExplorationBench is the first benchmark that simultaneously solves the two hard problems of evaluating scientific exploration in AI — verifiability (every answer is machine-checkable because the alien worlds run executable code) and contamination control (the rules deliberately conflict with real-world knowledge, so memorization from pre-training cannot help). This is a benchmark-design contribution, not a model contribution. The paper does not propose a new exploration algorithm; it proposes a new way to grade exploration. The benchmark consists of two sandboxes. AlienCode presents 31 discovery targets across 70 tasks in a programming-language-like environment where familiar operators behave differently. AlienLogic offers 24 discovery targets across 70 tasks in a formal logic environment with altered inference rules. Each sandbox gives the system a flawed manual — deliberately seeded with errors — plus environmental feedback from tool calls. The system must explore, update its beliefs, and then solve held-out tasks that require the corrected rules. This is structurally clever: the flawed manual is the benchmark's equivalent of prior knowledge that's partly right and partly dangerous. Ten AI systems were evaluated. The paper reports that the strongest systems can acquire and apply unfamiliar rules, but performance varies substantially across trajectories — meaning the same system, given different exploration paths, can produce wildly different outcomes. Perhaps most interesting: continued exploration can stall or reverse earlier gains. This is a real empirical finding about exploration dynamics, not just a leaderboard entry. It suggests that current systems lack robust exploration strategies and can degrade their own understanding by over-exploring or exploring poorly. The integrity picture is mixed. The benchmark design is sound — executable verification eliminates subjective grading, and the alien-world trick is a genuine advance for contamination control. But the evaluation is entirely self-contained: the authors built the worlds, built the tasks, and ran all the evaluations. There's no independent replication, no pre-registration of evaluation protocols, and the 10 systems tested are all current LLM-based agents. The paper doesn't compare against classical planning or search baselines that might perform well in small discrete rule-discovery settings. This matters because it leaves open whether LLMs are being tested on exploration or on a specific kind of language-mediated reasoning that happens to correlate with exploration in these sandboxes. The architecture is agent-based LLM evaluation with tool-call interfaces. Systems interact with the sandbox through a dedicated tool-call schema — they can run experiments, query the environment, and receive structured feedback. This places the work in the growing family of LLM-as-agent benchmarks (SWE-bench, WebArena, MINT), but with a distinctive twist: the environment is adversarial to the agent's pre-existing knowledge rather than merely unfamiliar. The compute requirement is inference-time only — no training involved — which makes the benchmark accessible but also means it's measuring prompted exploration, not learned exploration. The milestone question is where this gets interesting. The current benchmark has 55 discovery targets across 140 tasks in two sandboxes. For this evaluation paradigm to matter, it needs to scale to environments complex enough that exploration strategy — not just single-step reasoning — becomes the bottleneck. The authors haven't shown that their alien worlds are hard enough to distinguish sophisticated exploration from brute-force trial-and-error at scale. The obvious next experiment is testing whether systems that implement explicit hypothesis-tracking or Bayesian updating outperform vanilla chain-of-thought prompting — and the honest read is the authors are saving this for a follow-up, since this paper's contribution is the benchmark itself, not the exploration algorithms.
