Imagine you hand a toddler a ball that falls upward. Within seconds, they're throwing everything they can reach at the ceiling. Now imagine you hand that same ball to a very articulate adult who has memorized every physics textbook ever written — and they calmly report that the ball fell downward, because that's what balls do. That's roughly what happens when you drop most LLMs into a world where the rules have changed and ask them to figure out what's different. This paper introduces LongChang, an automated benchmark framework that creates simulated physical environments where one or more laws deviate from ordinary physics — heavier objects fall slower, friction works in reverse, that kind of thing. The twist: the LLM explorer receives no task, no hints, no research question. It must notice anomalies on its own, design experiments, build tools, and write a discovery report. Separate LLM agents handle scene generation, tool review, and report evaluation using the altered laws as ground truth, creating a fully automated loop with no human in it. The committed claim is stark: open-ended scientific discovery — the full cycle from anomaly detection through hypothesis formation to experimental validation — is a capability that current LLMs almost entirely lack. Among 12 models tested, only GPT-6-Astra achieves a non-zero discovery rate at 16.7%. The remaining 11 models, including what we can infer are strong frontier models, score exactly 0%. The exploration logs reveal a telling pattern: models frequently observe anomalous behavior but fail to investigate it, or investigate partially but never close the loop to a correct law. The architecture is a multi-agent pipeline rather than a single model loop. One LLM acts as the explorer (manipulating objects, proposing experiments). Others serve as scene generators, tool designers, and report evaluators. The scene evolves based on exploration records, so the environment adapts to push the explorer toward productive territory. This is closer to an agent scaffold than a single-inference benchmark — the explorer gets multiple rounds of interaction with a simulated physics engine. The framework sits in the growing family of agentic evaluation harnesses (SWE-bench, WebArena, ScienceAgentBench) but targets the uniquely open-ended task of unsupervised hypothesis generation. The integrity picture is mixed. On the positive side, the evaluation is automated against known ground truth (the altered laws are defined by the framework, so correctness is objective), and 12 models provide reasonable breadth. On the negative side, there's no independent replication, no public code at the time of review, and the evaluator LLMs introduce a layer of circularity — LLMs grading LLMs on scientific reasoning. The 16.7% result for GPT-6-Astra is based on an unspecified number of trials across an unspecified number of altered-law scenarios, making it hard to assess statistical reliability. The paper also doesn't compare against a non-LLM baseline (e.g., a simple anomaly-detection heuristic that flags deviations from Newtonian predictions), which would help calibrate whether the task is hard because it requires reasoning or hard because the interface is awkward. The milestone question is where this gets interesting. If 16.7% is the frontier today for single-law discovery, the field needs to track two numbers: discovery rate on single-law alterations and discovery rate on multi-law alterations. The paper implies multi-law scenarios exist in the framework but doesn't report separate rates. A reasonable next milestone is 50% single-law discovery rate from a frontier model — which would suggest the reasoning loop is becoming reliable rather than lucky. We're likely 2-3 model generations away from that, if the trajectory holds. The obvious experiment not run: giving the explorer a hint. The paper tests fully open-ended discovery, but a calibrated hint condition (e.g., 'something about gravity is different') would reveal whether the bottleneck is anomaly detection, hypothesis formation, or experimental design. This is a clean ablation the authors almost certainly considered. The most likely reason it's absent: they're saving the hint-gradient study for a follow-up paper, since the zero-discovery result for 11 models is already a strong headline finding.
