Imagine you're a chess player who can name every piece on the board and recite all the rules, but when asked "what happens after these five moves?" you freeze. You understand the grammar of chess perfectly — you just can't simulate a game forward in your head. That's the gap SWE-Flux exposes in LLMs: fluent code comprehension, broken execution reasoning. The committed claim is clean: no existing benchmark tests LLMs on dynamic execution reasoning at repository scale with automatically harvested, ground-truth answers. Prior work either tested static understanding ("what does this function do?") or tested execution reasoning on toy snippets. SWE-Flux fills the intersection — 480 instances across 12 real Python repositories, covering control flow, loops, program state, dataflow, exceptions, and invariants. The gold answers come from instrumented test runs, not human annotation or LLM judges, which eliminates a circularity problem that plagues much of the current benchmark ecosystem. The results are blunt. The best-performing model across five evaluated LLMs reaches only 37% accuracy. Models do passably on localized behaviors — invariants, intra-procedural control flow, simple loops, exceptions — but fall apart on dataflow tracing, inter-procedural execution, precise state reasoning, and suite-level aggregation. In plain language: LLMs can follow one function's logic but lose the thread when execution bounces across a real codebase. This is exactly the kind of reasoning that matters for real-world debugging and code review. The architecture of the benchmark is the real contribution. The authors instrument test executions in 12 Python repositories, automatically extracting execution traces and constructing questions with verifiable answers. This means no human annotator bias, no LLM-judge circularity, and — critically — a pipeline that can regenerate itself. They demonstrate this with input perturbation: the harvesting pipeline successfully produces valid variant questions for ~90% of selected instances, and those variants are substantially harder for the evaluated models. This is a benchmark that can stay ahead of contamination. The integrity story is solid for a benchmark paper. Ground truth comes from actual code execution, not human judgment. The 12 repositories are real open-source Python projects, not synthetic toy problems. The question categories (control flow, dataflow, state, loops, exceptions, invariants) cover the major dimensions of dynamic reasoning rather than cherry-picking the easiest ones. The main gap is the absence of pre-registration and the relatively small scale — 480 instances across 12 repos leaves room for questions about coverage. The milestone question is whether LLM execution reasoning can cross ~70% accuracy on benchmarks of this difficulty — roughly the threshold where you'd trust an LLM to reliably predict runtime behavior during code review. At 37% today, that's roughly a 2× improvement needed, which in LLM capability terms could be 2-4 years away but only if the training paradigm shifts to incorporate execution traces, not just static code. The perturbation pipeline matters here: it means future models can't just memorize their way past this benchmark. The obvious experiment the authors didn't run: chain-of-thought or tool-augmented evaluation, where models can call an interpreter or step through execution. The paper tests raw reasoning, which is the right starting point, but the practical question — can LLMs compensate for weak execution reasoning by using tools? — is left open. Most likely this is being saved for a follow-up, since it would be a natural second paper that doubles the citation count.
