Imagine you're playing a board game by mail. Your opponent sends you a list of ten moves they made while you weren't looking, and you have to reconstruct the entire board from scratch. Easy after one move. After ten? You're holding a stack of dependent transformations in your head, each one reshaping the board the previous one left behind. That's the exact cognitive demand Grid-Bench places on large language models — except the board can be two-, three-, or even four-dimensional. The committed claim: LLMs cannot reliably predict the complete state of a discrete grid world after sequential spatial transformations, and existing benchmarks don't measure this because they evaluate single-step reasoning, not accumulated multi-step state tracking. Grid-Bench fills that gap with eight operation families (movement, region transforms, environmental modifications) across 2D/3D/4D grids, where every predicted cell value is independently checkable against a deterministic reference execution. Six LLMs were evaluated, and the headline finding is a sharp accuracy cliff. Models that look comparable on single-event tasks diverge dramatically as history length increases. This is the key insight: single-step spatial benchmarks are hiding a critical failure mode. Two models with similar exact-match accuracy on short histories can differ substantially in the number of incorrect cells on shared long-history instances — meaning one model degrades gracefully while another falls off a cliff. Architecturally, the benchmark is model-agnostic by design. It provides text-encoded grid states, explicit transformation rules, and event histories as prompts, then demands a complete final grid as output. There's no special spatial encoding, no graph structure, no vision component — just raw sequential text-to-text reasoning. This is a deliberate choice: it tests the LLM's internal ability to compose state transformations, not its ability to leverage a spatial API. The eight operation families span rigid-body motion, cellular-automaton-style updates, flood fills, and multi-dimensional slicing. The integrity story is solid for a benchmark paper. Every answer is deterministically verifiable — no human annotation, no ambiguous scoring. The reference executor produces ground truth, so there's no grading-your-own-homework problem. The authors evaluate six models (names not specified in the abstract but detailed in the full paper's 39 pages and 15 tables), and the benchmark infrastructure — code, data, and project website — is fully released. The weakness is that there's no comparison to non-LLM baselines (classical planning algorithms, symbolic executors) that would contextualize whether the task is fundamentally hard or just hard for language models. The milestone question is where it gets interesting. Right now, complete-state accuracy degrades sharply with history length — we don't know the exact numbers from the abstract alone, but the qualitative finding is clear. The threshold to watch is whether any model can maintain, say, >90% cell-level accuracy on 20+ step histories across 3D grids. That number would signal genuine compositional state-tracking rather than pattern matching. We're clearly not there yet. The obvious next experiment the authors didn't run: testing whether chain-of-thought prompting, scratchpad mechanisms, or tool-augmented models (where the LLM calls a grid executor mid-sequence) close the gap. The honest read is probably (c) — saving it for the next paper. This benchmark is designed as infrastructure. The tool-use comparison is the natural follow-up, and publishing the benchmark first establishes the measuring stick before others build on it.
