LLMs Can Build World Models That Run — But Not Ones That Mean What You Think

Mechanically valid ≠ semantically faithful. 99% of LLM-generated game models pass execution checks, but only 21% match the intended structure. Executability is fool's gold for neuro-symbolic AI.

Imagine you ask five different architects to build a house from the same verbal description. Every house passes inspection — the plumbing works, the wiring is up to code, the doors open. But when you compare the floor plans, they're wildly different buildings. One has three bedrooms, another has five. The kitchen is upstairs in one, underground in another. They all 'work,' but they don't represent the same house. That's the core finding here: LLMs can generate formally valid world models, but formal validity tells you almost nothing about whether the model captures the actual decision problem you described. The committed claim: LLM-generated world models formatted as partially observable stochastic games (POSGs) achieve near-perfect mechanical validity (134/135 pass strict audit) while exhibiting poor structural agreement with human-authored references (mean F1 of 0.366, only 28/134 exact matches). Executability is necessary but insufficient for downstream neuro-symbolic reasoning. The paper introduces a benchmark of 44 scenarios across three tracks — game-theoretic situations with known formal solutions, open-ended situational scenarios, and culturally loaded dilemmas. Five LLMs (the paper tests across model families, though the abstract doesn't name them individually) each generate POSG formalizations. The game track is the most informative because it has authored reference models to compare against. The cultural track reveals something unsettling: simply naming a country in the prompt changes the structural output of most LLMs, even when automated semantic scores barely budge. The models are encoding cultural priors into their formalizations in ways that don't show up in surface-level quality metrics. Architecturally, this is an evaluation paper, not a methods paper. The contribution is a diagnostic benchmark and a three-dimensional evaluation framework (mechanical validity, structural agreement via permutation-invariant F1, and semantic criteria). The POSG formalism is the vehicle — it's the right choice because it handles multi-agent partial observability, which is where the ambiguity in formalization becomes sharpest. The paper's power comes from the gap it measures, not from a new algorithm. The integrity story is mixed. On the plus side, the benchmark is systematic (44 scenarios, 410 generations), the mechanical audit is strict (not just 'does it parse' but full POSG well-formedness), and the permutation-invariant F1 metric is a thoughtful choice that avoids penalizing equivalent re-orderings. On the minus side, the reference models for the game track are authored by the research team, not drawn from an independent corpus. The semantic evaluation uses automated scoring, which introduces circularity — you're measuring whether LLM outputs satisfy criteria that may themselves be underspecified. No code release or pre-registration is mentioned in the abstract. The milestone question is where this gets practical. Right now, neuro-symbolic systems that use LLM-generated world models for planning or verification are implicitly trusting that executability implies faithfulness. This paper says: stop trusting that. The concrete next number to watch is structural F1 — currently 0.366 on the game track. Getting that above 0.8 while maintaining mechanical validity would mean LLMs can reliably formalize known decision problems, which is the minimum bar for trusting them on unknown ones. The obvious experiment not run: testing whether a human-in-the-loop correction step (where a domain expert reviews and edits the generated POSG) can close the structural gap efficiently. The authors likely didn't run this because it's expensive and shifts the contribution from 'LLM evaluation' to 'human-AI collaboration,' which is a different paper. But it's the experiment that matters most for practitioners who want to actually use LLM-generated world models.