You know how a GPS trained only on your commute routes can give you perfect turn-by-turn directions to your office — but ask it to navigate to a new restaurant using those same roads, same intersections, same traffic rules, and it chokes? That's the core mechanism here. World models learn dynamics from expert demonstrations, but when you shift the goal — asking the model to recombine familiar pieces in unfamiliar ways — they fall apart. AnyGoal is a benchmark designed to expose exactly this failure mode. The committed claim: current world models do not compositionally generalize. The paper introduces AnyGoal, a suite of goal-shifted task variants built on top of widely used world-model environments (the paper doesn't name a single environment in the abstract, which is a flag). Each variant keeps individual state factors and actions within training support — the model has seen every piece before — but the specific combinations are underrepresented in expert trajectories. The result is a controlled test of whether the model actually learned dynamics or just memorized trajectories. The findings are damning but unsurprising. Across multiple environments and world model families, models consistently fail under goal-shifted recombinations. This is the compositionality gap that's haunted representation learning for years, now demonstrated cleanly in the world-model setting. The contribution is less "here is a surprising result" and more "here is a rigorous way to measure a problem everyone suspected existed." That's valuable — you can't fix what you can't measure — but it's diagnostic, not therapeutic. Architecturally, this is a benchmark-and-analysis paper, not a methods paper. The authors evaluate across multiple world model families (the abstract doesn't name them specifically, which limits our ability to assess breadth) and multiple interaction regimes. The open-source library and project page lower the barrier for others to run these evaluations. The design principle — hold factors in-distribution but shift their combinations out-of-distribution — is clean and generalizable beyond the specific environments tested. The integrity picture is mixed. The authors are testing their own benchmark, which means they chose the environments, the goal shifts, and the metrics. There's no pre-registration. The open-source release helps — others can replicate and extend — but until independent groups run AnyGoal on their own models and confirm the generalization failures, the results rest on the authors' experimental choices. The abstract mentions examining "dynamics and representation failures across model families, interaction regimes, and evaluation metrics," which suggests breadth, but the specifics are absent from this summary. The real value of AnyGoal is as infrastructure. If the benchmark gains adoption, it becomes a standard test for the field's central question: do world models learn physics or just replay tapes? The milestone to watch is whether any model family breaks 80% on goal-shifted tasks within 2-3 years. Right now, the implicit message is that no family does. The obvious next experiment — training with diverse goal distributions rather than just expert rollouts — is conspicuously absent, likely because it's either a separate paper or because early attempts didn't yield clean results. This paper matters most to people building agents that plan with learned world models. If your system only needs to replicate expert behavior, AnyGoal's failures don't bite you. But if you want a model that can plan to novel goals — which is the entire promise of world models over model-free RL — this benchmark tells you exactly where the floor drops out.
