
Read the paper
Auditing a Learned Simulator for Legged Robots: The Robotic World Model on ANYmal D
on GitHub
PRUNING
+30%
WEIGHTS KEPT
+90%
STEP AHEAD
+39%
STEPS YET COVERED
+4.6%
Imagine you're a driving instructor grading students by how well they parallel park once — but then sending them onto the highway for a road trip. A student who nails the parking test might stall on the first on-ramp. That's what's happening in learned simulators for legged robots: models are evaluated one step ahead, then asked to run forward for hundreds of steps on their own predictions, and the one-step grade doesn't predict the road-trip performance. Joyjeet Singh audits the Robotic World Model (RWM) for the ANYmal D quadruped — a published neural-network simulator that predicts the robot's next physical state and then chains those predictions forward to train walking policies. The audit is unusually thorough: 101 trained simulators, 200 pruned copies, 38 alternative architectures, all rebuilt to match the released code's outputs, losses, and gradients exactly, and all without a GPU. The core finding is that multi-step (rollout) training beats one-step training at 368 steps across all ten held-out episodes — a pre-registered result — even though one-step training looks better on the one-step metric. The implications for model selection are stark. Pruning 30% of a model's weights preserves 90% of its one-step skill but only 39% of its performance at two seconds of rollout. This means the cheap check everyone uses — one-step accuracy — actively misleads you when deciding which model to deploy. Ensemble disagreement from the follow-up system RWM-U ranks rollout error well (Spearman ρ = +0.90 at 100 steps), but its uncertainty coverage is catastrophically miscalibrated: 4.6% of outcomes fall within one sigma versus the 68% you'd expect from a calibrated estimate, because each individual network's variance collapses under the implemented loss. The most provocative result is at the robot level. All 18 fine-tuned model copies the author tested predicted better than the released model at both one-step and eight-step horizons — yet 13 of them produced policies that failed to walk. Prediction accuracy did not predict walking ability. Pre-registered tests of whether any evaluation check reliably ranks simulators by policy quality were "not settled." In a secondary (non-pre-registered) test, running the original authors' policy inside each simulator did rank them well, but that requires already having a working policy — a catch-22 for model development. The paper's integrity practices deserve explicit mention. The author pre-registered hypotheses, reports nine claims he withdrew after they failed, publishes all code and data across two repositories, and scopes the work honestly: one robot, one gait, simulated data, no hardware. This is a 39-page audit with 13 figures and 20 tables — it reads like a forensic accounting report, not a sales pitch. The deeper lesson is about the gap between surrogate metrics and task performance in learned simulators. The robotics community is building increasingly sophisticated "world models" to train policies in simulation, but if the metrics used to select and validate those models don't correlate with downstream task success, the entire pipeline has an undiagnosed measurement problem. This paper doesn't solve that problem, but it documents it with unusual rigor on a concrete, published system.
The analogy
You're a driving instructor who grades students by parallel parking quality, then sends them on a 400-mile road trip. The parallel-parking score doesn't predict who stalls on the highway. This paper documents exactly how badly that one-test-predicts-all assumption fails for a robot's learned simulator.
Driving test
Grading a single maneuver (parallel park) and assuming it predicts highway driving over hundreds of miles
1-step vs rollout
Evaluating a learned simulator one step ahead and assuming it predicts 100+ step rollout quality and policy success
Honest assessment
This is a rare and valuable thing: a meticulous, pre-registered audit of someone else's published ML system by a solo researcher who reports his own failures alongside his findings. It doesn't propose a new method — it documents that the standard way of evaluating learned robot simulators is broken, using 101+ models and transparent methodology. The scope is narrow (one robot, one gait, simulation only), and the robot-level disconnect between prediction accuracy and walking ability is documented but not resolved. Engineers building world models for robotics should read this for the methodology and the warning; everyone else should note that the field's evaluation practices have an undiagnosed measurement problem.
What Is CLAIMS?
77
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
4/5
Architecture
4/5
Integrity
5/5
Milestone
3/5
Successor
3/5
What is this paper actually committing to?
Genuine audit contribution: documents that one-step metrics misjudge design choices and prediction accuracy doesn't predict walking. Not a new method — a measurement finding on a published system, pre-registered.
The field fight
Can learned simulators (world models) replace or augment physics simulators for training robot locomotion policies?
Side A
Learned simulators are fast enough and accurate enough to train policies; one-step metrics validate them
Held by: RWM, DayDreamer, IRIS, world-model-for-robotics community
Side B
Learned simulators accumulate compounding errors over rollouts; standard metrics don't capture downstream task failure
Held by: Classical sim-to-real community, this paper
Milestone tracker
Unlocks: Reliable model selection for learned simulators without requiring a working policy or hardware access
Current
0.90
Spearman ρ
ensemble disagreement ranks rollout error
Next
0.90+
Spearman ρ
cheap metric that ranks simulators by policy success
Not reached — accuracy-to-walking correlation 'not settled' in this paper
Verified Facts (7)
Training on the model's own rollouts beat one-step training at 368 steps on all ten held-out episodes (registered)
Pruning 30% of the weights kept 90% of the model's skill one step ahead and 39% at 2 s
Ensemble disagreement ranked rollout error well (Spearman +0.90 at 100 steps), yet covered 4.6% of outcomes at one sigma, against 68% for a calibrated estimate
All 18 fine-tuned copies we tested predicted better than the released model one and eight steps ahead, yet 13 gave failing policies
101 trained simulators, 200 compressed copies and 38 simulators of three other architectures
39 pages, 13 figures, 20 tables
Nine of our own claims that we withdrew
Practical Implications (4)
Do not use one-step prediction error to select learned simulators for policy training — it misjudges pruning, architecture, and fine-tuning decisions
Ensemble disagreement ranks rollout error well (ρ=0.90) but is catastrophically miscalibrated for uncertainty coverage (4.6% vs 68% expected) — do not use it for safety bounds
Pruning 30% of weights looks fine one step ahead but destroys 61% of multi-step skill — compression requires rollout-aware evaluation
Better prediction accuracy does not predict better walking policies — 13/18 fine-tuned models with improved accuracy produced failing policies
More latest news
Read all