A Training-Free Patch for Robot World Models: Just Fix the Joystick, Not the Brain

When a robot's controls rotate, its world model collapses. This paper keeps the model frozen and corrects only the command mapping — doubling task success with zero retraining.

Imagine you're playing a video game and someone silently swaps your controller mapping — suddenly 'up' moves you right, 'left' moves you forward. Your brain's model of the game world is fine; your understanding of the physics hasn't changed. What broke is the link between your thumbstick and what happens on screen. You could relearn the controls from scratch, or you could watch what happens when you push a direction, figure out the remapping, and mentally correct for it. Library of Hands (LoH) takes the second path for robot world models. The committed claim: when a robot's camera viewpoint or controller mapping changes, you don't need to retrain the world model. Instead, LoH infers the new command-to-action mapping ('the hand') by observing what actually happens when the robot acts, then corrects all future commands through that inferred mapping. It stores each mapping it discovers, so if a previously seen configuration returns, it's recognized instantly rather than re-learned. This is training-free — no gradient updates, no new data collection, no catastrophic forgetting. The numbers are stark. The baseline world model (LeWM) drops from 77.5% success to 7.8% when commands are rotated 90 degrees — essentially random. With LoH applied, aggregate success across perturbation scenarios more than doubles, from 28.6% to 62.3%, while degrading performance by only 0.6 percentage points when the original setup is unchanged. The method transfers across three different world models (LeWM, DIAMOND, WorldDreamer) and works on two of three models pretrained on real robot data. Architecturally, LoH sits in the action-correction family — it doesn't touch the learned latent dynamics at all. The 'reading' mechanism is clever: given two consecutive camera frames, LoH searches over candidate actions to find the one that, when fed through the frozen world model, best predicts the observed transition. This is essentially inverse dynamics through the world model's own lens. No external sensor is required; the world model grades its own action hypotheses. The library component is a simple lookup: store identified hand transformations, match incoming observations to stored hands. The validation regime is simulation-only, using three MiniGrid tasks with systematic perturbation types (camera rotation, controller remapping). The authors test across five world models and include ablations on reading quality and library retrieval. Honest limitations are acknowledged: on one of the three real-robot-pretrained models (IRASim), performance drops rather than improves, which the authors attribute to that model's lower reading accuracy. The paper doesn't cherry-pick its best result — it reports the failure. The gap to watch is the jump from MiniGrid to real hardware. The reading mechanism assumes the world model is accurate enough that inverse-dynamics search through it produces reliable hand estimates. In messy real environments with stochastic dynamics, noisy observations, and model error, reading quality will degrade. The authors show that LoH's gain scales with reading quality — which means the method is only as good as the underlying world model's fidelity. The milestone isn't a single number but a threshold: when world models on real robots become accurate enough that single-frame inverse dynamics through them is reliable, LoH becomes a plug-and-play calibration layer that eliminates the need for per-setup retraining. The obvious experiment not run is real robot deployment with physical camera moves or hardware swaps. The authors almost certainly didn't run this because real-robot world models are still too noisy for the reading mechanism to work reliably — the IRASim partial failure hints at exactly this. This isn't sandbagging; it's an honest compute and platform gap. The method's value will be proven or falsified when real-robot world models cross the accuracy threshold that MiniGrid models already clear.