A Recipe Book for Teaching Robots to See, Speak, and Act — Without Forgetting How to See and Speak

Systematic study of how to scale world action models without destroying language-video alignment. 47K hours of data, clear ablations, and 40% real-world SR vs 23.3% for the strongest baseline.

Imagine you've trained a sous chef to identify ingredients and follow spoken recipes by watching thousands of hours of cooking shows. Now you want them to actually cook — hands on the stove. The moment you start correcting their knife technique, they begin forgetting what a carrot looks like. That's the core problem SWAM addresses: how do you bolt action learning onto a pretrained video-language model without breaking the alignment that makes the model useful in the first place? The committed claim is a data-and-supervision recipe, not a novel architecture. The authors curate ~47,000 hours spanning general video, egocentric footage, UMI demonstrations, and robot demonstrations, then systematically ablate which combinations of language-video supervision and language-video-action supervision preserve pretrained capabilities while extending the model to physical control. The key finding: mixing egocentric videos with UMI demonstrations under a fixed compute budget improves control generalization, and the quality of language annotations matters more than most practitioners assume. On the ladder, SWAM posts 88.10% success rate on RoboTwin Clean-to-Clean (Easy) and 61.56% on Clean-to-Random (Hard) in simulation. In the real world — three tasks under unseen conditions — it hits 40.00% SR versus 23.33% for the strongest baseline. These are meaningful numbers, but the absolute real-world rate (40%) underscores how far the field remains from robust deployment. The paper is honest about this: gains come from better data mixing and training recipes, not from an architectural moonshot. Architecturally, SWAM sits in the autoregressive video-prediction family — think of it as a next-token predictor that jointly generates video frames and action tokens conditioned on language. The paper studies architecture variants under fixed compute, finding that model structure affects how efficiently mixed experiences are utilized. Scaling pretraining under the selected configuration yields further gains. The hardware dependency is standard GPU-scale training, not exotic — the leverage is in data curation and supervision design. Integrity is mixed. Simulation evaluation uses RoboTwin, a recognized benchmark, and the real-world evaluation covers three tasks — but three tasks is thin. There's no pre-registration, and the ablation space is large enough that post-hoc narrative construction is a live risk. On the positive side, the authors release curated data, training code, and model weights, which is the strongest possible invitation for independent replication. The milestone question is concrete: at 40% real-world SR across 3 tasks, the field needs ~80% SR across ~20 diverse tasks to claim reliable generalist manipulation. That's roughly a 2× improvement in success rate and a 7× expansion in task diversity. If this data-recipe approach scales linearly (unlikely), that's 2-4 years of community effort. More realistically, new architectural ideas will be needed alongside better data. The obvious experiment not run: scaling to a significantly larger model and data budget to see if the recipe transfers. The authors ablate within a fixed budget, which is methodologically clean but leaves the big question unanswered. The honest read is (a) — compute budget limited what they could explore. The next paper from this group will almost certainly be the scaling curve.