A Robot That Rehearses in Its Head Before Touching Anything — and Beats Trained Policies by 20 Points

Frozen VLM learns manipulation tasks via simulated rehearsal written to context, not weights. 83.4% success on 50 tasks from one demo, zero gradient updates.

Imagine you're learning to cook a dish you've never made. Someone shows you a video of the recipe once, in a different kitchen with different pans. You can't practice in your own kitchen yet — but you CAN practice in a video game version of it, take notes on what went wrong, and then walk into the real kitchen with a cheat sheet. That's the core mechanism of World In-Context Learning (WICL): replace weight updates with rehearsal notes stored in the model's context window. The committed claim is this: a frozen vision-language model, given a single demonstration of a manipulation task, can learn that task by rehearsing it in simulation, storing the feedback as structured "Task Memory," and deploying that memory to new scenes — matching or exceeding trained action policies that required far more data and gradient updates. On RoboTwin 2.0's full 50-task benchmark, RoboScript hits 83.4% mean success across clean and randomized settings, outperforming the strongest trained policy by over 20 percentage points. That's a genuine result, not a cherry-picked subset. The architecture is a modular agentic pipeline built around a frozen foundation model (likely a large VLM). RoboScript takes a single human demonstration, parses it into semantic task steps and object roles (not pixel-level imitation), then enters an iterative rehearsal loop inside a simulator. Each rehearsal generates feedback — what worked, what failed, what to adjust — and this feedback gets frozen into a structured Task Memory. At deployment, the robot reads the Task Memory as context, with strict data isolation ensuring every input is available on real hardware. No gradients flow. The model's weights never change. All learning lives in the context. The ladder comparison is strong: 83.4% vs. the previous best trained policy's roughly 63% on the same benchmark. The gap isn't a rounding error — it's 20+ points on 50 tasks with both clean and randomized visual conditions. The paper also shows generalization across visual distributions, improvement through reflection loops, and scaling with the underlying foundation model's capability. Cross-embodiment transfer to a physical dual-arm robot is demonstrated, though the physical experiments are naturally smaller-scale than the simulation sweep. Integrity is mixed-to-solid. The benchmark (RoboTwin 2.0, all 50 tasks) is a community standard, not a custom-built playground. Results span both clean and randomized settings, which matters — randomized settings test whether the system memorized visual layouts or actually learned task structure. Code is released on GitHub. However, there's no independent replication yet, the physical robot experiments are limited in scope, and the reliance on a simulator for the rehearsal loop raises the sim-to-real transfer question that every robotics paper faces. The key insight — that what agents learn in context is semantic (task steps, object roles) rather than visual — is the load-bearing observation. If true generally, it explains why experience gathered in one world transfers to another and why context-based learning can beat weight-based learning that overfits to training distribution. The paper provides evidence but doesn't prove this is universally true; adversarial distribution shifts or tasks requiring fine motor precision might break the semantic abstraction. The obvious next experiment is scaling to tasks that require genuinely novel manipulation primitives not seen in any demonstration — true zero-shot generalization beyond semantic composition of known steps. The authors likely didn't run this because the current framework assumes you have at least one demo per task, and removing that assumption requires a different architecture for task decomposition. They're probably saving compositional zero-shot for the next paper, once the rehearsal-in-context paradigm is established.