
Read the paper
One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning
on GitHub
Imagine you have a friend who can watch any cooking video and predict the next five seconds — the sizzle, the flip, the plating — but can't tell you what the cook did to make it happen. They know the physics intuitively but lack the vocabulary. O²LaF is the trick that gives them the vocabulary by forcing them to compress their predictions through a tiny bottleneck: if you can describe the transition between frames in just a few numbers, and those numbers let you reconstruct what happens next, then those numbers ARE the action. The committed claim: a single pretrained video diffusion transformer (DiT), duplicated and adapted into two roles — inverse dynamics (what action caused this transition?) and forward dynamics (given this action, what happens next?) — learns transferable latent action representations from raw unlabeled video that beat all prior methods on few-shot action probing and enable zero-shot transfer across different robots and visual contexts. This is not the first latent action paper, but it is the first to show that video generation priors are sufficient scaffolding for both directions of the dynamics loop. The architecture is clean. Both the inverse and forward models start from the same pretrained video DiT weights. The inverse model strips out autoregressive causal masking and adds learnable action queries that attend to frame transitions, producing a compact latent action vector. The forward model keeps causal generation intact but conditions on these latent actions. A key design choice: information access is deliberately asymmetric — the inverse model sees full temporal context while the forward model is causal — and both are trained end-to-end through the bottleneck. The bottleneck is what forces meaningful compression. Without it, the latent codes would just memorize pixel patches. On the ladder, O²LaF reports new state-of-the-art on standard benchmarks for few-shot action probing in navigation (Minecraft, Habitat) and robotic manipulation (SIMPLER, Bridge). The specific baselines are LAPO, LAPA, and other recent latent action methods. The paper reports substantial gains: for example, in zero-shot cross-embodiment transfer on SIMPLER, O²LaF improves success rate by meaningful margins over LAPA. Critically, the authors also show the framework extends beyond generative priors — swapping in V-JEPA2 (a predictive representation model) as the base prior yields competitive performance, suggesting this is a general paradigm, not a DiT-specific trick. Integrity is solid but bounded. Evaluations span multiple domains and embodiments, which resists cherry-picking. Code and a project page are released. The benchmarks are community-standard for this subfield (Minecraft, Habitat, SIMPLER, Bridge). However, there's no independent replication, the evaluations are all in simulation or from pre-collected datasets, and the few-shot probing protocol — while standard — is effectively self-graded by the same community developing these methods. The honest caveat: real-robot deployment with these latent actions remains undemonstrated. The milestone to track is real-robot closed-loop control using latent actions learned purely from unlabeled video. Right now the pipeline is video-in, latent-actions-out, evaluated in simulation or via offline datasets. The gap is getting these compressed action codes to actually command a physical robot arm in a novel task. That's probably 1-2 years away if the latent actions prove robust to sim-to-real distribution shift. The experiment they didn't run — and the most telling absence — is closed-loop real-robot evaluation with the learned latent actions driving actual manipulation. The authors demonstrate zero-shot transfer across simulated embodiments, which is the prerequisite, but stop short of the real thing. Best read: they're saving it for the next paper, or the sim-to-real gap is not yet closed and they know it. Either way, it's the obvious next step.
The analogy
You have a friend who can watch any cooking video and predict the next five seconds — the sizzle, the flip — but can't name the action that caused it. O²LaF forces that friend to compress their implicit knowledge through a tiny vocabulary bottleneck: if a few numbers let you reconstruct the future, those numbers ARE the action.
Cooking intuition
A skilled viewer predicts what happens next in a cooking video but can't name the technique used
Video DiT prior
A pretrained video generator predicts future frames but encodes no explicit action labels — bottleneck compression forces it to invent them
Honest assessment
O²LaF is a well-engineered demonstration that pretrained video DiTs contain enough dynamics knowledge to bootstrap latent action learning without any action labels, and that this transfers across embodiments in simulation. The dual-fork-with-bottleneck design is elegant and the V-JEPA2 extension shows real generality. The ceiling on impact right now is clear: everything is in simulation, and whether these latent actions survive contact with real robots is the entire ballgame. If you're building robot learning pipelines, this is worth reading carefully as a leading indicator. If you're waiting for deployment-ready methods, check back in a year.
What Is CLAIMS?
73
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
4/5
Architecture
4/5
Integrity
3/5
Milestone
4/5
Successor
4/5
What is this paper actually committing to?
Real but not first-of-kind. Latent action learning exists; the genuine contribution is showing a single pretrained video DiT prior suffices for both inverse and forward dynamics via bottleneck compression.
The field fight
Do robots need action-labeled data to learn manipulation, or can world models learned from passive video provide sufficient action grounding?
Side A
Action labels are essential; latent actions are too noisy for real control
Held by: RT-2, Octo, π0 (action-labeled pipelines)
Side B
Pretrained video models encode enough physics that action grounding can be recovered from unlabeled video via compression
Held by: LAPO, LAPA, this paper (O²LaF)
Milestone tracker
Unlocks: Robot manipulation policies learned entirely from internet video — no teleoperation data
Current
SOTA
sim probing
O²LaF (this paper)
Next
1
real-robot task
Closed-loop control from latent actions
~1-2 years if sim-to-real transfer for latent action codes proves tractable
Verified Facts (10)
Authors: Yuxin Jiang, Wenqiang Sun, Jiaqi Li, Yuchao Gu, Lvmin Zhang, Ivor Tsang, Mike Zheng Shou
Method name: O²LaF (One Prior, Two Dynamics)
Both inverse and forward models initialized from the same pretrained video DiT
Inverse dynamics uses learnable action queries; forward dynamics uses causal action-conditioned generation
Evaluated on navigation (Minecraft, Habitat) and robotic manipulation (SIMPLER, Bridge)
Claims new state of the art in few-shot action probing and zero-shot cross-embodiment transfer
Framework extended to V-JEPA2 predictive representation prior with competitive performance
Project page: https://showlab.github.io/o2laf
Code: https://github.com/showlab/O2LaF
Categories: cs.CV, cs.AI, cs.LG, cs.RO
Practical Implications (4)
Teams building robot learning systems can potentially pre-train inverse and forward dynamics models from large unlabeled video corpora, reducing dependence on expensive teleoperation datasets
The dual-fork architecture pattern — one pretrained model, two adapted roles, information-asymmetric training — is a reusable design pattern for other self-supervised settings
V-JEPA2 compatibility means practitioners aren't locked into generative video models; predictive representation models also work as priors
Zero-shot cross-embodiment transfer, if it holds in real settings, would let labs train on cheap simulation embodiments and deploy on expensive real robots
More latest news
Read all