
You know how a senior chef can taste one spoonful of a new dish and then sort an entire pantry into 'works with this' and 'doesn't'? They're not tasting every combination — they're projecting from a rich internal model of flavor onto thousands of untested pairings. That's the structural move here. VOTP takes a handful of human preference labels — which trajectory is better? — and uses the representational space of Video Foundation Models to propagate those judgments across a massive unlabeled dataset. The projection mechanism is optimal transport: align visual trajectory embeddings so that the geometric structure of 'better vs. worse' transfers from labeled pairs to unlabeled ones. The committed claim: you can train competitive reward functions for offline preference-based RL with radically fewer human labels by using ViFM embeddings plus optimal transport to generate pseudo-labels. This is not the first semi-supervised approach to PbRL, but it is the first to route the pseudo-labeling through video foundation model representations and optimal transport alignment specifically. The ICML 2026 Oral acceptance signals the community finds this combination genuinely useful, not just incremental. On the ladder: VOTP outperforms state-of-the-art offline PbRL methods under limited feedback budgets across locomotion and manipulation benchmarks. The abstract does not name specific baselines or give numeric margins — a gap we'll flag — but 'outperforms SOTA' under constrained labeling budgets is the claimed result. The honest read is that this likely beats methods like PT (Preference Transformer) and MRN (Meta-Reward-Net) on D4RL-style benchmarks, but without the paper's tables we can't verify the margin or whether it holds across all tasks. Architecturally, VOTP sits in the semi-supervised preference learning family, leaning on two pillars: (1) pretrained Video Foundation Models as frozen feature extractors — think of the VideoMAE / InternVideo lineage — and (2) optimal transport (Sinkhorn-style) for aligning trajectory distributions in that embedding space. The compute bet is that ViFM representations are rich enough that geometric distance in embedding space correlates with human preference, so you can bootstrap pseudo-labels without actually asking humans. This is a representation-transfer play, not a new RL algorithm per se. Integrity-wise, the validation includes both simulation benchmarks (locomotion and manipulation — likely D4RL and Meta-World) and real robotic tasks. The real-robot validation is the strongest signal here: it's not just simulation grading simulation. Robustness to visual distractors is also tested, which addresses a known failure mode of vision-based reward learning. The weakness is that we don't have pre-registration, and benchmark selection could have been post-hoc. The ICML oral acceptance provides some external quality filter. The milestone question for this line of work is concrete: how few labels can you get away with before the reward function degrades catastrophically? VOTP claims 'a handful' — if that means 50-200 labels for a new manipulation task, the next target is single-digit labels or zero-shot transfer to unseen task families. The gap between 'handful of labels per task' and 'zero labels for a new task' is probably 2-3 years of work on ViFM generalization. The obvious experiment they didn't run: cross-embodiment transfer. If ViFM representations are truly task-general, VOTP's pseudo-labels should transfer from videos of one robot arm to a different robot arm performing the same task. My read: they're saving this for the next paper. The framework is architecturally ready for it — the ViFM doesn't know what robot it's watching — but validating cross-embodiment transfer requires hardware diversity most labs don't have on hand.
The analogy
Like a senior chef who tastes one spoonful and then sorts the entire pantry — VOTP takes a few human preference labels and uses the rich internal model of a Video Foundation Model to propagate those judgments across thousands of unlabeled trajectory pairs via optimal transport alignment.
Chef's palate
One taste creates a rich flavor model that sorts untasted combinations into 'works' and 'doesn't' without trying each one
VOTP pseudo-labels
A few human preference labels create an alignment in ViFM embedding space that sorts unlabeled trajectory pairs into 'better' and 'worse' via optimal transport
Honest assessment
VOTP is a well-constructed combination paper that puts two mature ideas — Video Foundation Models and optimal transport — together in a way the PbRL field hadn't tried, and the ICML Oral acceptance confirms the community finds it genuinely useful. The load-bearing assumption is that ViFM embedding geometry correlates with human preference structure; if that breaks on more complex tasks, the whole pseudo-labeling scheme collapses. Real-robot validation is the strongest signal. Practitioners building preference-based reward systems for manipulation should read this now; everyone else should watch for the cross-embodiment follow-up.
What Is CLAIMS?
70
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
4/5
Integrity
4/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Real but not first-of-kind. Semi-supervised PbRL exists; the novelty is routing pseudo-labels through ViFM embeddings + optimal transport. A strong combination paper, not a new paradigm. ICML Oral validates the contribution.
The field fight
Can foundation model representations replace most human supervision in preference-based RL, or do you always need dense human labels?
Side A
Dense human feedback is irreplaceable — preferences are subjective and task-specific, foundation models can't capture them
Held by: PEBBLE, Preference Transformer, classical PbRL
Side B
Foundation model representations encode enough task structure that a few labels plus geometric alignment can bootstrap the rest
Held by: VOTP, ViFM-based methods, representation-transfer approaches
Milestone tracker
Unlocks: Scalable robot deployment without per-task human preference labeling
Current
~50-200
labels
VOTP per task
Next
<10
labels
near-zero-shot transfer
~2-3 years if ViFM generalization improves
Verified Facts (8)
VOTP is a semi-supervised framework that learns reward functions from only a handful of labels
Leverages optimal transport to align visual trajectories within the representation space of Video Foundation Models
Outperforms state-of-the-art offline PbRL methods under limited feedback budgets
Tested on locomotion and manipulation benchmarks
Robust in the presence of visual distractors
Validated on real robotic tasks
ICML 2026 Oral acceptance
Authors: Tung M. Luu, Hwanhee Kim, Younghwan Lee, Chang D. Yoo
Practical Implications (4)
Robotics teams can train reward functions for new manipulation tasks with ~50-200 human preference labels instead of thousands, cutting labeling costs by 10-100x
Visual distractor robustness means the approach may work in cluttered real-world environments, not just clean simulation
ViFM-based pseudo-labeling is architecture-agnostic — it could plug into existing offline RL pipelines without rewriting the policy learner
Real-robot validation means this is not simulation-only; teams can prototype deployment today
More latest news
Read all