Imagine you're a driving instructor. Your student parks the car perfectly every time — but you never check whether they signal before lane changes, recover smoothly from a missed turn, or maintain speed consistency through curves. You'd pass every student who gets lucky once. That's the state of robotic value model evaluation today, and RoboValue is the first attempt to build the full driving test. The core claim: existing evaluations of robotic value models — the systems that watch a robot arm and score whether it's doing the task correctly — test only narrow slices of execution understanding. A model might correctly identify that a task succeeded (outcome detection) but completely fail to track intermediate progress, recognize recovery from mistakes, or maintain consistent scoring under reordered subtasks. RoboValue proposes a four-dimensional evaluation framework covering task-state understanding, temporal progress monitoring, failure-and-recovery reasoning, and value consistency, with shared interfaces so models face the same test regardless of their architecture. The dataset spans 15 simulation tasks and 20 real-world manipulation tasks, with expert demonstrations for training and annotated trajectories for fine-grained evaluation. This sim-and-real split matters: simulation lets you control for variables, real-world tests whether the signal survives sensor noise, lighting variation, and embodiment differences. The benchmark tests models in both zero-shot (no task-specific examples) and one-shot settings, and separately probes robustness under shifts in robot embodiment and environment — does the value model still work when the arm changes or the table moves? The results are the real contribution. Models that score well on forward-progress and outcome detection — the easy parts — systematically fail on instruction grounding (does the model actually understand what the instruction requires?), history-dependent progress tracking (does it know where in a multi-step sequence the robot is?), and recovery assessment (can it tell when the robot has corrected a mistake?). Perhaps most strikingly, the same semantic subtask receives substantially different value gains depending on execution order — a consistency failure that would silently corrupt reward signals during policy learning. Architecturally, RoboValue is benchmark infrastructure, not a new model. It evaluates "representative robotic value models" — likely including VLM-based reward models, learned success classifiers, and possibly language-conditioned value functions — though the abstract doesn't name specific models evaluated. The framework's value is in its diagnostic specificity: rather than a single leaderboard number, it produces a profile showing exactly where a value model breaks. This is the difference between knowing a student failed the test and knowing they can't parallel park. The integrity picture is mixed in useful ways. Sim-and-real coverage is a genuine strength; most robotics benchmarks live in one world or the other. The diagnostic trajectory design — testing instruction variations and execution order permutations — is methodologically sound for catching the specific failure modes the authors care about. But the 35-task scope, while respectable, is still narrow relative to the ambition of "unified evaluation," and the paper's own findings about order-dependent value inconsistency suggest the evaluation space is larger than what's been explored. The practical implication for anyone training robot policies with learned reward signals: your value model is probably lying to you about progress and recovery, even when it correctly identifies success. RoboValue gives you a way to measure exactly how much, and a public leaderboard to track whether the field is actually fixing it.
