
Imagine you're a bartender sliding a pint glass down a wet counter to a customer six stools away. You don't consciously calculate friction coefficients or center-of-mass offsets — but your brain is running a physics engine in real time, adjusting force and release angle based on how heavy the glass feels and how slippery the bar is. Now imagine a bartender who can SEE the glass perfectly, NAME it in seventeen languages, but has zero intuition about how hard to push. That's the state of current Vision-Language-Action models, and PhyDEX is the benchmark that proves it. The paper's committed claim: VLA models that appear competent at pick-and-place manipulation are physically blind — they cannot infer mass, center of mass, or friction from visual and proprioceptive input well enough to execute dynamic tasks like throwing. PhyDEX operationalizes this by creating a throwing benchmark where a robot arm must toss objects to destinations beyond its kinematic reach. The dataset is substantial: 35,000 trajectories across 143 hours, nine tasks, 100 scenes, covering uniform, irregular, and dynamic-mass objects with static, multiple, and moving targets. The pipeline — combining grasp generation, motion planning, and closed-loop throw optimization under randomized physical properties — is fully automatic and scalable. The ladder here is brutal. Three representative VLA models were tested: NVIDIA's GR00T N1.7, DynamicVLA, and ForceVLA. All achieved success rates below 3%. This isn't a marginal miss — it's a category failure. The paper's most telling diagnostic: when ground-truth physical properties (mass, center of mass, friction) were provided directly to the policies, success roughly doubled. That's still terrible in absolute terms, but the delta isolates the bottleneck. The models aren't just bad at throwing — they're bad at inferring the physics that throwing requires. A secondary bottleneck compounds this: pretrained model robustness is so fragile that most failures occur BEFORE the throw even happens, during grasping and pre-throw motion. Architecturally, the evaluated models span the current VLA landscape: GR00T N1.7 represents the large-scale pretrained foundation model approach (transformer-based, language-conditioned); DynamicVLA and ForceVLA incorporate force and dynamics awareness to varying degrees. The benchmark itself is simulation-first, built on a pipeline that randomizes physical properties and uses closed-loop optimization to generate expert demonstrations. The key design choice is that the task REQUIRES physics inference — you cannot solve throwing by interpolating from demonstrations without understanding how mass and friction change the optimal trajectory. This is what separates PhyDEX from existing gripper-contact benchmarks. On integrity: the validation regime is reasonably honest but has the standard simulation-to-real gap caveat. The benchmark is self-generated — the authors built the pipeline, generated the data, and evaluated the models, so there's inherent circularity in the benchmark design itself. However, they do validate with real-world experiments on a humanoid robot, and the same failure patterns reproduce. The three VLA models are third-party (NVIDIA's GR00T, two recent publications), which reduces self-grading risk. No pre-registration, no independent replication, but the sub-3% results are damning enough that cherry-picking is unlikely — you don't cherry-pick results that make your benchmark look impossibly hard. The milestone question is where this gets interesting for the field. Right now, state-of-the-art VLA policies essentially treat physics as a black box learned implicitly from demonstrations. PhyDEX suggests that implicit physics learning is fundamentally insufficient for dynamic tasks. The concrete next threshold: a VLA model that achieves >30% success on PhyDEX's core throwing tasks without being given ground-truth physics. That would demonstrate genuine physical reasoning, not pattern matching. We're nowhere near it. The obvious experiment not run: fine-tuning a VLA model ON the PhyDEX dataset and measuring how much physics-informed training data closes the gap. The authors generated 35K expert trajectories — that's a training set, not just a benchmark. The honest read is probably (c): they're saving it for the next paper, or possibly (a) — the compute cost of fine-tuning GR00T-scale models is non-trivial, and the paper's value proposition is cleaner as a pure benchmark contribution.
The analogy
You're a bartender sliding a pint glass down a wet counter to a customer six stools away. You adjust force and angle based on how the glass feels and how slippery the bar is. Current VLA models can see the glass and name it in seventeen languages — but they have zero intuition about how hard to push. PhyDEX is the slippery bar that exposes the gap.
Bar slide
Bartender infers glass weight and bar friction from feel, adjusts push force and angle in real time
PhyDEX throw
Robot must infer object mass, CoM, and friction from observation, adjust throw trajectory to reach target beyond arm reach
Honest assessment
PhyDEX is a well-constructed provocation more than a solution: it proves current VLA models cannot do physics-dependent manipulation, isolates the bottleneck cleanly (inference vs. motor), and provides 35K trajectories to train against. The benchmark design is solid but self-referential — the same team built the test and ran the evaluation. The real payoff comes when other groups adopt PhyDEX and the fine-tuning experiments materialize. Right now this is a diagnostic tool, not a treatment.
What Is CLAIMS?
70
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
4/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine gap identified: dynamic physics inference in VLA models has never been benchmarked at this scale. The claim is 'VLAs are physically blind' — specific, testable, and backed by sub-3% success rates across three models.
The field fight
Can VLA models learn physics implicitly from demonstrations, or do they need explicit physics reasoning modules?
Side A
Scale demonstrations and model size; physics emerges implicitly from enough data
Held by: Foundation model scaling advocates, RT-2, Octo, GR00T
Side B
Implicit learning is insufficient for dynamic tasks; explicit physics inference or world models are required
Held by: World-model proponents, PhyDEX, physics-informed RL
Milestone tracker
Unlocks: Robots that can throw, slide, toss, and use dynamic manipulation in unstructured environments
Current
<3
% success
Best VLA on PhyDEX
Next
30
% success
Physics reasoning threshold
Unknown — likely requires architectural change, not just scaling
Verified Facts (6)
PhyDEX contains 35K trajectories (143 hours) across nine tasks and 100 scenes
Three VLAs evaluated: GR00T N1.7, DynamicVLA, and ForceVLA, all achieve success rates below 3%
Providing ground-truth physical properties roughly doubles success
Covering uniform, irregular, and dynamic-mass objects with static, multiple, and moving destinations
Most failures occur before the throw due to limited robustness of pretrained models
Experiments on a humanoid robot and in the real world show the same failure patterns
Practical Implications (4)
Teams building VLA-based robot policies should not assume pick-and-place success transfers to dynamic tasks — PhyDEX shows the gap is categorical, not incremental
The ~2× gain from oracle physics suggests that adding explicit physics estimation modules (mass, friction, CoM) to VLA pipelines could yield outsized returns relative to pure scaling
Warehouse and logistics robotics requiring throwing, sliding, or tossing should not rely on current foundation model approaches without task-specific physics fine-tuning
The 35K-trajectory dataset, once released, is a training resource — not just a benchmark — for teams working on physics-informed manipulation
More latest news
Read all