DexSeed Turns Cheap Human Hand Videos Into Robot Training Data by Expanding Grasps Across Objects and Locations

A pipeline that recycles human hand-object interaction datasets into diverse dexterous robot demonstrations, bypassing expensive teleoperation entirely.

Imagine you're a choreographer who has footage of a dancer performing one routine on one stage. You need that dancer to perform variations of the routine on fifty different stages, with different props, in costumes that change how their limbs move — and every variation has to be physically possible, not just visually plausible. That's the core problem DexSeed solves, except the dancer is a robot hand, the stages are different objects and table locations, and 'physically possible' means surviving a physics simulator without interpenetration or joint-limit violations. The committed claim: given a modest library of human hand-object interaction sequences, DexSeed can expand them into a diverse, physics-validated dataset of dexterous robot demonstrations sufficient to fine-tune a vision-language-action (VLA) model that follows language-specified grasp instructions — without any teleoperated robot data at all. This is not the first paper to retarget human hand motions onto a robot hand, but the expansion axis — varying objects, locations, and arm trajectories while preserving the grasp topology — is what's new. The architecture is a four-stage pipeline. First, source human sequences are organized by grasp mode and tagged with shared language descriptions. Second, InteractionMesh-based retargeting transfers the hand-object spatial relationships onto the robot hand morphology, preserving local contact geometry even when the object shape changes. Third, arm trajectory augmentation generates diverse pick-and-place paths to new table locations. Fourth, an RL-trained finger-control policy executes the retargeted grasps in simulation while the arm tracks the precomputed trajectories; only successful rollouts survive to become training data. Rendered visual observations, recorded joint actions, and grasp descriptions from survivors are used to fine-tune a VLA. On the ladder, the paper compares retargeting contact fidelity against prior retargeting baselines and shows improved contact preservation. For whole-arm execution, it reports more reliable rollout success than evaluated baselines, though the paper does not name a single dominant prior system with head-to-head numbers on a community benchmark. The baselines are reasonable but not the kind of named-SOTA-with-delta comparison that makes a result unambiguous. The evaluation includes both simulation metrics and real-world deployment on a physical robot hand, which adds credibility but is demonstrated qualitatively rather than with large-scale quantitative results. Integrity is mixed in useful ways. The simulation-to-real transfer is demonstrated but not stress-tested at scale. The RL finger policy grades its own homework — successful rollouts are self-selected — which is standard practice for demonstration generation but introduces survivorship bias into the training distribution. There's no pre-registration, no community benchmark for this exact task (because the task formulation is partly novel), and no independent replication. Code availability is indicated via the project page but not explicitly confirmed as open-source at time of review. The milestone that matters is VLA generalization breadth. Right now, the paper shows that a VLA fine-tuned on DexSeed-generated data can execute language-specified grasps in both simulation and real-world settings. The next concrete target is demonstrating this pipeline scales to hundreds of object categories and grasp modes with consistent sim-to-real transfer — roughly an order of magnitude expansion in object diversity. If that holds, the implication is that the robotics community's dependence on teleoperation for dexterous manipulation training data drops sharply. The obvious experiment not run: scaling the pipeline to a much larger human hand-object dataset (like DexYCB or OakInk at full scale) and measuring VLA performance as a function of dataset diversity. The honest read is (a) compute budget — running RL finger-control policies for thousands of expanded sequences is expensive — combined with (c) saving the scaling story for the next paper. The framework is clearly designed to scale; demonstrating it at full scale is the natural sequel.