A Dataset That Pairs What Humans Look At and Say With What Robots Actually Do

HEIR fuses gaze, speech, and egocentric video with robot trajectories to study whether human intent signals improve long-horizon household tasks. Dataset paper, not a new model — but the right scaffold matters.

Imagine you're training a new employee by letting them shadow you while you narrate your work. They see what you see (your camera view), hear what you say ('grab the blue mug'), and a tracking system records where your eyes land. Then they watch the robot arm actually execute each subtask from its own perspective. That two-viewpoint, multi-signal recording is exactly what HEIR builds — a paired dataset where human intent channels (speech, gaze, egocentric video) are time-aligned with robot observation and teleoperated execution traces across long-horizon household tasks. The committed claim: HEIR is the first dataset that jointly captures human speech, measured gaze, and egocentric video alongside robot navigation-manipulation trajectories in a multi-episode, multi-subtask household interaction setting. This isn't a new model architecture — it's a resource paper arguing that the field's bottleneck is data infrastructure, not algorithms. The bet is that aligning human intent signals with robot execution traces in a single unified corpus will let researchers study intent grounding, execution planning, and task completion without stitching together mismatched datasets from different labs. Where does this sit on the ladder? The nearest predecessors are datasets like ALFRED (language-grounded household tasks in simulation), TEACh (human-human dialog for embodied tasks), and Ego4D (massive egocentric video, no robot execution). HEIR's differentiation is the simultaneous capture of gaze + speech + ego-video on the human side and observation + trajectory on the robot side, with multi-episode structure preserving successive requests within a session. None of the named predecessors bundle all five signal channels. The paper runs offline experiments showing that adding human-centric context (gaze, speech) improves model performance on intent grounding and planning tasks, plus real-robot experiments on execution. But the baselines compared are relatively standard VLM and planning architectures evaluated on their own new benchmark — there's no head-to-head against an established leaderboard metric. Architecturally, HEIR is agnostic — it's a data resource, not a method. The offline experiments evaluate transformer-based vision-language models and planning modules, but the dataset is designed to be consumed by any architecture. The key structural choice is the pairing mechanism: temporally aligned multi-modal streams with human-authored annotations bridging intent and execution. Hardware-wise, the capture rig combines eye-tracking glasses, egocentric cameras, and teleoperation interfaces in real household environments — not simulation. Integrity is mixed. The validation is self-referential by nature: the authors created the dataset, designed the benchmark tasks, and ran the evaluations. That's standard for resource papers, but it means every performance number is on their own test set with their own splits. No independent group has used HEIR yet. The real-robot experiments add credibility — they're not just running numbers in sim — but sample sizes and environment diversity are limited to 'naturalistic household environments' without specifying how many homes, how many users, or how many total hours of data. The demo and dataset are stated to be available online, which is the right move. The milestone question for this subfield is clear: when does a robot in a real kitchen reliably execute a 5-step task chain from a single natural-language request plus gaze context, achieving >80% completion across unseen homes? HEIR doesn't hit that bar — it provides the training data infrastructure and shows proof-of-concept gains. The gap between 'additional context helps on our benchmark' and 'deployed household robot understands you' remains large, probably 3-5 years of iteration on both data scale and model architecture. The obvious experiment not run: scaling. How does performance change as HEIR grows from its current size to 10× or 100× the sessions? The authors almost certainly didn't run this because data collection with real eye-tracking hardware and teleoperation is expensive and slow — this isn't something you can crowdsource on Mechanical Turk. The second missing experiment is cross-environment generalization: training on HEIR homes, testing in completely unseen layouts. That's the real test of whether the intent signals generalize, and it likely wasn't run because the capture rig limits the number of environments.