Imagine you're at a crowded farmers' market and you say to a friend, "hand me that one" while vaguely gesturing toward a table of tomatoes. Your friend doesn't just follow your finger like a laser pointer — they combine the direction of your gesture, the words you used, the context of what's on the table, and a sense of how confident they are about each cue. If they're still unsure, they ask. MIGU gives a robot this same reasoning loop. The committed claim: MIGU is the first modular framework that propagates explicit geometric uncertainty from human pointing gestures (eye-finger ray direction plus depth noise) and fuses it with vision-language semantic priors via Bayesian inference to decide whether to act or ask for clarification. This is not a new grasping algorithm or a new VLM — it's a new decision-theoretic bridge between multimodal perception and manipulation planning. The architecture is modular and geometry-first. MIGU constructs a 3D geometric likelihood by modeling the eye-finger ray as an uncertain cone — accounting for hand-direction estimation error, viewing angle, and depth — then projects this onto candidate objects in the scene. Separately, a vision-language model (VLM) scores those same candidates on semantic plausibility given the spoken instruction. These two signals are combined through Bayes-inspired fusion: the VLM provides the prior, the geometric cone provides the likelihood, and the posterior belief over objects becomes the input to a behavior planner. That planner has a threshold: confident enough, proceed to mobile manipulation or tabletop task-and-motion planning; not confident enough, ask for clarification. The explicit uncertainty propagation is the load-bearing novelty — most prior work either ignores gesture noise or treats language and gesture as independent classifiers to be majority-voted. On the ladder, MIGU outperforms all evaluated baselines on a real-world benchmark the authors constructed. The abstract doesn't name specific prior methods or provide numerical deltas, which is a gap — we know it wins, but not by how much or against whom specifically. The ablation study confirms that the multimodal uncertainty modeling is the key contributor, not just the VLM or just the geometry alone. Without named baselines and numbers in the abstract, we're trusting the authors' summary. Integrity is mixed. The evaluation is on a real-world benchmark (physical robot, real scenes), which is stronger than pure simulation. The ablation design is sound — removing components to isolate contributions. But the benchmark appears to be author-constructed rather than a community standard, and there's no mention of pre-registration or independent replication. The project website (multimodal-instruction.github.io) suggests code or supplementary materials may be available, but the abstract doesn't confirm code release. The milestone question is about scaling this to messier, more dynamic environments. Right now MIGU works on structured tabletop and mobile manipulation tasks with a finite set of candidate objects. The next unlock is handling open-world scenes with dozens of unlabeled objects, occluded targets, and continuous (not discrete) referent spaces. That likely requires scaling the geometric likelihood model to handle partial visibility and integrating open-vocabulary VLMs that don't need a pre-defined candidate set. We're probably 2-4 years from that being robust outside lab conditions. The obvious experiment not run: multi-turn dialogue. MIGU can decide to ask for clarification, but the abstract doesn't describe what happens after the robot asks — does it update its belief with the new information and re-fuse? A full Bayesian update loop across multiple clarification rounds is the natural extension, and its absence suggests either it's the next paper or the belief-update math gets messy when you condition on the human's response to your question about their previous response. My read: they're saving it for the follow-up.
