
Read the paper
How VLMs Plan What's Next
on Project page
Imagine you're reading a room at a party. Your eyes flick from person to person, but you never introduce the same guest twice. You keep a mental cursor — 'I've covered the left side of the room, now I move right.' You don't consciously plan this; your brain's social firmware handles the routing. This paper shows that vision-language models have evolved the same firmware, and the authors cracked it open to show exactly which attention heads do what. The committed claim: when a VLM like Qwen3-VL lists objects in an image, it uses a four-stage internal circuit — binding heads that match text tokens to image regions, a cursor mechanism that tracks progress through already-described regions, a select step that picks the nearest unwritten region (or decides to stop), and gaze heads that read the selected region and hand its concept to the language decoder. This is not a behavioral observation. The authors identify specific attention heads, name them, and validate each stage with causal interventions. The mechanistic evidence is the load-bearing part. The team uses causal patching — swapping activations between runs to test whether a specific head is necessary and sufficient for a specific subtask. Binding heads are shown to both resolve text references against image patches and maintain the progress cursor. Gaze heads are shown to extract the concept from the selected region. The paper tests these mechanisms against alternative hypotheses (e.g., that the model uses spatial heuristics rather than a true cursor) and rejects them. This is the kind of interpretability work where the authors don't just find correlations — they intervene. The ladder here is against prior mechanistic interpretability work in VLMs, which is sparse. Most existing mech-interp focuses on text-only transformers (Elhage et al.'s induction heads, Olsson et al.'s work on in-context learning circuits). Extending this to the cross-modal binding problem — how does the model ground text in image patches and plan a sequence over them — is genuinely new territory. No prior work has demonstrated a four-part planning circuit with per-head causal validation in a production-scale VLM. The validation regime is solid but bounded. The primary task is synthetic visual retrieval — colored shapes in controlled layouts — which gives clean causal signals but raises the question of whether these circuits survive in the wild. The authors address this directly: they show the same heads activate on natural photographs, across input formats, and across languages (a strong generalization check). The mechanism also appears at every model size from 2B to 32B parameters, which is the kind of scaling consistency that makes a circuit claim credible rather than artifactual. Code and demos are available at plan.baulab.info. The milestone question is about depth: this paper characterizes planning over a relatively simple task (enumerate objects). The obvious next frontier is whether the same circuit structure — bind, cursor, select, gaze — generalizes to multi-step reasoning tasks with richer dependencies (e.g., spatial relationships, counting, compositional questions). The authors gesture at this ('how models plan over complex multi-step tasks with long rollouts') but don't demonstrate it. That's the experiment to watch. The experiment the authors did not run, and the one that would change everything: testing whether you can surgically edit these heads to fix known VLM failure modes — object hallucination, repetition, missed objects. If the cursor mechanism is real, breaking it should produce hallucinations, and fixing it should eliminate them. The authors likely know this but are saving it. The interpretability-to-intervention pipeline is the prize, and this paper lays the groundwork without crossing the line into it.
The analogy
You're reading a room at a party, introducing guests one by one. You never re-introduce someone — your brain keeps a mental cursor tracking who's been covered and picks the nearest un-introduced person next. This paper shows VLMs have built the same firmware: bind faces to names, track the cursor, pick the next guest, read them out.
Party introductions
Scan the room, track who's been introduced, pick the nearest un-introduced guest, name them
VLM object listing
Binding heads match text to image regions, cursor tracks progress, select picks next unwritten region, gaze heads read it out
Honest assessment
This is a clean, well-validated piece of mechanistic interpretability work that opens genuinely new territory: nobody has previously demonstrated a multi-stage planning circuit with causal evidence in a production VLM. The synthetic task is narrow but the generalization checks (natural photos, multiple languages, three model scales) are what make it credible. The real payoff — using this circuit understanding to surgically fix VLM failure modes — is not demonstrated here, but the groundwork is solid. If you work on VLMs, multimodal AI, or interpretability, this paper changes how you think about what's happening inside these models.
What Is CLAIMS?
73
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
4/5
Architecture
3/5
Integrity
4/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine first: a four-stage planning circuit with per-head causal validation in a production VLM. No prior work has shown binding, cursor, select, and gaze as separable, causally validated stages in a cross-modal model.
The field fight
Can we mechanistically explain VLM behavior at the attention-head level, or are these models too complex for circuit-level interpretability?
Side A
Probing and behavioral tests are sufficient; circuit-level claims in large models are fragile and don't generalize
Held by: Scaling skeptics, BERTology-era probing camp
Side B
Causal interventions can identify real circuits in large models; mech-interp scales if you pick the right tasks
Held by: Anthropic (induction heads), Bau Lab (this paper), Neel Nanda
Milestone tracker
Unlocks: Targeted surgical fixes for VLM hallucination, repetition, and missed-object failure modes
Current
4-stage
circuit
Object enumeration in Qwen3-VL
Next
compositional
reasoning tasks
Multi-step visual QA with branching
~1-2 papers if circuit structure generalizes
Verified Facts (7)
The mechanism has four main parts: binding heads, cursor/progress signal, select step, and gaze heads
Validated through causal patching in Qwen3-VL models on a synthetic visual retrieval task
Tested against alternative hypotheses
Mechanism holds across input formats, natural photographs, and languages
Binding, relay, and gaze heads appear at every model size from 2B to 32B
Project available at https://plan.baulab.info/
Authors: Rohit Gandikota, Arnab Sen Sharma, Adam Belfki, David Bau
Practical Implications (4)
VLM debugging: if hallucinations or object repetition stem from cursor/binding head failures, targeted interventions become possible without full retraining
Model editing: knowing which heads perform binding vs gaze enables surgical modifications for specific failure modes
Architecture design: the four-stage circuit suggests that VLM planning is not diffuse — concentrated head-level structure could inform more efficient attention patterns
Evaluation: the cursor mechanism suggests new diagnostic tests — perturb the cursor and check if repetition/omission rates change predictably
More latest news
Read all