ALL-LAYER ADAPTER ROUTES
-51%
AGAINST MOST
+7%
EFFECT COMES
-86%
Imagine you're watching two security monitors side by side. You can clearly see the same person on both screens — same jacket, same gait. But when someone asks 'Is that the same person?', your mouth says 'I don't know.' Your eyes did the work. Your brain processed it. The answer is sitting right there in your visual cortex. But the pathway from perception to speech is broken. That is, almost exactly, what happens inside vision-language models when you ask them to compare objects across two images. The committed claim: VLMs' cross-image comparison failures are not perception failures — they are readout failures caused by a small, identifiable set of attention heads (16-32) that carry comparison information but fail to route it to the output token. Training only the query and key projections of these heads (0.02% of parameters) recovers 25-28 accuracy points on semantic correspondence benchmarks. This is a mechanistic interpretability result with an immediately useful repair. The paper works across three model families (Qwen3-VL at 2B-32B, InternVL3.5-8B, Gemma-3-12B) and three benchmarks spanning exact matching, instance correspondence, and semantic correspondence. The diagnostic is clean: an answer-position probe finds 15-25 accuracy points sitting in the hidden states that never reach the output. Steering along this probe's direction does no better than random — meaning the information is present but not causally connected to the model's speech. The authors then identify which heads matter by redistributing cross-image attention mass: knocking out just 16-32 heads drops accuracy 15-32 points on HPatches, while matched-attention random heads cost at most 7. The bottleneck is real, specific, and consistent across families. The repair is where this gets practically interesting. Training only the Q/K projections of these identified heads — a 0.02% parameter intervention — raises SPair-71k accuracy by 28.3 points on Qwen3-VL-8B and 25.1 on InternVL3.5-8B. This beats both layer-matched and attention-matched control groups, and transfers to HPatches and BLINK without further tuning. When the authors train an unconstrained all-layer adapter instead, 21-51% of its gain routes through these same heads' attention, and 83-86% of that effect comes from changing what the heads read rather than their weights. The model's own gradient-based training rediscovers the same bottleneck. The integrity story is unusually careful for a mechanistic interpretability paper. Every decision rule was frozen before execution, every number recomputed by independent verification scripts — the authors state this explicitly. They test on three distinct benchmark types, compare against multiple control conditions (random heads, attention-matched heads, layer-matched heads), and show consistency across three model families spanning 2B to 32B parameters. The two-failure-point taxonomy (exact copies compared but not read out; semantic correspondences present in visual tokens but never reaching the answer) is clean and falsifiable. The ladder position is strong within the emerging field of VLM mechanistic interpretability but needs context. This is not competing with dedicated correspondence methods like DKM or ASpanFormer — it's explaining why general-purpose VLMs fail at a task they should handle, and offering a minimal fix. The 28.3-point SPair improvement is against each model's own baseline, not against specialized systems. The contribution is diagnostic precision, not SOTA correspondence. The obvious next experiment is scaling this analysis to larger models (70B+) and to proprietary systems. The authors stopped at 32B. The honest read: compute constraints. Running the full ablation suite — probing, attention redistribution, targeted fine-tuning, unconstrained adapter comparison — across multiple benchmarks for a 70B model is expensive. Whether the same 16-32 heads scale or whether larger models distribute comparison more broadly is the question that determines if this is a universal architectural property or a mid-scale artifact.
The analogy
You're watching two security monitors side by side. You can clearly see the same person on both screens. But when someone asks 'Is that the same person?', your mouth says 'I don't know.' Your eyes did the work — the pathway from perception to speech is broken. That's exactly what happens inside VLMs during cross-image comparison: the visual system computes the answer, but a small bottleneck of attention heads fails to relay it to the output.
Security guard
Sees the same person on two monitors but can't verbalize the match — perception-to-speech pathway is blocked
VLM readout
Hidden states contain correct cross-image comparison but 16-32 attention heads fail to route it to the answer token
Honest assessment
This is a clean mechanistic interpretability result with an unusually disciplined experimental design. The two-failure-point taxonomy is falsifiable, the multi-family consistency is convincing, and the 0.02%-parameter repair is immediately useful for practitioners fine-tuning VLMs for visual reasoning. The main limitation is that the ladder is entirely internal — we don't know how repaired VLMs compare to specialist correspondence methods, so the practical ceiling is unknown. The 70B+ gap is real but forgivable given compute costs. If you work on VLMs and wonder why they fail at seemingly simple visual comparisons, this paper gives you the most precise answer currently available.
What Is CLAIMS?
80
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
4/5
Integrity
5/5
Milestone
4/5
Successor
4/5
What is this paper actually committing to?
Genuine first in localizing cross-image comparison failure to specific attention heads with a falsifiable two-failure-point taxonomy. Not just diagnosis — the 0.02%-parameter repair makes it actionable.
The field fight
Are VLM visual reasoning failures caused by insufficient visual encoding or by broken information routing from vision to language?
Side A
Visual encoders need better training data and architecture; failures are perceptual
Held by: Scaling-first approaches, contrastive pre-training work
Side B
Visual information is already present in hidden states; the problem is internal routing and readout
Held by: Mechanistic interpretability, this paper, circuit-level analysis work
Milestone tracker
Unlocks: If bottleneck holds at scale, universal minimal repair for VLM cross-image reasoning without full fine-tuning
Current
32B
params
Largest model analyzed
Next
70B+
params
Universal bottleneck test
~1-2 years, compute-gated
Verified Facts (8)
An answer-position probe exceeds the calibrated output by 15 to 25 points in every family
Steering along the probe's direction moves no more answers than random directions
Redistributing the cross-image attention of 16-32 heads removes 17-22 points of synthetic and 15-32 points of HPatches accuracy, against at most 7 for random or attention-matched heads
Training only these heads' query and key projections raises SPair-71k accuracy by 28.3 points (Qwen3-VL-8B) and 25.1 points (InternVL3.5-8B)
An unconstrained all-layer adapter routes 21-51% of its gain through the attention of the same heads, against at most 7% for random heads
83-86% of the effect comes from changing what they read rather than their weights
Every decision rule was frozen before its run, and every reported number is recomputed by independent verification scripts
Tested on Qwen3-VL (2B-32B), InternVL3.5-8B, and Gemma-3-12B
Practical Implications (4)
VLM fine-tuning for cross-image tasks can target just 16-32 attention heads' Q/K projections instead of full model — 0.02% of parameters with 25-28 point gains
Diagnostic: if your VLM fails at visual comparison, probe hidden states first — the answer may already be there, just not routed to output
The identified heads transfer across benchmarks (SPair → HPatches → BLINK), suggesting a single targeted fine-tune generalizes
Unconstrained adapters spontaneously route 21-51% of their gain through these same heads — gradient-based training already knows about this bottleneck
More latest news
Read all