
Read the paper
The Illusion of Grounding: Do Vision-Language Models Weigh Retrieved Text and Images Like an Ideal Observer?
on GitHub
ACROSS MODELS
+92.5%
ACROSS MODELS
-94.2%
ACCURACY FALLS
+41.3%
ACCURACY FALLS
-59.9%
Imagine you're a student taking an open-book exam. You can see the question clearly on the board, and you have your textbook open in front of you. The rational strategy: read the board first, check the book if you're unsure. But what if, every time the book said something different from the board, you just went with the book — even when the board was perfectly legible? That's what most retrieval-augmented vision-language models are doing, and this paper is the first to measure the problem with Bayesian precision. The committed claim: VLMs don't perform anything resembling ideal Bayesian cue combination between text and image evidence. The authors derive a formal framework from ideal-observer theory — the same math that governs how the human brain merges sight and hearing — and use it to measure exactly how much each model over-weights or under-weights each modality. They introduce two crisp failure modes: Knowledge Inertia (the model trusts text too much relative to its own image evidence) and Sensory Capture (the reverse). This isn't just naming a problem; it's handing the field a thermometer. The benchmark, GIB-200K, is built from COCO and Visual Genome with controlled stress conditions: hard-negative retrieval noise scaled from 1 to 20 documents, visual degradation via Gaussian blur, motion blur, and low-resource crops, plus counterfactual flip pairs that swap the correct answer in the retrieved text. The evaluation split is frozen at 2,000 items with a fixed seed. All four models tested — two LLaVA variants, InternVL3, and one more — score 92.5–94.2% on clean inputs. Under adversarial conditions, accuracy collapses to 41.3–59.9%. That 50-point gap is the story. The weight estimates tell you which way each model falls. Both LLaVA models exhibit Knowledge Inertia: their text-to-image weight ratios (w_T/w_I) are 1.229 and 1.322, meaning they trust retrieved text 23–32% more than their own visual processing warrants. InternVL3 flips the bias with a ratio of 0.680, over-weighting the image. The 'Grounding Illusion Score' — measuring how much of clean accuracy is not actually grounded in image evidence — reveals that between 34.8% and 71.4% of correct clean answers are abandoned when the retrieval document lies. Translation: a huge fraction of what looks like visual understanding is really just parroting text. Methodologically, the measurement-error-corrected regression is the load-bearing move. The authors verify that their estimator recovers known synthetic weights before applying it to real model outputs, which addresses the most obvious objection — that noisy accuracy proxies would corrupt the weight estimates. The counterfactual flip decomposition separates genuine modality bias from concept priors, preventing the analysis from confusing 'this model likes dogs' with 'this model trusts text.' These are careful choices that elevate the paper above a typical benchmark drop. The training-free consistency check — comparing answers with and without retrieval — is the practical contribution. It trades coverage for accuracy under misleading documents, essentially flagging cases where the model might be parroting bad text. But the authors are honest: it falls back to the image when retrieval is suspect, and that fallback fails when the image is heavily degraded. There's no free lunch. The paper tests only four open models on a single frozen split of 2,000 items. Closed-API models (GPT-4o, Gemini) are absent, which limits the ladder. The deeper question this paper enters is whether RAG-augmented VLMs are doing multimodal reasoning at all, or just routing around image understanding whenever text is available. If 71% of a model's correct answers evaporate when the text lies, the 'vision' in 'vision-language model' is more decorative than structural. The formal framework — grounding illusion scores, Bayesian weight estimation — gives the field a shared measurement tool. The next step is obvious: does training with adversarial retrieval noise shift those weights toward the ideal observer?
The analogy
You're a student with a textbook open during an exam. The question is on the board — you can see it clearly. But every time the textbook contradicts the board, you go with the textbook, even when the board is perfectly legible. That's Knowledge Inertia: trusting the retrieved document over your own eyes.
Open-book exam
Student can see the board clearly but defaults to whatever the textbook says, even when it's wrong
RAG-VLM inference
Model has image evidence but over-weights retrieved text, abandoning correct visual answers when documents lie
Honest assessment
This paper delivers a genuinely useful diagnostic instrument — Bayesian cue-combination weights for VLMs — that the field didn't have before. The finding that up to 71% of correct answers evaporate when retrieved text lies is striking and well-measured. The scope is limited: four open models, one frozen split, no training intervention. But as a diagnostic paper, it doesn't need to fix the problem — it needs to prove the problem is real and measurable, and it does that convincingly. ML engineers building RAG-VLM systems should care today; the broader field should care when someone pairs this framework with training interventions.
What Is CLAIMS?
70
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
3/5
Integrity
4/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine novelty: first Bayesian cue-combination framework applied to VLMs with formal definitions (Knowledge Inertia, Sensory Capture, Grounding Illusion Score). Diagnostic, not architectural — a measurement tool the field lacked.
The field fight
Do multimodal models genuinely integrate visual evidence, or do they route around it when text is available?
Side A
VLMs perform real multimodal reasoning; scaling and better training data will close remaining gaps
Held by: GPT-4o, Gemini, scaling-optimist camp
Side B
Current VLMs are text-dominant; image understanding is shallow and collapses under adversarial conditions
Held by: Hallucination benchmarks (POPE, MME), this paper
Milestone tracker
Unlocks: VLMs that genuinely ground answers in visual evidence rather than parroting retrieved text
Current
1.3
w_T/w_I
LLaVA text over-weight
Next
1.0
w_T/w_I
Ideal observer parity
~1-2 years if adversarial training experiments are run on GIB-200K
Verified Facts (9)
GIB-200K contains 200,000 image-question items built from COCO and Visual Genome
Hard-negative retrieval noise scaled from 1-20 documents
Clean accuracy across 4 models: 92.5%-94.2%
Adverse accuracy falls to 41.3%-59.9%
LLaVA models text-to-image weight ratios: w_T/w_I = 1.229 and 1.322
InternVL3 w_T/w_I = 0.680
Between 34.8% and 71.4% of correct clean answers abandoned when document lies
10 pages, 5 figures, 11 tables
Code and data at https://github.com/Basavarajearlybird/GIB-200K and https://doi.org/10.5281/zenodo.24320174
Practical Implications (4)
RAG-VLM system builders should assume their models will parrot bad retrieved text even when the image clearly contradicts it — build retrieval-quality safeguards upstream
The training-free consistency check (answer with vs without retrieval) is deployable today as a cheap hallucination flag, trading coverage for accuracy
The Grounding Illusion Score can be used as an audit metric for any VLM pipeline to quantify how much visual grounding is real vs illusory
Model selection matters: InternVL3's image-biased weights make it more robust to bad retrieval but more fragile under image degradation — pick based on your failure mode
More latest news
Read all