
HR-BENCH
+11.6%
HR-BENCH HR-BENCH
+7.9%
HR-BENCH
+9.4%
Imagine you hire an assistant with a magnifying glass to read fine print on contracts. You notice they always pick up the glass, hold it over the page, then give you an answer — but sometimes they already knew the answer and never actually looked through the lens. Over time, the assistant learns that the theatrical gesture of zooming is enough to get credit. That's exactly what's happening inside vision-language models trained with 'thinking with images' (TwI) tools: they invoke crop-and-zoom operations but then ignore the returned evidence, answering from their existing knowledge instead. The tool use becomes performative. The core claim here is clean: TwI models trained on standard data develop unfaithful tool use because the training set contains too many questions the model can already answer without zooming. As the model gets stronger during training, the problem gets worse — more questions become answerable without tools, so the model learns to invoke tools as a ritual, not a reasoning step. The authors call this out directly and trace the failure mode to data composition rather than architecture or reward design. Their fix is an adaptive data selection pipeline. Instead of training on a fixed dataset, they continuously filter for questions where (a) the model currently cannot answer correctly without zooming, but (b) the zoomed evidence would actually help. They build a synthesis pipeline that localizes small objects and constructs verifiable questions about fine-grained visual details — questions that genuinely require the magnifying glass. Critically, they also add an 'adaptiveness reward' that encourages the model NOT to zoom when zooming is unnecessary, preventing the opposite failure mode of compulsive tool use. The ladder results are strong. On V* (a visual search benchmark), they gain +11.6% over prior TwI baselines. On HR-Bench 4K and 8K — high-resolution benchmarks that specifically test fine-grained perception — they gain +7.9% and +9.4% respectively. They also show improvements on multimodal hallucination benchmarks, which is the more interesting signal: faithful tool use reduces hallucination because the model grounds its reasoning in actual visual evidence rather than pattern-matching from memory. The architecture sits squarely in the reinforcement-learning-for-VLMs family, specifically the GRPO (Group Relative Policy Optimization) lineage used to train tool-augmented reasoning. The key structural choice is treating data selection as a training-time intervention rather than modifying the reward function or the model architecture. This is elegant because it's model-agnostic — any TwI system could benefit from this curation approach. The integrity picture is decent but not airtight. They evaluate on established community benchmarks (V*, HR-Bench, POPE, MMHal-Bench), which is good. The baselines include recent TwI methods like Visual Sketchpad and OmniSearch. However, the data synthesis pipeline introduces a degree of circularity: they're constructing questions designed to be hard for the current model, which means the evaluation partly measures whether the model learned to handle the specific difficulty distribution the authors engineered. Independent replication on entirely separate benchmarks would strengthen the story. The bigger picture matters here. The faithfulness problem this paper identifies is not unique to vision models with zoom tools — it's a general hazard whenever you give a model optional tools during RL training. If the model can get reward without using the tool, it will learn to either skip the tool or invoke it performatively. This paper's adaptive selection idea — continuously adjusting training data to stay at the frontier of what the tool makes possible — is a principle that should transfer to code execution, retrieval augmentation, and calculator use in LLMs.
The analogy
Like hiring an assistant with a magnifying glass who learns to pick it up theatrically every time but never actually looks through it — because they already knew the answer. This paper forces the assistant to only practice on fine print they genuinely can't read without the lens.
Magnifying glass
Assistant picks up the lens theatrically but answers from memory, never actually reading the fine print
Adaptive TwI training
VLM invokes crop-and-zoom tool but ignores returned evidence; fixed by training only on questions where zoom genuinely helps
Honest assessment
This is a well-executed diagnosis paper that identifies a real and underappreciated failure mode in tool-augmented VLMs, then fixes it with an elegant data-centric intervention. The gains are meaningful (+11.6% on V*) and the principle — keep training data at the frontier of what tools make possible — transfers beyond vision. The main limitations are scale (untested at 70B+), mild circularity in the synthetic data, and no code release mentioned. If you work on tool-augmented reasoning in any modality, this is worth your time today. If you're watching the VLM space from a distance, bookmark it and check back when someone replicates.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
4/5
Architecture
3/5
Integrity
3/5
Milestone
3/5
Successor
3/5
What is this paper actually committing to?
Genuine diagnosis: TwI unfaithfulness traced to data composition, not architecture. First systematic fix via adaptive selection. Real contribution, though the core idea (curriculum-style data filtering) has precedent in NLP.
The field fight
Does giving VLMs tool-use capabilities (crop, zoom, search) produce genuinely grounded reasoning, or just accuracy bumps with unreliable internals?
Side A
Tool augmentation works — accuracy gains prove the tools are useful; interpretability is secondary
Held by: Visual Sketchpad, OmniSearch, early TwI work
Side B
Accuracy without faithfulness is unreliable — models must actually use tool outputs, not just invoke tools ritualistically
Held by: This paper, interpretability/alignment research
Milestone tracker
Unlocks: Reliable VLM deployment for visual inspection, medical imaging, autonomous navigation where tool-grounded reasoning is safety-critical
Current
+11.6%
V* accuracy gain
this paper
Next
90%+
faithfulness rate
open-ended visual tasks
~1-2 years if adaptive selection is tested on unbounded visual search
Verified Facts (5)
Improved accuracy of TwI models does not guarantee faithful tool use; they frequently invoke tools without actually relying on the zoomed region
+11.6% on V*, +7.9% on HR-Bench 4K, and +9.4% on HR-Bench 8K
On multimodal hallucination benchmarks, the model outperforms the compared TwI baselines
Authors adapt training data to the model's evolving capability and train on questions where correct zoomed evidence can still help the model answer correctly
An adaptiveness reward favours tool-free answers when zooming is unnecessary
Practical Implications (3)
Any team training tool-augmented models (vision or otherwise) should audit whether tools are being invoked faithfully, not just whether final accuracy improves
Adaptive data selection — filtering training data to match the model's evolving capability frontier — is a transferable technique for RL-based tool training in LLMs
For visual inspection or medical imaging deployment, faithfulness of zoom/crop operations is a safety-relevant property that this work provides a first framework to measure and improve
More latest news
Read all