LLMs Guess When They Should Admit They Don't Know — and They're Wrong Half the Time

When retrieved passages answer at the wrong granularity, language models confidently guess instead of abstaining. The guesses are wrong ~50% of the time, and simple fixes don't work.

Imagine you ask a librarian for a book by a specific author and they hand you a book from the right shelf — same genre, same era — but by a different writer. They don't say 'I couldn't find the exact one'; they just hand you the wrong book with a confident smile. That's what this paper catches language models doing when their retrieval system returns text that's accurate but at the wrong level of specificity. The core claim: retrieval-augmented LLMs systematically fail to recognize when a retrieved passage answers a broader category rather than the specific entity the question asks about. The authors name this failure mode CLADE — Category-Level Answering from Deficient Evidence. When a question asks about familial hyperlipidemia but the passage only mentions hyperlipidemia (the broader family), models answer confidently instead of flagging the gap. Abstention drops 13.8 percentage points compared to truly uninformative passages, and the resulting answers are wrong roughly half the time. The experimental architecture is straightforward and well-designed. Using PrimeKG (a biomedical knowledge graph) and GloBI (an ecological species-interaction database), the authors construct question–passage pairs at three levels: exact-match passages, family-level (coarse) passages, and uninformative passages. A counterfactual swap test — replacing the exact disease name with its broader family name while holding everything else constant — isolates the causal mechanism precisely. That swap alone drops abstention by 17.4 points. This isn't a vague correlation; it's a clean manipulation. The 'simple fixes don't work' finding is the most practically important result. Confidence filtering, scaling to a larger model, and explicit abstention instructions all fail to resolve CLADE. The models are not uncertain — they are confidently wrong. This distinguishes CLADE from standard hallucination, where calibration tricks sometimes help. The failure replicates across two entirely different domains (biomedicine and ecology), which raises confidence that this is a structural property of how LLMs process hierarchical information, not a quirk of one dataset. The ontology-rescue experiment is where the paper gets honest about its own limits. Given the correct family, an ontology lookup includes the true answer 98.5% of the time — the ontology works. But the bottleneck is upstream: string matching identifies the correct family on the retrieved page only 19.8% of the time. Letting the model itself choose the family pushes this to 30.0%, and end-to-end recovery from coarse passages rises from 10.0% to 26.8%. These are real but modest numbers. The ontology is the right tool, but getting the LLM to hand off to it reliably remains unsolved. The paper's position in the field is clear: it argues that 'context sufficiency' — the binary yes/no framing used by prior work on when models should abstain — misses a structured middle ground. Some insufficiency is resolvable if you route through external knowledge structures. The contribution is taxonomic (naming and measuring CLADE) and diagnostic (showing that standard mitigations fail), not yet prescriptive. The authors do not claim to solve the problem; they claim to have identified and characterized it. For anyone building RAG systems in domains with hierarchical ontologies — medicine, biology, law, taxonomy-heavy enterprise search — this paper is a direct operational warning. Your retrieval system may be returning passages that look relevant and are relevant, just at the wrong grain. Your model won't tell you. It'll guess, and it'll be wrong half the time.