Imagine you're a reference librarian. Someone asks about a book that hasn't been published yet. With no resources in front of you, you'd probably shrug and say 'I don't know.' But hand you a stack of vaguely related catalogs and suddenly you feel obligated to give an answer — any answer — even if none of those catalogs actually contain the book. That's the core finding of DeclineCraft: giving language models more retrieval opportunities doesn't make them more honest. It makes them more confident about things they shouldn't be confident about. The benchmark is cleverly constructed around Minecraft — specifically, feature additions after a corpus cut-off date. Because Minecraft updates on a known schedule, you can generate questions about features that provably didn't exist when a model was trained, and you can audit whether the model could have seen the information during pre-training. This isn't a static test set that models can memorize into oblivion; later cut-off dates naturally produce new unanswerable questions for future models, giving the benchmark a built-in shelf life that most abstention tests lack. The headline numbers are damning for the 'just add more retrieval' school of RAG design. Single-shot retrieval — one round of document fetching — actually helps models decline appropriately, raising the Correct Abstention Rate in 38 of 40 tested configurations. But the moment you switch to iterative RAG, where the model self-assesses its context and can request up to two more retrieval rounds, CAR collapses. In 30 of 40 configurations, iterative RAG significantly lowers correct abstention versus single-shot, with a mean drop of 46.4%. Answer accuracy barely budges (68.5% to 69.3%), meaning the extra rounds don't help with answerable questions either — they just erode the model's ability to say 'I don't know.' The thinking-mode results are particularly unsettling. When chain-of-thought reasoning is enabled on models that support it, 8 of 13 models show significantly lower CAR — by an average of 14 percentage points. But the failure mode isn't just 'answers anyway.' The models shift from acknowledged uncertainty ('I'm not sure this exists') to premise denial ('This feature doesn't exist in Minecraft'), which is actively worse: you get a confident, authoritative wrong answer instead of a hedge. The model reasons itself into a corner and doubles down. Scaling doesn't fix this. Larger models answer answerable questions more accurately (Spearman ρ = +0.60 with total parameters), but neither model size nor release date shows a significant association with correct abstention rate. This means the 'just scale it up' assumption — that bigger models will naturally get better at knowing what they don't know — has no support in this data. The 27 models tested span a wide range of sizes and architectures, and the abstention problem is orthogonal to capability. The retrieval pipeline analysis adds nuance. Adding query rewriting and reranking to hybrid dense-and-sparse retrieval further lowers CAR for all 12 models tested. Every component that makes retrieval 'better' at finding relevant-looking documents makes the model worse at recognizing when the answer isn't there. The pipeline optimizations that help with answerable questions are actively counterproductive for unanswerable ones. This is a fundamental tension in RAG design that the field hasn't resolved. DeclineCraft's contribution is methodological more than algorithmic — it doesn't propose a fix, it builds a rigorous measuring stick. The Minecraft domain is narrow but the problem it exposes is universal: deployed RAG systems face unanswerable queries constantly, and the standard engineering response (more retrieval, better retrieval, chain-of-thought reasoning) makes the core failure mode worse, not better. Any team shipping RAG to production needs to grapple with this.
