You know that moment when you've written a decent email, then start editing it, and somehow end up with something worse? That's the core problem this paper formalizes. In retrieval-augmented QA, you have a draft answer — maybe from a closed-book LLM — and you can revise it using retrieved documents. But revision isn't free: sometimes it injects noise and flips a correct answer to wrong. The question is whether you can learn to predict, before committing, whether revision will help or hurt. The committed claim: you can train a binary scorer on paired (draft, revised) outcomes — what the authors call 'recoverability' — that outperforms a scorer trained only on draft correctness at deciding when to revise. Across 25,870 held-out open-domain questions and three revision setups (sparse retrieval, dense retrieval, LLM-based revision), the recoverability scorer achieves 0.23–0.68 accuracy points more on average at development-selected thresholds than the draft-confidence baseline. The gain is statistically significant only for the dense-retrieval setup across all seeds. The architecture is straightforward: take Llama (and OLMo as a second model family), generate draft answers, retrieve documents, revise, then grade both versions with the same correctness judge. This creates paired labels — repair (bad→good), harm (good→bad), and the gap to an oracle that always picks the best option. A logistic-style scorer is trained to predict the paired effect before revision occurs. The key structural insight is that draft confidence alone doesn't capture the signal — you need the paired outcome because some wrong answers are recoverable by retrieval and some correct answers are fragile. Here's where it gets interesting — and honest. The policy closes more than a third of the oracle gap on average, but still applies 38–46% of harmful revisions. That's a real limitation plainly stated. The always-revise baseline is beaten, but almost half the damage still leaks through. The paper doesn't paper over this. The most important finding might be the negative result buried in the last paragraph of the abstract. When a draft-free standard-RAG answer is available as an alternative, simply choosing between the draft and that RAG answer — no revision step at all — beats the revision policy by about two accuracy points for Llama and four for OLMo. Adding revision as a third option yields no significant gain. This means the elaborate revision machinery the paper analyzes may be less useful than a simpler routing decision. The evaluation is careful within its scope: three retrieval setups, two model families, multiple seeds, significance testing across training runs. The 25,870 held-out questions come from open-domain QA benchmarks. The authors distinguish between what's significant across seeds and what isn't, which is more discipline than most papers in this space show. But validation is self-contained — same team, same pipeline, same correctness judge grading both sides. The broader lens here is the growing recognition that RAG pipelines need decision points, not just bigger retrievers. The field is moving from 'retrieve-then-read' monoliths toward modular systems that reason about when retrieval helps. This paper contributes a clean formalization (recoverability) and an honest accounting of when revision is and isn't worth the trouble — then somewhat undermines its own premise by showing simpler routing wins.
