Imagine you're packing for a move and your friend offers to 'summarize' the contents of each box on a label. They get the gist right — 'kitchen stuff' — but they leave out the fact that your grandmother's irreplaceable teacup is in there. When you unpack and ask 'did I keep my kitchen stuff?' the answer is yes. When you ask 'where's the teacup?' it's gone. That's the information bottleneck in context compression: QA metrics say the summary is fine because the answer to the obvious question is still retrievable, but the specific, fragile, minority-view evidence has been quietly destroyed. The committed claim: existing QA-based evaluations systematically overestimate compression faithfulness, and KEEP-Bench — 4,102 cases across three evidence structures (plausible reference resolutions, explicit qualifiers, minority perspectives) — exposes the gap. The best open-weight model (31B parameters) preserves all required evidence in only 66.8% of cases. The best proprietary model manages just 61.6%. These aren't cherry-picked failure modes; they're the headline numbers on inputs averaging only 3.1K tokens, well within every tested model's context window. The ladder here is revealing for what it doesn't show. The authors don't compare against a single reigning compression SOTA and declare victory — this is a diagnostic benchmark, not a method paper. The comparison axis is across model families and sizes. What emerges is that performance varies sharply by evidence structure: a model strong on qualifier preservation may fail on minority perspectives. Any evaluation using only one structure type produces biased conclusions. The paper introduces Evidence Preservation Return (EPR), a metric relating evidence recall to inference FLOPs, and finds it peaks at moderate model scales (4B-8B), precisely where recall itself is still low — meaning the compute-efficient sweet spot is also the accuracy-poor sweet spot. Architecturally, KEEP-Bench is model-agnostic: it's a benchmark, not a compression method. The tested compressors span open-weight instruction-tuned models up to 31B and proprietary APIs, evaluated under both zero-shot and tailored prompting. The diagnostic design — three evidence structures, each requiring preservation of specific textual signals rather than general gist — is the core contribution. The EPR metric is a ratio (evidence recall per inference FLOP), designed to penalize brute-force scaling and reward information density. The integrity profile is solid for a benchmark paper. All 4,102 test cases are constructed with explicit ground-truth evidence annotations. The benchmark tests three structurally distinct evidence types, reducing the risk that conclusions overfit one pattern. Code and project pages are public. The main limitation: no pre-registration, and the specific evidence structures (reference resolution, qualifiers, minority perspectives) were presumably chosen by the authors, leaving room for the question of whether other structures would tell a different story. The milestone that matters is closing the gap between gist-preservation and evidence-preservation. Right now, even the best models lose roughly a third of critical evidence. Getting to 90%+ evidence preservation at the 4B-8B compute sweet spot would make compression actually trustworthy for downstream tasks that depend on specifics — legal, medical, scientific summarization. The paper doesn't estimate a timeline, but it lays the target plainly. The obvious experiment the authors didn't run: testing whether retrieval-augmented compression (compress, then retrieve dropped evidence from the original on demand) closes the gap. My read is (a) — this is a benchmark paper scoped to diagnosis, not to proposing a fix, and building a retrieval-augmented compressor is a separate research effort they're likely setting up for the next paper. The other missing piece: evaluating compression chains (compress, then compress again) to see whether evidence loss compounds multiplicatively.
