You know how a kitchen scale drifts? You bought it, calibrated it once, and three years later it's reading 5% heavy on everything — but you never notice because you never re-weigh the reference weight. That's what's happening to AI benchmarks. The tests we use to rank coding agents are silently decaying, and nobody is recalibrating the scale. The committed claim: benchmarks accumulate "maintenance debt" exactly the way software accumulates technical debt, and you need a continuous-validation pipeline — community reports, automated oracle checks, human-gated repairs — to keep scores meaningful. The authors apply this to Terminal-Bench, a benchmark that was already extensively red-teamed before release, and still find roughly 30% of tasks have issues ranging from brittle verifiers that reject valid solutions to stale environmental assumptions. This is not sloppy benchmark design; this is entropy. The repairs matter. Upgrading from Terminal-Bench 2.0 to 2.1 shifts measured accuracy by over 10 percentage points and moves leaderboard positions by up to three ranks. The overall ranking correlation remains high (Spearman ρ = 0.958), so it's not chaos — but if your model's competitive edge is 5 points, a 10-point measurement artifact can flip the result entirely. The benchmark was grading homework with a broken answer key. The most striking finding isn't about task artifacts at all — it's about resource budgets. Extending the wall-clock time limit closes a 20-percentage-point harness gap on GPT-5.4 and shrinks a 30pp gap on GPT-5.4 mini down to just 5pp. In other words, when you thought one agent was dramatically better than another, the real difference was that one ran out of time. The ruler was measuring the clock, not the capability. Architecturally, this isn't a new model or training method. It's infrastructure: a continuous-validation workflow combining automated oracle checks with human review gates. The closest analogue is CI/CD for software — the insight is that benchmarks ARE software, and they need the same maintenance discipline. The paper sits in the growing meta-evaluation literature alongside work like BigBench, HELM, and Chatbot Arena's Elo-based approach, but focuses specifically on the rot problem rather than scale or coverage. Integrity is mixed in instructive ways. The authors audit their own benchmark, which is inherently circular — but they release all repairs, traces, and the continuous-validation tooling, which is the honest move. The 30% defect rate is self-reported against their own prior work, which takes some intellectual courage. There's no independent replication yet, and benchmarks chosen post-hoc always carry selection risk, but the transparency is above field norms. The broader takeaway is that the AI evaluation ecosystem has a silent crisis. If a carefully red-teamed benchmark ships with 30% issues, what does that imply about benchmarks that were NOT audited? Every leaderboard ranking you've seen might carry this magnitude of measurement noise. The field needs benchmark maintenance as a first-class engineering discipline, not a one-time validation step at publication.
