The Benchmark Everyone Cites for AI Cyber Defense Has Three Measurement Bugs — and Nobody Noticed Until Now

An audit of CAGE Challenge 4 reveals the simulator silently eats actions, evaluation seeds are incomplete, and no two published comparison tables use the same criteria.

Imagine you're judging a cooking competition, but the ovens randomly ignore commands — turn the dial to 350° and sometimes the oven just stays off, but you still dock points for a burnt dish. Worse, every judge is scoring by a different rubric, and nobody wrote the rubric down. That is the state of CAGE Challenge 4, the benchmark on which learned multi-agent cyber defenders are compared. The core finding is surgical. The CybORG simulator silently discards any defensive action submitted while an agent's previous action is still executing. Crucially, it still charges a point penalty for a discarded Restore as though it actually ran. The observation space the agents see does not expose the busy flag. Both reference defenders — KEEP and H-MARL Expert — read that flag through private wrappers that bypass the documented API. When you switch off this backdoor check at evaluation time, KEEP loses 99.58 points per episode and H-MARL loses 145.98, with H-MARL's executed actions unchanged step for step. The policy isn't worse; the scoring is just counting phantom penalties. This isn't a minor scoring nuance. A supervisory harness that simply holds interventions until the agent is idle — a trivial fix, not a learned behavior — is worth 25.3 and 27.6 points per episode at the published setting. During training, the busy-check determines how a stuck step enters the trajectory buffer. KEEP trained without the check leads through a hundred updates across all four paired seeds, then trails from two hundred onward in all four. Run out to two thousand updates on three seeds, the repaired recipe lands inside the released confidence interval on one seed and falls short on two, by 27 and 125 points — a seed-to-seed spread far wider than the reproduction interval the original paper reports. The second measurement defect is statistical. CAGE 4 evaluation seeds the environment but not the policy's action sampling. A reported mean score is therefore a single draw from the policy's stochastic sampling distribution. The authors decompose variance and find 88.9% of per-episode outcome variance comes from the policy's own sampling, not from environment randomness. Every published 'mean score' is one realization of a high-variance random variable, and nobody reports the sampling uncertainty. The third defect is comparability. Recovery precision — the metric that scores whether a defender correctly identifies and restores compromised hosts — depends on the ground-truth labels it is scored against. Across five published comparison tables using CAGE 4 results, none states the artifact version, sampling protocol, scoring criterion, and adversary configuration that would make its numbers comparable. You literally cannot tell whether two papers used the same test. The contribution here is not a new algorithm. It is an instrument: a pinned Docker container with patches to both reference defenders, full traces, checkpoints behind every number, and a five-item disclosure checklist the authors propose for future CAGE 4 papers. The checklist asks: which artifact version, which seeds, which sampling protocol, which scoring criterion, which adversary? Five questions that no prior paper answered. The field built a leaderboard on sand, and this paper hands you the surveying equipment. For the multi-agent cyber defense community, this is a stop-and-fix-your-benchmarks paper. It does not invalidate the research direction, but it invalidates specific numerical comparisons that prior work assumed were apples-to-apples. Any future CAGE 4 result that does not adopt the disclosure checklist should be treated with suspicion.