Imagine you're a teacher grading essays on plagiarism detection software. The software flags every paper written in Comic Sans as plagiarized — not because it analyzed the text, but because every known-cheater in the training set happened to use Comic Sans. Your detection rate looks stellar until a cheater switches to Times New Roman. This paper is about that exact problem, except the essays are AI agent trajectories and the Comic Sans is things like shell command length and tool-name markers. The committed claim: safety benchmarks used to evaluate AI agent guards — the systems that decide whether an agent's action is safe or dangerous — contain surface-level cues that let both evaluators and guards get the right answer for the wrong reason. The authors call these 'shortcuts,' and they build a three-level auditing framework to detect them: first, can you predict labels from superficial features alone? Second, do guards' decisions correlate with those features? Third, if you surgically edit the shortcut cue while holding everything else constant, does the guard's behavior actually change? The results are concrete and uncomfortable. In LinuxArena, shorter shell commands strongly predict whether a trajectory is labeled safe or unsafe — the label leaks through command length alone. In AgentDojo-Traj, a tool-name lookup achieves F1 = 0.901 when domains are mixed across train/test splits, but collapses to 0.495 under domain-holdout — meaning the classifier was memorizing which tools belonged to which domain, not learning safety. In AgentHarm-Traj, simply reweighting from per-row to per-interaction scoring drops the published TS-Guard F1 from 0.902 to 0.719. The benchmarks were flattering the guards. The sharpest evidence comes from a seven-configuration paired experiment on a frozen 64-step AgentDojo-Traj cohort using Claude Haiku. Rewriting the formatting marker — a surface cue inherited from the dataset's construction — lowered the guard's blocking rate by 8.33 percentage points. Replacing real tool names with opaque aliases raised blocking by 11.46 points. Crucially, all genuinely unsafe calls remained blocked in both conditions; the changes hit only false positives against inherited labels. Both effects survived subtraction of a formatting-placebo baseline, with exploratory 95% ranges excluding zero. This is direct, interventional evidence that benchmark artifacts change a guard's decisions and its measured performance. The limitation is equally direct: the Haiku-sized effects did not reproduce in Luna or deterministic quantized TS-Guard. The authors report this honestly rather than burying it, which is the right call — it means the shortcut sensitivity is model-dependent, not universal, and the field needs to understand which architectures are vulnerable and why. The three-level framework itself (label predictability → guard association → paired decision sensitivity) is the more durable contribution than any single effect size. The paper's practical recommendations — domain-group holdouts instead of random splits, evaluation-unit metrics instead of row-level metrics, and placebo-controlled paired interventions — are immediately actionable for anyone building or evaluating agent safety systems. These aren't aspirational suggestions; they're specific methodological fixes for problems the paper demonstrates with numbers. The NeurIPS 2026 workshop acceptance (TAE and AIWILD) is appropriate scope: this is a methods-and-measurement paper, not a new architecture, and it knows what it is.
