Imagine you trained a customs dog to sniff for six specific drugs. It's great at those six. But when a seventh arrives — same contraband logic, different molecular shape — the dog wags its tail and waves it through. Meanwhile, a generalist inspector who has "seen everything" flags it immediately, but you can't verify the inspector hasn't already memorized the seventh drug's profile from a leaked manifest. That's the core tension this paper dissects. The committed claim: in a two-tier web-application firewall where a fast model triages most traffic and escalates uncertain requests to a slow (expensive) tier, supervised detectors trained on six of seven OWASP attack families will confidently mark large fractions of the held-out seventh family as benign — meaning those attacks are never examined by the slow tier at all. Over 10 training runs, ModernBERT assigns a sub-0.05 probability to 37–60% of held-out path-traversal payloads and only escalates 50–81% within a 5% false-positive budget, compared to near-100% when traversal is in-distribution. Zero-shot decision models (Jev and Clef, hosted LLM-based services) escalate ≥97.6% of every family under the same budget — but their undisclosed training corpora likely contain the very attack signatures being held out. The experimental design is unusually honest. The authors splice real attack payloads into parameters of genuine browsing traffic, which avoids the common pitfall of testing on synthetic requests that look nothing like production logs. They compare five system types: supervised classifiers (ModernBERT, DeBERTa), a local language model, the OWASP Core Rule Set, an anomaly baseline, and the two hosted decision models. Each of the seven OWASP families — XSS, SQLi, command injection, SSRF, path traversal, open redirect, SSTI — takes a turn as the unknown. This leave-one-family-out protocol is the paper's structural backbone and its most valuable contribution to methodology. The integrity picture is nuanced. Reporting variance across 10 training runs with explicit ranges (not just means) is above-average rigor for applied security ML. The "benign twins" control — replacing the attack payload with a legitimate value while keeping the request structure identical — catches a critical confounder: Jev escalates 37% of benign twins, suggesting it's partly detecting scanner fingerprints, not attack semantics. But the paper's benchmarks are entirely self-constructed; there's no community-standard held-out attack benchmark for web-request triage, which means the experimental design is both the paper's strength and its unverifiable foundation. The adversarial search result deserves attention. A 300-query search guided by Jev's own probability scores gets 43% of held-out attacks past Jev — but when the same requests are resubmitted, Jev escalates them 60% of the time. This stochastic inconsistency in hosted decision models is a practical problem the paper surfaces but doesn't resolve. It means an attacker with API access can play a variance lottery: submit the same payload repeatedly until the model flips to benign. The paper's architecture comparison is its most generalizable lesson. Supervised detectors are fast and cheap but brittle to distribution shift by attack family. Zero-shot LLMs are robust but expensive, opaque, and potentially contaminated by their training data. The OWASP Core Rule Set (regex-based) is interpretable but misses novel payload encodings. No single tier solves the problem, which is exactly the argument for two-tier triage — but the paper shows the fast tier's failure mode is silent confidence, the most dangerous kind. What's missing is the obvious scaling experiment: adversarial retraining. If you take the fast tier's false negatives on held-out families and feed them back as training data, how quickly does the detector generalize? The authors don't run this, and the honest read is budget — each held-out family requires 10 fresh training runs across multiple architectures. But it's the experiment that would convert a diagnostic paper into a prescriptive one.
