Your Security Scanner Can Be Tricked Into Silence for 51 Cents

LLM-generated code rewrites preserve real vulnerabilities while silencing CodeQL alerts 26% of the time. The gaps aren't random — they cluster at seven library-modeling boundaries.

Imagine you have a smoke detector that reliably catches fires in your kitchen. Now imagine that rearranging the furniture — moving the toaster to a different counter, swapping shelves — makes the detector stop going off, even though the fire hazard is identical. You'd never trust a "no alarm" signal again. That is the core mechanism this paper exposes in CodeQL, the most widely deployed static analysis tool in JavaScript security. The committed claim: an LLM with zero knowledge of CodeQL's internals can rewrite vulnerable code so that the real vulnerability survives but CodeQL's taint-tracking alert vanishes — 26.2% of the time across 172 real npm vulnerability flows, at roughly $0.51 per silenced alert. The critical control: when the same LLM is asked to refactor the same functions without being told about the alert, it silences none of them. This is not random drift. The rewrites are targeted enough to hit blind spots but natural enough to pass code review. Every silencing rewrite traces to one of seven specific gaps at what the authors call the "library-modeling boundary" — the seam where CodeQL's hand-written models of library behavior meet actual API surface. The largest single gap is the WHATWG URL constructor: CodeQL tracks taint flowing into a URL's query parameters but loses it when taint flows through the pathname. 64 of 70 silencing rewrites cross a gap inside an API that CodeQL already partially models. These aren't unknown APIs — they're known APIs with incomplete coverage. The validation regime is unusually honest for a security paper. The authors manually audit every rewrite to confirm the vulnerability survives. They test not just CodeQL but an independently built taint analyzer and a combination of three analyzers. The cross-tool results are damning: 74% of CodeQL-silencing rewrites also silence the independent analyzer, and 53% escape all three tools combined. This means the gaps are structural to how taint analyzers model libraries, not quirks of one implementation. The architecture here is adversarial LLM prompting constrained to vulnerability-preserving transformations, evaluated against dataflow-based static taint analysis. The LLM serves as a black-box fuzzer for analyzer robustness — it doesn't need to understand the analyzer's implementation, only the alert location. This is a lightweight, scalable approach: three rewrites per flow, commodity LLM pricing, no specialized training. The practical threat model flips the usual SAST story. We normally ask "does the tool find the bug?" This paper asks "once the tool found the bug, can ordinary-looking code changes make it lose the bug while keeping the vulnerability alive?" In CI/CD pipelines where a disappearing alert is treated as evidence of a fix, this is a concrete attack surface. The seven named gaps give SAST developers a finite, actionable repair list — this is not "everything is broken" nihilism but a specific engineering punch list. The paper's limitation is scope: JavaScript taint tracking only, CodeQL plus two other unnamed analyzers, 172 flows from npm vulnerabilities. The authors do not test compiled languages, interprocedural analysis depth, or whether the gaps persist after CodeQL model updates. The honest next experiment — running the same methodology against CodeQL's Java and C++ taint trackers — would reveal whether library-modeling boundary gaps are a JavaScript-ecosystem problem or a fundamental architectural weakness of dataflow SAST.