A Benchmark That Catches Vulnerability Databases Lying by Omission — and Measures How Well AI Can Fix Them

149 vulnerable Python versions missing from four major CVE databases. Best AI agent migrates exploits across versions 58.3% of the time. Attack-vector shifts remain brutal.

Imagine you have a master key that opens a specific lock in a building, and someone asks: does this key also open the same lock on the floor below, where the lock is slightly older but from the same manufacturer? You can't just eyeball the key and the lock — you have to walk downstairs and try it. That's CrossPloit's core mechanism: given a working exploit (the key) for a known vulnerable software version (the lock), can you adapt it to work on an earlier, unlisted version where the same flaw likely exists but nobody bothered to check? The committed claim: vulnerability databases — NVD, GitHub Advisory, OSV, Snyk — systematically undercount which software versions are actually exploitable. The authors built 127 migration instances across 45 CVEs in Python web frameworks and confirmed that 149 prior versions are vulnerable but absent from all four databases' affected-version ranges. This isn't a theoretical gap. Every missing version is a deployment that scanners wave through as safe. The benchmark architecture is container-based and execution-grounded. Each instance ships a version-pinned Docker environment, a reference proof-of-concept exploit, and a vulnerability-specific oracle that checks whether the observed effect (crash, data leak, privilege escalation) actually traces to the disclosed vulnerability rather than some unrelated breakage. Two validators gate every result. The migration task itself is a compound problem: you need source-code analysis to understand API changes between versions, environment repair to handle dependency shifts, and execution-based verification to confirm the exploit lands. This is not a static analysis benchmark — the exploit must run. The ladder is clear but the ceiling is low. Eighteen agent configurations were tested — nine LLMs across two harness designs (one general-purpose, one with structured migration scaffolding). The best configuration hit 58.3% resolve rate, compared to 8.7% for an adapted trace-guided baseline. That 58.3% sounds decent until you look at attack-vector shifts — cases where the exploit path changes between versions because an endpoint moved or an API was refactored. No configuration cracked 37.9% on those. The failure mode analysis is honest: higher resolve rates correlate with fewer missed-API failures and more incomplete-execution failures, meaning the better agents at least find the right code paths but sometimes can't finish the job. The most revealing ablation: stripping the reference PoC and its environment specification from one configuration dropped resolve rate from 50.4% to 15.0%. This quantifies exactly how much of the task is novel reasoning versus guided adaptation. The agents are largely doing sophisticated copy-editing of exploits, not discovering vulnerabilities from scratch — which is exactly the right framing for this task. The integrity setup is strong for a benchmark paper. Every instance is containerized and reproducible, oracles are vulnerability-specific rather than generic crash detectors, and the two-validator design reduces false positives. The baselines are not stale — the trace-guided approach is a reasonable prior-art choice. What's missing is external validation: no independent team has run the benchmark yet, and the 45 CVEs are drawn exclusively from the Python web ecosystem, which limits generalizability claims. The practical implication is uncomfortable: if the best AI agent can only migrate 58.3% of exploits across versions, and attack-vector shifts remain below 38%, then the 149 missing vulnerable versions the authors found are likely a lower bound. The real exposure in production systems running unlisted versions — across all languages, not just Python — is almost certainly larger. CrossPloit gives the security community a concrete, reproducible way to measure that gap and track whether AI-assisted auditing can close it.