Imagine you've been washing dishes by hand for years, and someone installs a dishwasher. The dishes don't get cleaner — they were always dirty. You just see the full pile now because the machine processes them faster than you ever could. That's what LLM-based vulnerability agents are doing to software security: the bugs were always there, but automated discovery is surfacing them at a rate humans never matched. The core claim of this SoK (Systematization of Knowledge) is structural, not empirical in the usual sense. The authors build a capability taxonomy — modeled explicitly on SAE's levels of driving automation — that rates LLM-based vulnerability research tools across discovery, validation, and patching on a scale from no autonomy to full autonomy without human intervention. They then grade every published claim (academic papers, vendor disclosures, frontier-model reports) by correctness, availability, and evidence strength. The punchline: agents achieved autonomous discovery, validation, and patching of real-world vulnerabilities before the restricted frontier models of 2026, and open-weight models subsequently reached comparable autonomy for discovery and validation. The empirical weight comes from measuring what actually happened in the vulnerability ecosystem. In 2026, the number of disclosed vulnerabilities more than doubled. The share rated critical or high increased. Severe bugs that had survived a decade or more in widely-used codebases were reported at several times the historical rate. This is the dishwasher effect — the bugs aren't new, but the discovery throughput is unprecedented. The paper uses a decade of public CVE records, advisories, and exploitation data to establish these shifts, which is a real contribution: most prior work either benchmarks a tool on CTF-style challenges or makes qualitative claims about capability. Critically, the paper finds that the temporal structure of vulnerability lifecycles has not shifted noticeably. The time from report to fix and from disclosure to attack remains roughly constant. This is both reassuring and alarming: defenders haven't sped up, but neither have attackers shortened exploit timelines yet. The key word is "yet" — the paper is measuring a moment when attacker and defender tooling are scaling simultaneously, and the equilibrium could break in either direction. The autonomy framework itself is the paper's most durable contribution. By rating claims on an explicit scale with explicit evidence grading, the authors create a common language for comparing results that were previously scattered and incommensurable. The SAE-level analogy is not just decorative — it forces precision about what "autonomous" actually means (tool assists human vs. agent acts without human in the loop), which matters enormously for policy and risk assessment. The mitigations section grades proposed defenses with an explicit rubric, which is rare and welcome. Most SoKs list mitigations without evaluating them. Here, the authors draw stakeholder-specific guidance from the best-supported proposals, though the abstract gives no detail on which mitigations scored well. The honest limitation: this is a systematization, not a primary result. It synthesizes existing evidence rather than generating new benchmarks. Its value depends entirely on how carefully the authors triaged the evidence quality of each included claim — and without the full paper, we can't audit that triage. The bigger picture: this paper lands squarely in the debate over whether LLMs are net-positive or net-negative for software security. The data suggest both simultaneously — more bugs found, more bugs fixed, but the attack surface is now visible to anyone with API access. The driving-automation analogy is apt: Level 4 autonomous driving didn't eliminate accidents, it changed who was responsible and how fast the system operated. The same transition is happening in vulnerability research right now.
The analogy
Think of washing dishes by hand versus installing a dishwasher. The dishes were always dirty — the machine just processes the full pile faster than you ever could. LLM vulnerability agents don't create new bugs; they surface the existing backlog at throughput humans never matched. The key question is whether the kitchen gets cleaner or just more chaotic.
Dishwasher
Automated processing of an existing backlog (dirty dishes) at throughput impossible by hand
LLM vuln agents
Automated discovery of an existing backlog (latent software bugs) at throughput impossible by manual audit
Honest assessment
A well-structured survey that provides a genuinely useful common language for a scattered field, plus a striking empirical finding (CVE doubling, decade-old bugs surfacing). It doesn't generate new primary results, and its load-bearing contribution — the evidence-grading rubric — can't be audited from the abstract alone. Most valuable for security policy teams and researchers who need to quickly triage the flood of LLM-capability claims. The 'days-to-exploit hasn't shifted' finding is the most policy-relevant data point and the one most likely to change soon.
What Is CLAIMS?
60
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
2/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Useful but not first-of-kind. The autonomy taxonomy is original; the CVE-doubling finding is striking. But this systematizes existing claims rather than generating a new primary result.
The field fight
Are LLM vulnerability agents net-defensive (more bugs found and patched) or net-offensive (more bugs exploitable by attackers)?
Side A
Net-defensive: accelerated discovery means faster patching and reduced latent attack surface
Held by: Google DeepMind, Microsoft Security, Trail of Bits optimists
Side B
Net-offensive: democratized discovery lowers the barrier for attackers; defenders can't patch faster than agents discover
Held by: RAND, some CISA analysts, offense-first security researchers
Milestone tracker
Unlocks: Policy inflection: if exploit speed drops below patch speed, the entire responsible-disclosure model breaks
Current
2×
CVE rate vs 2025
2026 disclosure surge
Next
<7
days disclosure-to-exploit
offense outpaces defense
Unknown — currently stable, but could shift rapidly if open-weight models reach Level 5 patching autonomy
Verified Facts (6)
In 2026 the number of disclosed vulnerabilities more than doubled
Severe bugs that had survived a decade or more in widely used code were reported several times more often than before
The time between the stages of a vulnerability's life, from report to fix and from disclosure to attack, has not shifted noticeably so far
Agents discovered, validated, and patched vulnerabilities in real-world software without human intervention before the restricted frontier models of 2026
Open-weight models subsequently attained comparable autonomy for discovery and validation
Framework rates capabilities on an autonomy scale modeled on the levels of driving automation
Practical Implications (4)
Security teams should assume LLM agents are already finding real bugs in their codebases autonomously — plan patching capacity for 2× or higher discovery throughput
The autonomy taxonomy gives product security orgs a shared vocabulary for evaluating vendor claims about AI-assisted vuln scanning
The stable days-to-exploit finding is temporary comfort — build exploit-response capacity now before the equilibrium breaks
Open-weight models reaching discovery/validation parity means the capability is not containable through API restrictions alone
More latest news
Read all