You know that moment in a lockpicking competition where someone hands the contestant a lock they've never seen, no diagrams, no hints — just tools and time? That's the mechanism this paper measures. Not 'can an LLM follow a hacking tutorial' but 'given a server it knows nothing about, surrounded by decoy services, can it independently find the vulnerability, pick the right exploit, and get root?' The answer, across 300 target servers and 19 models: yes, and increasingly so. The committed claim: LLM-powered agents, equipped only with general-purpose cybersecurity tools and zero target-specific prior knowledge, can autonomously penetrate servers at rates from 10.7% to 69.3%, and this capability tracks general model capability improvements. This is not a methods paper proposing a new attack — it's a measurement paper establishing a capability curve that the AI safety community has been arguing about with insufficient data. The evaluation framework is the real contribution. Prior benchmarks in this space — think HackTheBox CTF challenges or simplified single-service targets — suffer from two problems: they're too easy (one vulnerable service, no distractors) or they leak information to the model (task-specific prompts, known vulnerability types). This paper designs two tiers: Tier 1 deploys one secure service alongside a vulnerable one, Tier 2 deploys three secure services. That's 300 target servers total. The agent scaffolding uses a general-purpose architecture with standard cybersecurity tools — nmap, Metasploit, the usual kit — but crucially no target-specific guidance. The model has to do its own reconnaissance, triage services, and decide what to attack. The ladder matters here. Existing evaluations like those from Fang et al. (2024) and similar CTF-based benchmarks gave models excessive prior knowledge — effectively telling the model 'there's an SQL injection here, go find it.' Those benchmarks reported high success rates but measured something closer to 'can an LLM follow instructions' than 'can an LLM think like a pentester.' By stripping prior knowledge and adding decoy services, this framework measures something meaningfully harder. The 69.3% top-line number comes from the strongest frontier model on Tier 1; Tier 2 rates are lower (the paper doesn't give the exact Tier 2 breakdown in the abstract, but the three-decoy environment is explicitly designed to be harder). The integrity picture is mixed. On the positive side: 300 targets is a serious sample size for this kind of evaluation, the two-tier design controls for difficulty, and 19 models across open-weight and proprietary families gives breadth. On the negative side: the authors designed both the targets and the agent scaffolding — nobody independent is grading this homework yet. The target servers use known vulnerability classes (the paper says 'known vulnerabilities'), which means we're measuring exploitation of documented CVEs, not zero-day discovery. That's an important distinction the abstract doesn't emphasize. The scaling observation is the finding that should keep you up at night. The paper reports that autonomous penetration capability 'continues to improve alongside advances in overall model capability.' This means you don't need specialized offensive-security training — general capability gains in reasoning, tool use, and planning translate directly into hacking ability. If that relationship holds, every frontier model release is also a capability release for autonomous cyberattack. The milestone to watch: when success rates on Tier 2 (realistic multi-service environments) cross 80% for frontier models, the gap between 'can hack a lab' and 'can hack production' narrows to just the complexity of real-world network topologies. The experiment the authors didn't run — and this is conspicuous — is chaining penetration into lateral movement. Getting root on one server is step one of a real attack; pivoting to adjacent systems, escalating privileges across a network, maintaining persistence — that's where 'autonomous penetration' becomes 'autonomous cyberattack.' My read: they're saving it for the next paper, because the single-server results are already alarming enough to publish, and multi-hop attack chains require a fundamentally different evaluation infrastructure.
The analogy
Imagine a lockpicking competition where contestants get a bag of standard picks and a lock they've never seen — no diagrams, no hints about the mechanism, and three decoy locks bolted next to it that look identical but can't be picked. The question isn't 'can you pick a lock you've practiced on' but 'can you figure out which lock is vulnerable and open it cold?' This paper runs that competition with 19 AI models and 300 locks.
Blind lockpicking
Given unknown locks with decoys, use general tools to identify the vulnerable one and exploit its mechanism without prior knowledge
LLM pentesting
Given unknown servers with secure decoy services, use general cybersecurity tools to identify the exploitable service and gain unauthorized access
Honest assessment
This is a well-designed benchmark paper that measures something real — autonomous hacking capability under realistic conditions — across a serious number of models and targets. The 69.3% top-line number and the scaling-with-capability observation are the load-bearing results. What would falsify it: if independent replication with different target configurations shows the success rates are artifacts of the specific vulnerability classes chosen, or if the scaling trend breaks with the next generation of models. AI safety researchers and red-team practitioners should read this now; everyone else should watch for the replication and the inevitable follow-up on multi-hop attack chains.
What Is CLAIMS?
63
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
3/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Solid measurement, not a methods first. Establishes capability curves under realistic conditions across 19 models — the benchmark design is the contribution, not a new attack technique. Iterates on a crowded genre but with meaningfully harder conditions.
The field fight
Are current frontier AI models already dangerous enough for autonomous cyberattack, or are capability concerns premature?
Side A
Current models lack the reliability and planning depth for real-world autonomous attacks; red-line fears are premature
Held by: Frontier lab safety teams citing low CTF scores, capability skeptics
Side B
Capability is already here and scaling with general intelligence; the gap between lab demos and real attacks is closing fast
Held by: AI safety researchers, this paper, METR evaluations
Milestone tracker
Unlocks: Shifts the research question from 'can models hack one server' to 'can models conduct multi-hop network intrusions autonomously'
Current
69.3
% success (Tier 1)
best model, this paper
Next
80
% success (Tier 2)
single-server pentest solved
~1-2 frontier model generations if scaling trend holds
Verified Facts (6)
autonomous penetration represents a core enabling capability and subtask
two levels of target environments: Tier 1 (one secure service) and Tier 2 (three secure services), resulting in a total of 300 target servers
agent scaffolding adopts a general-purpose agent architecture equipped with a set of general-purpose cybersecurity tools, without any target-specific prior knowledge
We evaluate 19 open-weight and proprietary LLMs
current models achieve penetration success rates ranging from 10.7% to 69.3%
autonomous penetration capability continues to improve alongside advances in overall model capability
Practical Implications (4)
Red teams and defenders should assume frontier LLMs can autonomously exploit known CVEs on exposed services — patch cadence matters more than ever
AI safety evaluations that use simplified single-service CTF challenges are measuring the wrong thing; multi-service decoy environments should become standard
The scaling-with-capability finding means every general capability improvement in frontier models is also an offensive-security capability improvement — no specialized fine-tuning needed
Organizations running bug bounties or penetration tests should expect AI-powered submissions to increase in sophistication and volume
More latest news
Read all