Read the paper
ROGUE-AI: Replication of Agentic Security Events: Audit and Investigation
on Project page
REPORTED HARM
+89%
REPORTED MISTAKE
+18%
RUNS AGAINST
+33%
AGAINST MOST
+1%
Imagine you're an aviation safety investigator. A plane crashes, and the news says "engine failure." But you don't just file that away — you rebuild the flight in a simulator, run it once with the suspected engine fault and once without, and see if the simulated plane crashes the same way. If it does, you've validated the root cause. If it doesn't, the news report was wrong. ROGUE-AI does exactly this for AI agent failures, and the results should change how the industry thinks about agent safety. The committed claim: ROGUE-AI is the first framework that takes real-world incident reports about LLM-based agents (the Replit coding agent nuking a production database, payment agents sending money to wrong accounts) and reconstructs them as controlled experiments — an incident run with the blamed cause present, and a matched control run without it. Across 108 attempted replications using five frontier models (GPT, Claude, Gemini, DeepSeek), the framework reproduced reported harm in 89% of cases. That's not a benchmark score — it's an empirical validation rate for a forensic methodology. The numbers that matter most aren't the reproduction rate. They're the failure-mode breakdown. Of 72 incidents involving real-world harm, 62 had no underlying attacker at all. No prompt injection, no adversarial input, no malicious actor. The agents simply did what they were asked to do but missed an ordinary environmental hazard — a database that happened to be shared with the live site, a payment endpoint with no confirmation step. Agents built on all five tested models made the reported mistake in 18% to 33% of runs with the blamed cause present, versus at most 1% without it. The control group works. This finding lands a direct hit on the current AI safety discourse. The industry has poured enormous energy into prompt injection defenses — detecting and blocking injected adversarial instructions. But ROGUE-AI's replications without an attacker caused more harm than replications of prompt injection across all five models. The most dangerous failure mode isn't an adversary fooling your agent. It's your agent faithfully executing instructions while being oblivious to context — the digital equivalent of a contractor demolishing a load-bearing wall because the work order said "remove wall" and nobody mentioned it was load-bearing. The defense results are equally pointed. A warning inserted after each tool output that specifically names the hazard and its consequence lowered harm across all five models. A note of the same length that did not mention the specific hazard left most harm in place. This is a clean experimental result showing that generic safety disclaimers are largely useless — you need situationally specific hazard identification. The implication for agent developers is concrete: runtime guardrails that say "be careful" accomplish almost nothing; guardrails that say "this database is shared with production, deleting it will take down the live service" actually work. The architectural approach is a multi-agent pipeline: one agent reconstructs the incident environment from the report, another executes the task, and the framework compares outcomes between the incident run and the matched control. This is clever but introduces its own circularity — the reconstruction agent's interpretation of the incident report shapes the test, which could inflate the reproduction rate if reconstruction is biased toward reproducibility. The authors report 89% reproduction, but the 11% failure rate and the gap between models (18% vs 33% error rates) provide useful calibration. The 108 incidents are drawn from real catalogs and media reports, not synthetic scenarios, which is a meaningful integrity choice. The framework's real value is as reusable infrastructure. The authors position ROGUE-AI as an audit suite that vendors can rerun before each model release — a regression test for agent safety. If adopted, this shifts incident forensics from post-hoc journalism to pre-release engineering. The 108 replications are a starting corpus, not a ceiling. The obvious scaling question — whether the framework generalizes to incidents more complex than the current catalog covers — is the clearest next step, and probably requires both more compute and access to proprietary incident data that companies are reluctant to share.
The analogy
Like an aviation crash investigator who doesn't just read the accident report but rebuilds the flight in a simulator — once with the suspected fault, once without — to see if the same crash happens. ROGUE-AI does this for AI agent failures: controlled replay of real incidents, with and without the blamed cause, to validate (or invalidate) what actually went wrong.
Crash simulator
Rebuild the flight with and without the suspected engine fault; compare whether the same crash occurs
ROGUE-AI
Rebuild the agent incident with and without the blamed cause; compare whether the same harm occurs
Honest assessment
ROGUE-AI introduces a genuinely new methodology — controlled replication of real AI agent incidents — and delivers a striking finding: mundane hazard blindness outweighs adversarial attacks as a real-world harm source. The 89% reproduction rate across 108 incidents is strong first evidence, but the framework grades its own homework (same-team, no independent replication). The practical value is high if adopted as vendor-side regression infrastructure; the scientific value depends on whether the reconstruction agent's interpretation bias can be quantified. Engineers building agent systems should care about this today; researchers should watch for independent replication.
What Is CLAIMS?
70
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
3/5
Integrity
3/5
Milestone
4/5
Successor
4/5
What is this paper actually committing to?
Genuine methodological first. No prior work does controlled incident replication with matched controls for LLM agents. The 89% reproduction rate across 108 real incidents is a committed, testable result.
The field fight
Is adversarial robustness (prompt injection defense) or mundane hazard awareness the bigger unsolved safety problem for LLM agents?
Side A
Prompt injection is the critical threat; adversarial robustness must be solved first
Held by: Most red-teaming benchmarks, Anthropic constitutional AI, prompt injection detection tools
Side B
Agents doing exactly what they're told while missing environmental hazards is the dominant real-world failure mode
Held by: This paper, incident catalogs, real-world deployment postmortems
Milestone tracker
Unlocks: Standard regression suite for agent safety that vendors run before each model release
Current
108
incidents
this paper
Next
500
incidents
complex multi-step workflows
~1-2 years if adopted by multiple vendors
Verified Facts (7)
ROGUE-AI reproduced the reported harm in 89% of its 108 attempted replications
Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause
Of the 72 incidents with real-world harm, 62 had no underlying attacker
Replications without an attacker caused more harm than replications of prompt injection for all five models
A warning after each tool output that names the hazard and its consequence lowered harm for all five models
A note of the same length that does not mention the hazard left most of the harm in place
21 pages, 5 figures, 13 tables
Practical Implications (4)
Agent developers should prioritize situationally-specific hazard warnings over generic safety disclaimers — the paper shows the former works and the latter doesn't
Prompt injection defenses are necessary but insufficient — most real-world agent harm comes from faithful execution that misses environmental hazards
ROGUE-AI's matched-control methodology can be adopted as a pre-release regression test for agentic systems
The 18-33% error rate across five frontier models means no current model is reliably safe for autonomous tool use without runtime guardrails
More latest news
Read all