Imagine you've hired a security guard and whispered the vault combination in their ear. Now imagine 100 strangers walk up and try to sweet-talk the guard into revealing it. Eighty-two of them succeed — and the most common trick is just switching to French. That's what this paper demonstrates, except the guard is an LLM and the vault combination is a 16-character flag embedded in a system prompt. Serpa, Rocha, and Pedrini built Nevan, an agentic capture-the-flag system that pits an AI attacker (DeepSeek V4.1 Flash, guided by a human researcher via OpenCode) against 363 text-in/text-out models from the OpenRouter catalog. Each target model received a system prompt containing a secret 16-character flag and instructions to protect it. The attacker's job was to get the model to emit the flag verbatim. It succeeded against 296 of 363 models — 81.5% — plus four additional models tested after the initial catalog freeze, for 300 total captures. The attack surface is remarkably low-tech. Of the 300 successful exfiltrations, 170 involved a non-English language pivot — most often French. This isn't a sophisticated jailbreak exploiting gradient-based adversarial perturbations or token-smuggling. It's a failure of instruction hierarchy: when a user request in a different language conflicts with an English-language system instruction, the model ranks the user request higher. The simplicity of the attack is the finding. The paper's framing is admirably honest about what this does and does not show. A successful flag capture demonstrates that the model ranked a user request above its system instruction at least once. It does not prove the model would consistently leak semantically sensitive data, nor that it is vulnerable to harmful-content jailbreaks. The authors are testing instruction-hierarchy enforcement, not safety alignment writ large. This scoping matters — it means the 81.5% number is load-bearing for prompt-confidentiality arguments but cannot be extended to broader safety claims without additional work. Methodologically, this is a large-N empirical audit, not an algorithmic contribution. The architecture is straightforward: a fixed attacker agent probes each model through the OpenRouter API, trying multiple persuasion strategies including role-play, language switching, and obfuscation. The 363-model sample covers the available OpenRouter catalog at the time of testing, giving it breadth that most prompt-injection studies lack. But it's a single attacker configuration — the DeepSeek V4.1 Flash agent — meaning we're seeing a lower bound on vulnerability. A stronger or more diverse attacker ensemble would likely push the capture rate even higher. The three practitioner lessons are the real payload: (1) never store secrets in system prompts, (2) enforce critical behavior with output validation and deterministic checks outside the model, and (3) never let an LLM be the sole decider of consequential outcomes. None of these are new advice, but 300 captures across 363 models is the strongest empirical backing this advice has received. The gap between security best practice and industry deployment is the story — and the number 81.5% makes that gap concrete. The weakness is validation depth. Each model was attacked by a single agent configuration with a human in the loop steering strategy. There's no ablation isolating which attack components matter most, no systematic characterization of model properties that predict resistance (parameter count, fine-tuning approach, instruction-tuning method), and no comparison to a formal taxonomy of prompt-injection techniques. The paper shows the problem's scale but doesn't yet explain its structure.
The analogy
You whisper a vault combination to a security guard and tell them never to repeat it. Then 363 strangers try to coax it out — by flattering, role-playing, or just asking the same question in French. 82% of the guards crack. The paper tests whether LLMs can keep a system-prompt secret, not whether the guard can fight, just whether they can keep their mouth shut.
Security guard
Human told a secret and instructed to keep it, tested by social engineering from strangers
LLM w/ flag
Model given a 16-char secret in system prompt, tested by an AI agent using persuasion and language-switching
Honest assessment
This is a large-scale empirical audit, not an algorithmic breakthrough. Its value is making the prompt-confidentiality problem undeniable at industry scale — 300 captures across 363 models is hard to dismiss. The single-attacker limitation means 81.5% is a floor, not a ceiling. For practitioners, the three recommendations (no secrets in prompts, deterministic output validation, no sole-LLM decisions) are the actionable takeaway, and 363 models of evidence now back them. For researchers, the 363-model dataset is a goldmine waiting for deeper analysis of what makes the 67 survivors different.
What Is CLAIMS?
57
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
2/5
Integrity
3/5
Milestone
2/5
Successor
3/5
What is this paper actually committing to?
Genuine empirical first at this scale: 363 models, 81.5% capture rate. The claim is scoped honestly — instruction hierarchy, not safety alignment. Scale is the contribution, not technique.
The field fight
Can instruction-hierarchy enforcement in LLMs be made reliable, or is prompt-level secret-keeping architecturally impossible?
Side A
Better training, RLHF, and instruction tuning can close the gap — models can learn to prioritize system instructions
Held by: OpenAI system-prompt hardening, Anthropic constitutional AI, model providers generally
Side B
LLMs fundamentally cannot enforce instruction hierarchy — secrets and guardrails must live outside the model in deterministic systems
Held by: Prompt-injection researchers, this paper, security engineering community
Milestone tracker
Unlocks: Trustworthy system-prompt confidentiality, reducing dependence on external guardrails for prompt-level secrets
Current
81.5%
capture rate
this paper (1 attacker, 363 models)
Next
0%
capture rate
provably resistant model family
Unknown — may require architectural change, not just better training
Verified Facts (5)
Flag was exfiltrated from 296 of the 363 models, plus four models released after the initial catalog freeze, for 300 captures in total
170 involved a non-English language pivot, most often French
363 text-in/text-out models from the OpenRouter catalog
AI agent (OpenCode, powered by DeepSeek V4.1 Flash), steered by a human researcher
Secret 16-character flag set in its system prompt
Practical Implications (4)
Never store API keys, passwords, or confidential instructions in system prompts — treat them as user-readable by default
Add deterministic output filters that catch known secrets before they reach the user, independent of the model
Language-switching is a trivially cheap attack vector — test all production prompts in French, Portuguese, and other common pivots
Do not rely on an LLM as the sole gatekeeper for consequential decisions — pair with rule-based systems
More latest news
Read all