
CAPABILITY COVERAGE
+67.0%
CAPABILITY COVERAGE
+32.4%
COMPUTED GPT-5.5
+58.6%
You know how a stuck crossword solver will stare at the same corner, declare it impossible, and put the puzzle down — even though they haven't tried erasing the wrong answer in 14-across that's been blocking everything? That's essentially what LLM agents do during long-horizon vulnerability exploitation. They hit a wall, emit a stop token, and quit with turns left on the clock. This paper asks: what if you force them to keep going, and also quietly erase the "this is impossible" note they wrote to themselves? The committed claim: LLM exploitation agents underperform not because they lack capability but because they prematurely stop, and a targeted context-management strategy (SCoRe — Stop-burst Compaction and Reassessment) can recover substantial hidden capability. Using ExploitBench as the testbed with DeepSeek-V4-Flash-0731 and MiMo-V2.6-Pro, the authors show that simply continuing execution past the first stop yields additional verified gains — but those gains decay as "stop bursts" accumulate and impossibility summaries calcify in the context window. SCoRe attacks both failure modes. When the agent emits a cluster of stop responses, SCoRe compacts the context (reducing noise from repeated dead ends) and rewrites retained summaries that contain impossibility judgments — marking them as "unproven" rather than settled conclusions. This is mechanistically interesting: it treats the agent's own context as a belief state that can be intervened on, not just a transcript to summarize. The result is that MiMo-V2.6-Pro jumps from 32.4% capability coverage under its native stopping policy to 67.0% at 2,000 turns. For reference, GPT-5.5's ExploitBench number at 300 turns is 58.6% on the same tasks. The ladder comparison deserves scrutiny. The GPT-5.5 number comes from ExploitBench's reference results at 300 turns, not from running GPT-5.5 under SCoRe for 2,000 turns. The authors are comparing a souped-up smaller model at high turn budgets against a stronger model at low budgets. That's a real finding — it shows the value of test-time compute scaling with smart context management — but it's not an apples-to-apples model comparison. The honest read is that SCoRe is an inference-time intervention that makes budget-rich settings much more productive, not that MiMo is "better" than GPT-5.5. Two diagnostic findings carry weight beyond the specific method. First, stop-burst frequency increases within a fixed context — agents become progressively more likely to give up once they've started giving up, suggesting a self-reinforcing pessimism pattern in the context. Second, impossibility judgments in summaries ("I cannot achieve X") persist and constrain future exploration even when the judgment was premature. Both observations are relevant to anyone running agentic loops on hard tasks, not just exploit generation. The integrity picture is mixed. ExploitBench is a community benchmark, which is good. But all validation is same-team execution — no independent replication, no pre-registration. The "verified" capability gains use ExploitBench's own verification machinery, which is reasonable but not independent. Code and benchmark availability are not stated in the abstract. The 2,000-turn budget is expensive, and the authors don't report compute costs, making it hard to judge practical efficiency. The security implications are direct and uncomfortable. If a context-management wrapper can double an agent's exploit coverage, then capability evaluations that use native stopping policies systematically underestimate risk. The gap between "what the agent chose to do" and "what the agent can do with prodding" is large enough to matter for safety benchmarking.
The analogy
A crossword solver who writes 'impossible' in the margin, then treats that note as fact every time they revisit the corner. SCoRe erases the note and compacts the messy scratch work so the solver sees the grid fresh — and suddenly fills in answers that were always reachable.
Crossword solver
Writes 'impossible' in the margin, re-reads it as fact, stops trying that corner even with time left
SCoRe agent loop
Rewrites impossibility judgments as 'unproven' and compacts context at stop-bursts so the agent re-explores
Honest assessment
This is a well-diagnosed finding about a real failure mode — LLM agents quit too early and their own context poisons future attempts — with a clean intervention that doubles performance on a community benchmark. The GPT-5.5 comparison is attention-grabbing but budget-asymmetric; the load-bearing result is the 32→67% jump on MiMo under controlled conditions. The security implication is the most consequential part: capability evaluations that respect native stopping policies are systematically underestimating what agents can do when pushed. Anyone running agentic security evaluations or red-team benchmarks should pay attention now, not in five years.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
4/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Real finding, not a first-of-kind. The claim that agents underexplore is known; the contribution is diagnosing stop-burst patterns and showing context editing recovers capability. Solid iteration, not paradigm shift.
The field fight
Is LLM agent capability bounded by model weights or by inference-time orchestration and compute scaling?
Side A
Model capability is the ceiling; inference tricks hit diminishing returns fast
Held by: Scaling-laws camp, chinchilla-optimal training advocates
Side B
Inference-time compute scaling with smart orchestration can unlock capability far beyond naive single-pass execution
Held by: o1/o3 reasoning scaling, Tree-of-Thought, this paper
Milestone tracker
Unlocks: Reliable automated exploit generation that changes how vulnerability disclosure and red-teaming operate in practice
Current
67.0
% capability coverage
MiMo + SCoRe @ 2K turns
Next
80
% capability coverage
Efficient + generalizable
~1-2 years; requires frontier-model testing and turn-budget efficiency improvements
Verified Facts (6)
MiMo-V2.6-Pro reaches capability coverage of 67.0% at 2,000 turns with SCoRe
32.4% capability coverage under the original stopping policy
58.6% computed for GPT-5.5 from ExploitBench's 300-turn reference results on the same tasks
Agents tested: DeepSeek-V4-Flash-0731 and MiMo-V2.6-Pro
Two patterns identified: increasing proportion of stop responses within same context, and earlier stopping associated with summaries retaining impossibility judgments
SCoRe compacts context at stop bursts and marks impossibility judgments as unproven
Practical Implications (4)
Capability evaluations using native stopping policies underestimate agent exploitation potential — safety benchmarks need budget-extended runs
Context-window pollution from self-generated impossibility judgments is a general failure mode for any long-horizon agentic loop, not just security tasks
SCoRe-style belief-state editing is cheap to implement and could be applied to non-security agentic tasks (debugging, research, planning)
Red teams should assume adversaries will use inference-time orchestration to push past apparent model limitations
More latest news
Read all