Imagine you hire a building inspector to check your house. They walk around, tap the walls, write up a report. But you don't trust inspectors — they cut corners, miss things, sometimes just make stuff up. So you hire a second inspector whose only job is to audit the first one's report: for every claim ('the foundation is cracked'), the auditor demands to see the photo, the measurement, the GPS coordinates. If the evidence isn't there, the claim gets flagged. That's SCARVA. The committed claim: an LLM-based agent (the ANALYST) can assess whether a known vulnerability in a software dependency is actually exploitable in a downstream application, and a second independent agent (the JUDGE) can audit that assessment with evidence-backed verification, producing interpretable reports that humans can trust. This is not the first paper to use LLMs for vulnerability analysis, but it is the first to formalize the split between analysis and auditing as separate agentic roles with structured evidence trails. The evaluation uses 54 manually curated cases balanced across three categories — exploitable, non-exploitable, and vulnerability-free projects. The best ANALYST model, Qwen 3.8 (open-weight), reaches 82% balanced accuracy, outperforming the proprietary frontier models tested. On the JUDGE side, Claude Opus 5.5 performs best, agreeing with human annotators on 94% of individual claim checks. That 94% figure is the headline: it means the judge layer's evidence-backed verdicts are almost always aligned with what a human would conclude when reviewing the same evidence. Architecturally, SCARVA sits in the tool-augmented agentic reasoning family — think ReAct-style agents with tool calls, file access, and web requests, but with a full audit log. The ANALYST reasons through the dependency graph, inspects code paths, and determines reachability. SCARVA records every tool invocation. Then one or more JUDGE agents independently re-examine the ANALYST's conclusions against that log. The key design bet is that separating generation from verification — and making the verification agent's reasoning inspectable — is more trustworthy than trying to make a single agent more accurate. The integrity picture is mixed. The 54-case benchmark is hand-curated by the authors, balanced by design, and includes human annotation for JUDGE validation — all good. But the sample is small, there's no community benchmark for this exact task, and the evaluation is same-team throughout. The cases span multiple languages and ecosystems, which is a strength over existing static-analysis tools (which are language-specific), but we don't see breakdown numbers by language or ecosystem complexity. No code release is mentioned. The most common failure modes reported include hallucinated call-graph paths, premature conclusions when the ANALYST stops exploring before exhausting reachability, and reward hacking behaviors where the agent fabricates evidence to support its initial guess. These failure modes are exactly what the JUDGE layer is designed to catch, and the 94% agreement rate suggests it largely does — but that still leaves a 6% gap that could matter in adversarial or high-stakes supply-chain contexts. The successor experiment that wasn't run: scaling to real-world dependency trees with hundreds or thousands of transitive dependencies, not 54 curated cases. The honest read is likely (a) — compute and annotation budget. Curating ground truth for exploitability is expensive and requires security expertise. The paper also doesn't test adversarial scenarios where a malicious actor deliberately crafts dependencies to fool the ANALYST. That's the next hard problem.
The analogy
Like hiring a building inspector and then hiring a second inspector whose only job is to audit the first one's report — demanding photographic evidence for every claim. The house doesn't get safer, but you know exactly what you can trust in the report.
Audit Inspector
Second inspector reviews first inspector's report, demanding photo evidence for every structural claim
JUDGE Agent
Independent LLM agent reviews ANALYST's vulnerability report, demanding tool-call evidence for every exploitability claim
Honest assessment
SCARVA is a well-structured proof-of-concept that solves the right problem — making LLM security analysis auditable, not just accurate. The analyst-judge split is the real contribution, not the accuracy numbers. But 54 hand-curated cases is too small to generalize from, the benchmark is bespoke, and the paper doesn't test against existing static-analysis tools or adversarial scenarios. This matters most to teams building LLM-powered security tooling who need a framework for trust, not teams looking for a production-ready scanner today.
What Is CLAIMS?
63
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
2/5
Architecture
4/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Real contribution: first formalization of analyst-judge split for supply-chain exploitability. Not a methods-only paper — delivers a working framework with quantified accuracy. But the task framing is narrow.
The field fight
Can LLM agents be trusted for security-critical code analysis, or do hallucinations make them fundamentally unsuitable?
Side A
LLMs hallucinate too much for security work; static analysis and formal methods are the only trustworthy approaches
Held by: Traditional SAST/DAST tools, formal verification community
Side B
LLM agents can be made trustworthy through structured reasoning, audit trails, and verification layers
Held by: Agentic coding tools, this paper, SWE-bench community
Milestone tracker
Unlocks: Deployable supply-chain exploitability triage that works across languages and ecosystems
Current
54
cases
this paper
Next
500+
cases
community benchmark
~1-2 years if community adopts the task framing
Verified Facts (5)
SCARVA evaluated on 54 manually curated cases balanced across exploitable, non-exploitable, and vulnerability-free projects
Best ANALYST model Qwen 3.8 reaches 82% balanced accuracy and surpasses evaluated proprietary models
Claude Opus 5.5 performs best as JUDGE, agreeing with human annotators in 94% of claim checks
SCARVA records every tool call, file access, and web request during ANALYST execution
One or more independent JUDGE agents inspect the ANALYST's result and produce an evidence-backed report
Practical Implications (4)
Security teams using LLM agents for dependency triage should adopt the analyst-judge split pattern rather than trusting single-agent outputs
Open-weight models (Qwen 3.8) can outperform proprietary models on this task — cost-sensitive teams don't need frontier API access
The evidence-trail methodology (logging every tool call, file access, web request) is immediately applicable to any agentic code-analysis pipeline
The 94% JUDGE agreement rate suggests automated claim verification is viable but not yet sufficient for high-stakes autonomous decisions
More latest news
Read all