A 4B-Parameter Model Can't Find Vulnerabilities Alone — But It Can Tell You Where to Look First

Small open-weight decision models, wrapped in deterministic analysis harnesses, triaged 4,000+ C/C++ projects on commodity GPUs and surfaced 187 confirmed memory-safety findings — 160 of them critical or high severity.

Imagine you're sorting a warehouse of 10,000 unmarked boxes, and you know roughly 200 contain something dangerous. You don't have a scanner that can tell you which boxes are dangerous — your scanner is mediocre, barely better than a coin flip on the hard cases. But you do have a forklift driver who knows which aisles to check, and your scanner is good enough to say "check this aisle before that one." That ordering advantage is the entire contribution of this paper. The committed claim: a 4-billion-parameter open-weight decision model (Intern-Decision-4B), embedded in a deterministic harness of incomplete-fix miners and shape localizers, can prioritize vulnerability candidates well enough to find 187 confirmed memory-safety bugs across 4,000+ open-source C/C++ projects — running on three Intel GB10 systems or a single RTX 4090 gaming PC. No frontier model required. The model itself is weak: AUC 0.73 separating vulnerable from benign functions, a near-useless 0.56 distinguishing a vulnerable function from its own patch. Its value is entirely in queue ordering, not classification. The architecture is refreshingly honest about what does the real work. The harness — not the model — selects targets by mining incomplete fixes in version control and using shape localizers to narrow candidate functions. The decision model then scores every candidate in a single forward pass, returning numeric scores rather than text. This is a classifier-as-ranker design within the broader family of neural-guided program analysis, leaning on the insight that ordering a queue is easier than binary accept/reject. The throughput numbers are concrete: 900–1,100 repositories per hour on GB10 hardware, 4.4 decisions/second on GB10, 9.8 on RTX 4090. The ladder placement is the paper's most careful section and also its most revealing. On 75 public CVEs from projects outside the campaign, the model placed the fixed function in the top 25 of its repository only 21% of the time — just 1.4× better than random ordering. That's the honest, unbiased number. On the campaign's own queues, the model's ordering placed 7.5× more confirmed findings in the top 10 than random — but the authors correctly flag this as an observed, selection-biased upper bound. The gap between 1.4× and 7.5× is the gap between clean evaluation and operational reality, and the paper doesn't try to hide it. Integrity is mixed. The 187 findings are being disclosed under coordinated disclosure before publication, which is the right process. The CVE-based evaluation on external projects is a reasonable independent check. But the campaign's own findings are inherently selection-biased — the harness found what the harness was designed to find. There's no pre-registration, no independent replication, and the CVSS severity breakdown (12 critical, 148 high, 23 medium, 4 low) skews heavily toward high severity in a way that likely reflects the localizers' bias toward memory-safety patterns rather than a representative sample of all bugs. The milestone question is where this gets interesting for practitioners. The paper demonstrates that commodity hardware can triage thousands of repositories per hour with a small model. The next concrete threshold: if model discrimination improved from AUC 0.73 to ~0.85 on the vulnerable-vs-benign task (and from 0.56 to ~0.70 on vulnerable-vs-patched), the 1.4× ordering advantage on unseen CVEs would likely cross 2–3×, making the system competitive with much heavier static-analysis pipelines. That's a model-quality problem, not a systems problem. The obvious experiment the authors didn't run: swapping Intern-Decision-4B for a larger open-weight model (say, 13B or 70B parameters) or a frontier model to measure how much discrimination improves with scale. The paper explicitly says "the search uses no frontier model," which reads like a deliberate constraint — either because they want to prove the small-model thesis, or because early experiments with larger models didn't improve discrimination enough to justify the cost. Given the paper's emphasis on commodity hardware, the honest read is probably (c): they're saving the scaling study for the next paper, where it becomes the headline.