Imagine you're teaching someone to pick a lock. You show them the mechanism diagram, explain the pin-tumbler system, hand them a tension wrench and a rake. Every student can describe how the lock works after the lesson. But only some of them can actually feel the pins set and open it. That gap — between knowing the vulnerability and actually crafting a working exploit input — is the core finding of this paper. The committed claim: open-weight LLMs small enough to run on a single consumer GPU can already reproduce most software vulnerabilities from a description alone, but the harder tasks require models at the 27B-parameter scale. The authors evaluate twelve configurations across three model generations (Qwen2.5-Coder 7B and 14B, Gemma3 4B/12B/27B, and QwQ 32B as a MoE), three quantization levels (FP16, Q8, Q4), and ten CyberGym vulnerability-reproduction tasks, running each configuration three times under a single agent harness. Six of the ten tasks fall to every configuration, including the 4B model. The remaining four only yield to the 27B-class model. What makes this interesting is the trajectory analysis. The authors examined agent execution traces and found that almost every run — including failures — correctly identifies the vulnerable function and states the right hypothesis about how to trigger it. The difference between a pass and a fail isn't diagnosis, it's synthesis: turning a correct hypothesis into a working crash input. This is a meaningful finding because it locates the bottleneck not in reasoning or code comprehension but in the mechanical act of crafting and iterating on a concrete exploit payload. On the ladder: there is no strong prior SOTA for this exact task formulation. CyberGym is a relatively new benchmark, and the closest prior work uses hosted frontier models (GPT-4-class) or substantially larger open-weight models. The paper does not benchmark against any hosted frontier model directly — an important absence. It compares across its own twelve configurations, which gives useful internal signal about scaling behavior but leaves the question open of where these local models sit against, say, GPT-4o or Claude on the same tasks. Quantization is the pleasant surprise. Across all matched pairs (same model at FP16, Q8, and Q4), no quantization level changed which tasks were solved or unsolved. It changed wall-clock time — the quantized models ran faster — but not capability. The MoE architecture (QwQ 32B) matched the dense 27B Gemma3 on solve rates while using less wall-clock time per run, suggesting that mixture-of-experts is a real efficiency gain for agentic security tasks, not just a training trick. The integrity picture is honest but limited. Ten tasks, three runs each — that's a thin statistical envelope. The authors acknowledge this and release the full artefact (prompts, trajectories, harness code) via a Hugging Face DOI, which is good practice. But there is no independent replication, no pre-registration, and the benchmark selection (CyberGym) is relatively new and not yet a community standard at the scale of, say, LMSYS or SWE-bench. The validation is the authors grading their own homework against a benchmark they chose after model selection. The practical takeaway for security teams: if your threat model includes attackers with consumer-grade hardware and a 27B open-weight model, assume they can reproduce known vulnerabilities from CVE descriptions automatically. The defense implication is that patch latency — the time between disclosure and deployment — is now the critical variable, not attacker skill. The offensive implication is that local, private, audit-trail-free vulnerability reproduction is now within reach of a motivated individual with a $1,000 GPU.
