First Benchmark Tests Whether AI Agents Can Sneak Malware Past Real Endpoint Defenses — Spoiler: They Can

BypassGym pits 5 LLM agents against 6 real EDR products on live Windows VMs. DeepSeek-V4.1 Flash passes 17/18 runs. The offense-defense gap just got a measurement tool.

Imagine you're a locksmith school, and you've just built a standardized exam where students must pick six different commercial locks under timed conditions, with the locks fighting back — re-keying themselves, sounding alarms, calling the cops. You grade pass/fail on whether the door opens without anyone noticing. That's BypassGym: a structured gym environment where AI agents attempt to write shellcode loaders that execute command-and-control payloads on real Windows machines running real EDR products, and the EDR actively tries to block or terminate the payload. The committed claim is straightforward: this is the first benchmark that systematically evaluates LLM agents' ability to evade endpoint detection and response on real deployed systems. Prior offensive-security benchmarks like CyBench, HackTheBox, and PurpleLlama cover penetration testing and vulnerability discovery, but none measure the specific capability of generating evasion-capable loaders against commercial EDR. BypassGym fills that gap with a Proxmox VE infrastructure running actual Windows VMs, real EDR installs, and structured detection feedback loops. The architecture is an agentic loop: the LLM receives a C2 shellcode payload and must generate a loader (C/C++ source), which gets compiled and executed inside the VM. If the EDR blocks compilation, execution, or the callback, the agent receives structured feedback — which detection fired, at what stage — and iterates. The supplementary "evasion knowledge skill" injects domain-specific technique knowledge (API unhooking, syscall stubs, sleep obfuscation) so the benchmark can compare informed vs. uninformed agents. Five lightweight models were tested: DeepSeek-V3, DeepSeek-V4.1 Flash, Qwen3-32B, Qwen3-30B-A3B, and Gemma3-27B-IT. Six EDR products: Windows Defender, Kaspersky, ESET, Bitdefender, Elastic, and Symantec. Three payloads. The results are striking in their asymmetry. DeepSeek-V4.1 Flash succeeded on 17 out of 18 total runs (6 EDRs × 3 payloads). That's a 94% evasion rate from a lightweight model with agentic scaffolding and domain knowledge. On the defender side, Bitdefender was the hardest to crack — only 5 of 15 runs across models succeeded. Windows Defender, by contrast, was the most permissive. The evasion knowledge skill mattered: agents with it generally performed better, though the delta varied by model and EDR. Gemma3-27B-IT was the weakest performer, suggesting model capability still matters even with good scaffolding. Integrity is mixed. The benchmark runs real software on real systems — not simulations of EDR behavior — which is a genuine strength. But validation is entirely self-contained: the authors built the benchmark, chose the EDR products, ran the models, and reported the results. No independent team has replicated. The EDR product selection is reasonable but not exhaustive — notably absent are CrowdStrike and SentinelOne, two of the largest enterprise EDR vendors. Benchmark configurations (default EDR settings vs. hardened enterprise deployments) matter enormously and aren't fully explored. The code and environment specifications appear to be released, which helps. The milestone question is the most uncomfortable one. This benchmark establishes a baseline: lightweight LLMs today evade most consumer/SMB EDR at high rates. The next number to watch is whether frontier models (GPT-4o, Claude, Gemini) with full agentic scaffolding can crack hardened enterprise EDR configurations at >50% rates. If that number lands within 12-18 months, the defender investment thesis changes fundamentally — EDR vendors will need to assume AI-generated evasion is the default, not the exception. The obvious missing experiment is testing frontier-class models. The authors used only lightweight/open models (≤32B parameters). The honest read: this is partly responsible disclosure caution (demonstrating the capability with weaker models is less dangerous), partly compute budget, and partly strategic — if DeepSeek-V4.1 Flash already hits 94%, showing GPT-4o hitting 100% doesn't add much scientific value while adding substantial dual-use risk. The second missing experiment is longitudinal: how quickly do EDR signature updates neutralize previously successful loaders? Static benchmarks measure a snapshot; the real game is an arms race with a time dimension.