Imagine you hire a plumber who fixes the pipe under your sink perfectly — no leaks, passes every pressure test — but the moment you turn on the dishwasher and the washing machine simultaneously, the whole system floods. That is the gap SWE-Serve is designed to measure. The paper's committed claim: existing benchmarks let AI coding agents look competent by testing pipes in isolation, while nobody checks whether the whole house still works. SWE-Serve is a 53-task benchmark grounded in real commits to SGLang, an open-source inference serving framework. Each task requires coordinating changes across the serving stack — model support, runtime execution, public APIs — the kind of multi-file, multi-layer engineering that production inference work actually demands. Tasks fall into six families and run on either CPU or a single H100 GPU. The evaluation uses hidden functional tests, regression tests, and crucially, end-to-end serving tests with calibrated performance gates. The headline finding is brutal in its simplicity. On the 19 tasks with full E2E test coverage, models that achieved 69.4% pass rates when E2E tests were excluded dropped to 45.9% when those tests counted. That is roughly one-third of seemingly correct patches failing when evaluated against production-grade correctness. The best overall configuration across 11 models and 31 model-effort combinations hit 75% mean pass@1, which is strong but still leaves a quarter of tasks unsolved even under the most favorable conditions. The evaluation infrastructure is more careful than most benchmarks in this space. They include no-op controls (confirming tasks actually require changes) and oracle controls (confirming the original developer patches pass). Adversarial verifier review catches false positives. Closed-book execution prevents data leakage from training on the SGLang repo. These are not afterthoughts — they are structural integrity features that many benchmarks skip. The ladder position is clear but modest. SWE-Serve is not competing against another inference benchmark because there really is not one at this scope. SWE-bench covers general software engineering; terminal-agent benchmarks like SWE-bench Verified include a few inference tasks; kernel-generation benchmarks like KernelBench focus on isolated optimization. SWE-Serve occupies the specific gap of repository-scale production inference engineering. The contribution is the benchmark itself, not a new agent architecture. The architectural lens here is the evaluation methodology rather than a model family. The key design choice is layered testing: functional correctness, regression safety, E2E serving validation, and performance gates form a gauntlet where each layer catches failures the previous one misses. The 33% rejection rate at the E2E layer is the paper's central empirical contribution — it quantifies what practitioners already suspect but could not measure. The obvious missing experiment is longitudinal tracking: running the same benchmark as SGLang evolves, measuring whether agents improve on production correctness over time or whether the gap persists as tasks get harder. The authors frame SWE-Serve as a tracking instrument, but this paper provides only the baseline snapshot. My read: this is being saved for follow-up work — the benchmark's value proposition depends on showing temporal trends, and publishing the baseline first is the standard playbook.
