Imagine you're grading student essays by whether they fit in the envelope. An essay that's blank fits perfectly. An essay that's brilliant but one page too long gets an F. That's compile rate as a metric for LLM-generated security patches: it measures whether the output satisfies a mechanical gatekeeper, not whether the output is any good. This paper runs five controlled experiments to prove the point, and the results are damning. The core claim, stripped bare: compile rate is scientifically unreliable for evaluating single-function vulnerability repair by LLMs. The authors demonstrate this across 203 vulnerable C/C++ functions from the Big-Vul dataset, three open-source code LLMs spanning 350M to 6.7B parameters, and three prompting strategies. The headline number is brutal — about 64% of compile failures are attributable to evaluation-harness and dataset artifacts, not the model. That ratio barely moves across models. You're not measuring the student; you're measuring the envelope. Five failure modes stack up. First, an intervention that substantially improves code quality (better prompting) barely budges compile rate — the metric is insensitive to real improvement. Second, switching a single compiler-standard flag (like -std=c11 vs -std=c17) shifts compile rate by 1.8× to 2.7× on identical patches with zero regressions, meaning your choice of compiler flag matters more than your choice of model. Third, compile rate ranks the three models in the opposite order to reference-similarity metrics — the metric doesn't just add noise, it inverts the signal. Fourth, and most corrosively, optimizing for compile rate via a compiler-feedback loop actually degrades output quality: compile rate goes up while similarity to the human fix goes down. Manual inspection reveals deletion-style and placeholder non-repairs among the newly compiling outputs. The metric rewards gaming. The natural fallback metric, whole-function CodeBLEU, also collapses under scrutiny. An unchanged copy of the vulnerable input — literally returning the bug — outscores every model. This is because CodeBLEU computes similarity over the entire function, and the vulnerable version is overwhelmingly similar to the fixed version except at the edit site. The metric drowns the signal in shared context. The paper proposes diffF1 as a change-aware screen. It scores only the edited region, gives exactly zero credit to a no-op (returning the input unchanged), and near-zero credit to some deletion-based gaming patches. It's not positioned as a repair-quality metric — the authors are explicit about where it falls short — but as a cheap filter before execution-based analysis. That intellectual honesty is notable. Most papers proposing a metric oversell it; this one undersells deliberately. The broader argument matters more than any single finding: the LLM-for-security-repair subfield has been grading itself on a curve defined by toolchain noise, not model capability. Every paper that reports compile rate as its primary success metric is, to some degree, reporting the behavior of its evaluation harness. The field needs change-aware, execution-grounded evaluation, and it needs it before the next wave of "LLM fixes 80% of vulnerabilities" headlines lands. This is a measurement paper, not a capability paper, and measurement papers are undervalued. The code and data are public on GitHub. The experiments are straightforward to replicate. The findings are uncomfortable for anyone who has published compile-rate numbers as evidence of progress — which is exactly why the paper matters.
