Imagine you hired a building inspector who catches every code violation — every single one — but also condemns 94% of buildings that are perfectly up to code. That's not a good inspector. That's a liability. This paper builds the test that finally measures the difference. The core claim: LLM-based backtest auditors, as currently prompted, cannot distinguish flawed quantitative-finance backtests from methodologically clean ones. The authors construct a 96-item benchmark — 48 flawed strategies paired with 48 clean controls — where each pair holds everything constant (strategy logic, date ranges, code style, reporting structure) except one methodology detail. This paired design is the paper's real contribution. It means a model can't game the score by just flagging everything. Over 1,440 cached audits across four text endpoints, the flagship DeepSeek auditor hits 100% recall on flawed backtests in closed-prompt mode — it catches every rigged test. But on open prompts, it flags 93.8% of clean controls as flawed too. That number is devastating. In a real audit pipeline, you'd be drowning in false alarms, and the signal — the actual rigged backtests — would be indistinguishable from noise. Reporting recall alone, as the field currently does, would rank three of the four tested models identically. Adding the clean-control false-positive rate separates them by 79 points. The engineering fix is instructive: a single clean-aware prompt warning drops DeepSeek's code false-positive rate from 20.8% (95% CI: 11.7–34.3) to 0.0% (0.0–7.4) with no loss in recall. The budget anchor model, by contrast, still flags 38 of 48 clean controls under the same prompt — the warning doesn't help a weaker model. This is a clean demonstration that prompt engineering can substitute for model capability, but only above a capability threshold. The benchmark's deterministic scorer decomposes performance into four orthogonal axes: flaw recall (did you catch it?), clean-control false positives (did you cry wolf?), evidence localization (did you point to the right line of code?), and fix relevance (did your suggested repair actually address the flaw?). This decomposition matters because a model can score perfectly on one axis while being useless on another — and the field has been reporting only the first axis. Architecturally, this isn't a new model or a new training method. It's an evaluation infrastructure paper in the tradition of BIG-bench and HumanEval — but domain-specific to quantitative finance. The 96-item benchmark is small by NLP standards but large by quant-finance audit standards, and the paired design gives it statistical leverage that unpaired benchmarks three times the size wouldn't have. The integrity story is mixed. The authors test four models and report all results, including embarrassing ones. The scorer is deterministic and the benchmark is described in enough detail to reproduce. But the benchmark is author-constructed, not drawn from real audit failures, and there's no independent replication. The 48-pair size means confidence intervals are wide — that 20.8% false-positive rate has a CI spanning 11.7% to 34.3%. This is honest reporting, but it means the point estimates should be held loosely.
