Can an LLM Pipeline Catch Corruption Risks in Draft Laws? This One Matches a Human Agency 28-46% of the Time.

A hybrid AI system tries to replicate Ukraine's anti-corruption reviewers. It finds real risks but misses entire categories — and nobody's checked whether the humans agree with the AI judge grading its own work.

Imagine you're a building inspector with a checklist of 100 known code violations. You walk through a new building and flag everything that looks wrong — but you're using a flashlight, not floodlights. You catch a lot of obvious stuff, you misidentify some violations (calling a fire-exit problem a ventilation problem), and entire categories of structural weakness — the ones that only show up when you cross-reference two different blueprints — you miss completely. That's this system: a flashlight inspector for legislative corruption risk. The committed claim: a modular pipeline combining deterministic legal-text parsing, LLM-based single-provision risk classification, rule-based factor assignment, and human oversight can replicate 28-46% of the findings that Ukraine's National Agency on Corruption Prevention (NAZK) produces when reviewing draft legislation. The system uses the TRACE framework (Transparency, Regulation, Accountability, Competence, Enforcement) to categorize corruption risks. This is not a claim of replacing human experts — it's a claim that automated screening can surface a meaningful fraction of the same risks, faster. The ladder here is unusual because there's no established ML baseline for anti-corruption legislative review. The authors are competing against NAZK's own published conclusions — 132 draft laws with 474 model-extracted findings. On 100 draft laws, the local Gemma 4 31B model hits 0.28 agreement in top-10 flags (strict), Sonnet 5.5 reaches 0.30, and the full flag set reaches 0.43-0.46 under non-blind conditions. The frozen blind baseline sits at 0.21 (95% CI 0.16-0.26). On 21 held-out acts, the pipeline outperforms a monolithic prompt on strict measure but not on the lenient one. Opus 5.5 adds nothing over Sonnet. These numbers are honest but modest — less than half the findings are matched even in the best case. The architecture is a four-stage pipeline: deterministic parsing splits legislation into individual provisions, an LLM qualifies each provision against TRACE risk categories, a rule engine assigns corruption-risk factors, and a human-in-the-loop layer provides oversight. This is deliberately not end-to-end — the authors bet on composability over monolithic prompting, and the held-out comparison shows this pays off under strict matching. The LLM judge that evaluates agreement uses two access routes that agree on 73 of 74 labels, but there's a critical circularity: the 474 findings themselves are model-extracted from NAZK conclusions, and the agreement judge is also an LLM. No human expert has validated any of this. The integrity picture has a conspicuous hole. The benchmark is constructed from real government output (strong), the code evaluates multiple models including a local open-weight option (good), and the authors are transparent about the blind vs. non-blind gap and the complete failure on all 32 legal-collision findings (admirably honest). But the entire evaluation chain — extraction of ground-truth findings, matching, and agreement scoring — is LLM-mediated without human expert validation. The authors acknowledge this explicitly, which earns credit, but it means we don't know whether the 0.43-0.46 agreement rate reflects genuine corruption-risk detection or shared LLM biases in interpreting legal text. The most telling failure mode: most misses occur on provisions the system located but classified under the wrong risk category, and zero of 32 legal-collision findings were matched. Legal-collision risks — where corruption potential emerges from the interaction between a new bill and existing law — require cross-document reasoning that single-provision qualification architecturally cannot perform. This isn't a tuning problem; it's a design ceiling. The next real milestone is cross-provision and cross-statute reasoning that can detect emergent corruption risks, which likely requires retrieval-augmented architectures over the full legislative corpus. The obvious experiment not run: human expert validation of the pipeline's output. The authors built the benchmark from NAZK conclusions and used LLM judges to score agreement, but never asked a human anti-corruption expert to evaluate whether the system's flags are actually useful, whether its misclassifications matter in practice, or whether it surfaces risks NAZK missed. The honest read: this is a compute/access constraint — getting Ukrainian anti-corruption experts to systematically validate AI output on 100+ draft laws is expensive and logistically difficult, especially in wartime. But without it, the entire evaluation rests on LLM-judges-judging-LLM-output, which is a known fragility in the field.