Imagine you're buying a used car and the dealer shows you the average of all its inspection scores — engine, brakes, tires, frame — as a single number. A 78 out of 100 sounds fine. But what if the brakes scored 41? The average hid the thing that could kill you. That's the core mechanism this paper attacks: how averaging across risk benchmarks can conceal the specific failure modes that matter most. The committed claim is this: the authors have built the first open, traceable evaluation pipeline that maps 19 existing public benchmarks onto the four systemic-risk categories defined by the EU's General-Purpose AI Code of Practice — CBRN threats, cyber offense, harmful manipulation, and loss of control — and delivers an interactive dashboard where anyone can switch between average-case and worst-case aggregation. Across 18 models, switching from average to worst-case drops scores by 14 to 37 points. That delta is the paper's real finding: it quantifies how much information a single-number safety rating can bury. The architecture is straightforward pipeline engineering rather than algorithmic novelty. The team curates 19 public benchmarks (they don't build new ones), applies 'harm-preserving perturbations' to test inputs (prompt rewording, simulated deployment contexts), and feeds outputs through LLM-based judges for scoring. The LLM judges achieve Cohen's kappa of 0.78–0.82 against human graders, which is solidly in the range of human–human agreement on subjective tasks. A separate blind audit confirms that 83% of their prompt transformations preserve the original harmful intent — meaning the perturbations aren't accidentally defanging the tests. The ladder here is unusual because the paper isn't competing against a prior method on a performance metric — it's competing against the absence of a standard. The EU AI Act's Code of Practice defines risk categories but doesn't prescribe how to operationalize them into benchmark suites. Prior work like HELM, DecodingTrust, and individual CBRN evals exist as fragments, but no one has stitched them into a single framework aligned to the EU taxonomy with traceable evidence chains. The contribution is organizational and infrastructural, not algorithmic. Integrity is mixed. On the strong side: all 19 benchmarks are public, the dashboard is interactive and open, and the LLM-judge validation with human agreement statistics is a genuine check. On the weak side: N=21 for the usability survey is tiny and likely convenience-sampled (the paper doesn't describe recruitment), no pre-registration is mentioned, and the benchmark selection itself — which 19 and why not others — is a consequential choice that isn't fully defended. The 83% harm-preservation rate in the blind audit is encouraging but means 17% of transformations may not be testing what they claim to test. The milestone question is where this gets interesting. The EU AI Act's Code of Practice is live policy, and compliance tools are being built now. The Systemic Risk Index positions itself as a public-good alternative to proprietary evaluation suites that model providers might self-administer. The concrete next number: adoption. If 5+ major model providers or national regulators reference or integrate this pipeline within 18 months, it becomes infrastructure. If it stays academic, it's a demo paper. The obvious experiment the authors didn't run is longitudinal: track how model scores shift across versions (GPT-4 → GPT-4o → GPT-5, Claude 3 → 3.5 → 4) to see whether the risk landscape is improving, static, or worsening over time. This is almost certainly (a) — they ran out of time for a 6-page demo submission — and it's the feature that would make the dashboard genuinely indispensable to regulators rather than a snapshot.
