Imagine you're comparing batting averages across baseball leagues. A player hitting .320 in one league and .310 in another seems straightforwardly comparable — until you realize the leagues have different distributions of pitcher quality, ballpark sizes, and schedule structures. The overall league average isn't just a weighted average of team averages; it's warped by the composition of who faces whom. Now replace batting averages with AUC (the c-statistic), leagues with demographic subgroups, and you've got the core problem this paper dissects: the metric everyone uses to check if clinical AI models are 'fair' across groups doesn't actually work the way people assume it does. The committed claim: AUC is non-collapsible, meaning the overall AUC is not a simple function of subgroup-specific AUCs, and three widely cited AUC-based fairness metrics — gap-to-overall AUC, equity-scaled AUC, and minimum-based metrics — are mathematically comparing quantities that should not be compared. They are not fit for purpose. This isn't a minor quibble about interpretation; it's a structural invalidity in metrics that regulatory and clinical governance teams are actively using. The paper examines eight AUC-based fairness metrics from the literature and maps each one to its estimand — what it actually measures, formally. The key decomposition: overall AUC breaks into within-group AUC components (how well the model discriminates within each subgroup) and cross-group AUC components (how well the model discriminates between individuals from different subgroups). Within-group AUC parity — simply comparing AUC across subgroups — survives the authors' scrutiny as the recommended metric. Cross-group AUC (xAUC) parity provides supplementary signal but conflicts with calibration, a well-known impossibility result in fairness research. The xAUC¹ and xAUC⁰ metrics have theoretical merit as single-number summaries of within- and cross-group components, but their interpretation is non-trivial. The architecture here is pure statistical methodology — no ML training, no new algorithms. The paper leans on the mathematical properties of rank-based statistics and their behavior under mixture distributions. The simulation study uses EuroSCORE II, a real clinical prediction model for in-hospital mortality after cardiac surgery, to illustrate how the eight metrics behave under controlled conditions where you know the ground truth. This is the right approach: you need a setting where you can manipulate the data-generating process to isolate what each metric is actually responding to. Integrity is solid for what this is. The validation is mathematical characterization plus simulation, not empirical benchmarking against a held-out dataset. The authors are transparent about non-collapsibility as the central technical issue and don't oversell. The EuroSCORE II simulation grounds the theoretical results in a realistic clinical context. No code availability is mentioned, which is a gap — the simulation setup should be reproducible but isn't verifiably so from the abstract alone. The baselines here aren't competing methods but competing metrics, and the authors engage with all eight rather than cherry-picking a weak straw man. The practical stakes are significant. Clinical AI fairness auditing is a growing regulatory requirement — the EU AI Act, FDA guidance on clinical decision support, and UK MHRA frameworks all gesture toward fairness evaluation. If governance teams are using gap-to-overall AUC or equity-scaled AUC to certify model fairness, they're using broken rulers. The paper's recommendation — report the full AUC matrix (overall plus all within- and cross-group components) alongside population characteristics — is the kind of concrete, implementable guidance that changes practice rather than just advancing theory. The obvious successor experiment: apply this framework to real multi-site clinical data with genuine demographic subgroups and show that the broken metrics lead to materially different fairness conclusions than the recommended ones. The authors likely didn't run this because the paper's contribution is methodological — establishing which metrics are valid before applying them — and because real multi-site clinical data with sufficient subgroup sizes is hard to access and publish on. But the field needs that empirical demonstration to move from 'these metrics are theoretically wrong' to 'using them led to a real fairness failure in deployed system X.'
