The Crypto Agility Maturity Models That NIST Cites Don't Actually Measure What They Claim To

Three industry-driven maturity models for cryptographic transitions fail basic design-science criteria. The tools guiding post-quantum migration lack reproducible metrics and prescriptive guidance.

Imagine you're grading students with a rubric that has no point values — just vibes. 'Shows understanding' but no definition of what understanding looks like, no threshold for passing, no way for two graders to arrive at the same score. That's the state of crypto agility maturity models right now: organizations are being told to assess their readiness for post-quantum cryptography using instruments that can't produce reproducible results. The committed claim here is diagnostic, not constructive: the three industry-driven maturity models that NIST references in its post-quantum migration guidance all fail established design-science quality criteria, particularly around measurable and reproducible assessment and prescriptive improvement pathways. This is a meta-assessment — a maturity model for maturity models, hence the paper's title. The authors apply design-science principles from the maturity model literature to grade CHQS, Quantum-Safe Cryptographic Maturity, and related frameworks that organizations are actually deploying today. The architecture of the analysis is straightforward qualitative evaluation: take established design principles for maturity models (from the IS literature — Becker et al., Pöppelbuß & Röglinger, de Bruin et al.), operationalize them as assessment criteria, and systematically score each of the three crypto agility maturity models against those criteria. This is not a quantitative benchmark — it's a structured expert review applying known quality gates. The method leans on design-science rigor rather than empirical data or simulation. The ladder question is interesting because there's essentially no prior systematic evaluation to compare against. NIST's own guidance (IR 8547, CSWP 39) references these models without evaluating their design quality. The authors are doing what NIST didn't: holding the recommended tools to the same rigor standard that the maturity model research community has established over two decades. The finding that all three models lack measurable, reproducible criteria is damning precisely because these are the instruments organizations will use for compliance and governance planning. On integrity: the validation is expert analysis against established criteria, not independent experiment. The authors are transparent about the qualitative nature of the evaluation. The risk is subjectivity in applying the design principles — two different teams might weigh individual criteria differently. However, the criteria themselves are well-established in the IS community, and the recurring weaknesses the authors identify (non-reproducibility, lack of prescriptive guidance) are structural enough that they'd likely survive independent review. No code or dataset to release; the artifact is the evaluation framework itself. The milestone that matters isn't a number — it's whether the next generation of crypto agility maturity models incorporate measurable, reproducible assessment criteria and prescriptive improvement paths before organizations commit to post-quantum migration plans based on the current flawed instruments. With NIST's PQC timeline pushing organizations toward transition now, the window for fixing these tools is narrow. The practical question: will NIST update its references or will organizations plan billion-dollar migrations using rubrics with no point values? The obvious successor experiment the authors didn't run: empirically testing inter-rater reliability by having multiple independent assessors apply each maturity model to the same organization and measuring agreement. This would convert the structural critique ('these criteria aren't reproducible') into a quantified finding ('assessor agreement was κ = 0.3'). The honest read is (a) — this requires organizational access and coordinated multi-assessor studies that a single academic team can't easily arrange, and it's the natural next paper.