Martyn Thomas, Fellow and emeritus professor of IT at Gresham College, poses a question that the AI governance debate keeps dodging: what evidence would actually convince a regulator that a frontier AI system is safe? Not "aligned," not "responsible" — safe in the engineering sense, with quantified probabilities and auditable risk analyses. The trigger is OpenAI's withdrawal of a frontier model over internal safety test failures on 28 September. Thomas notes the familiar cycle: release, fail, withdraw, call for oversight. What never arrives is the specification of what oversight would look like in practice — what metrics, what thresholds, what burden of proof. Thomas draws on the established engineering tradition of safety-critical software regulation. Aircraft flight control systems, nuclear reactor controllers — these operate under international standards requiring formal "safety cases." A safety case is a structured argument, backed by evidence, that a system's probability of causing a catastrophic failure is vanishingly small. The bar for aviation: demonstrate with at least 99% confidence that a multi-fatality accident occurs no more than once per 1,000 years. That bar is already extraordinarily difficult to meet for aircraft, which are bounded physical systems with well-understood failure modes and decades of operational data. Frontier AI systems are unbounded, opaque, and operate in open-ended environments. The developers themselves claim these systems could, in the worst case, kill all of humanity. Yet none have produced the kind of rigorous, independently assessable risk analysis that a nuclear plant would be required to file before switching on. The asymmetry Thomas identifies is stark: the more catastrophic the claimed risk, the higher the evidentiary bar should be — yet frontier AI has the lowest evidentiary bar of any safety-critical domain. No detailed risk analyses exist. No safety cases have been filed. No one has even demonstrated that such safety cases could be produced for these systems. The gap between claimed danger and demonstrated safety is not narrowing; it is widening with each generation of model. Thomas's implicit argument cuts both ways. If the developers genuinely believe their systems pose existential risk, their failure to produce formal safety cases is reckless. If they don't believe the risk is real, the existential framing is marketing — a way to signal importance without accepting the regulatory consequences that would follow from genuine belief. Either way, the call for "independent oversight and regulation" without specifying what evidence that regulation would require is, as Thomas puts it, fantasy. The letter is short but its structural point is load-bearing: the AI safety debate has skipped the hardest step. Before arguing about who regulates, the field must answer whether regulation is even possible for systems whose failure modes cannot be enumerated and whose risk probabilities cannot be quantified. That question remains unanswered.
