Imagine you have a friend who's superb at predicting where a thrown ball will land — they can read the arc perfectly — but consistently misjudges how hard it will hit. That's the state of AI weather models on tropical cyclones: extraordinary at tracking where storms go, still unreliable on how strong they'll be when they arrive. This paper is the first systematic probabilistic stress-test of that split. The claim is straightforward but important: leading AI weather models (GenCast, FourCastNet 3, FGN) don't just match but substantially outperform ECMWF's operational physics-based ensemble (IFS-ENS) on cyclone track prediction. FGN achieves better probabilistic location scores for 90% of evaluated storms in 2025. This isn't a cherry-picked subset — it's an extensive benchmark across a full storm season. The authors built a custom suite of probabilistic metrics rather than relying on deterministic point comparisons, which is where most prior AI-vs-physics cyclone studies stopped. The ladder is honest and well-constructed. IFS-ENS is the real operational baseline — this is what national weather services actually run. The paper names it, compares against it head-to-head, and is forthcoming about where AI models still lose: intensity prediction and ensemble spread (the model's own uncertainty estimate) remain systematically biased. AI models underpredict peak intensity and produce ensembles that are too tight — they're overconfident about storm strength even when the track is correct. Architecturally, the three AI models span distinct families: GenCast is a diffusion-based probabilistic model, FourCastNet 3 uses adaptive Fourier neural operators, and FGN is a graph neural network approach. All are trained on ERA5 reanalysis data and produce ensemble forecasts, but through fundamentally different generative mechanisms. The comparison across these families is what gives the benchmark its analytical depth — track skill appears robust across architectures, while intensity bias varies in model-specific ways. The integrity picture is solid for an evaluation paper but has natural limits. The authors use real observed cyclone data (IBTrACS) as ground truth, not model-generated targets. They develop bespoke metrics for probabilistic assessment including CRPS on track, intensity biases, and wind-pressure relationship diagnostics. The main caveat: this is a same-team evaluation with metrics designed for the paper. There's no pre-registration, and the metric suite — while thoughtful — was developed alongside the results. The finding that pretraining data visibly influences model failure modes is an unusually candid diagnostic. The milestone framing is implicit rather than explicit, but the trajectory is clear. Track prediction is effectively solved for operational purposes — 90% storm superiority is decisive. The open frontier is intensity: current AI models still cannot reliably distinguish a Category 2 from a Category 4 in ensemble forecasts. Closing that gap would make AI models operationally deployable for evacuation decisions, not just track warnings. The obvious next experiment not run: fine-tuning or retraining these models specifically on tropical cyclone-dense data to address the intensity bias. The authors note the "lingering influence of pretraining data" as a likely culprit, which strongly suggests targeted fine-tuning is the natural next step. My read: this is being saved for a follow-up paper, not a compute limitation — the diagnostic analysis in this paper practically writes the recipe for that experiment.
