Imagine two basketball scouts watching the same player. One counts made shots; the other counts shot attempts. The aggressive shooter looks great to the second scout and terrible to the first. The cautious shooter — reverse. Neither scout is wrong, but their rankings are incompatible. That's what's happening in AI weather forecasting for extreme events: the metric is the ranking. The committed claim here is that conflicting conclusions about AI weather models' ability to forecast extremes are not contradictions in the science — they're artifacts of evaluation protocol choices. Within a single controlled experimental design, the authors show that swapping from event-conditional evaluation (did you get the extreme right when it happened?) to weighted scoring functions (how well-calibrated are your extreme-tail probabilities overall?) flips the model rankings. The physics-based model wins the first contest; AI models win the second. The mechanism is elegant and diagnostic. A contingency table decomposition reveals that AI models systematically underpredict extreme frequency — their errors are dominated by misses. Physics-based models overpredict extremes — their errors are dominated by false alarms. These are structurally different failure modes. Metrics that punish misses more heavily (like restricting evaluation to observed extremes) penalize AI. Metrics that reward well-calibrated probability assignments across the full distribution favor AI's smoother, less trigger-happy outputs. Score decompositions confirm the root cause: the performance gap between AI and physics-based models arises primarily from miscalibration, not from differences in discrimination ability. Both model families can distinguish extreme-prone situations from calm ones at roughly comparable skill levels. But AI models assign systematically wrong probabilities to those extremes — too low, too rarely. This is a calibration problem, not a pattern-recognition problem, and that distinction matters enormously for what comes next. The practical implication is immediate and operational: AI weather models are already used in production. Agencies selecting or benchmarking these models are making metric choices that determine which model appears best. If you're a national weather service evaluating whether to trust an AI model for extreme event warnings, the evaluation protocol you use is load-bearing infrastructure, not a methodological footnote. This paper doesn't tell you which metric to use — it tells you that the choice is consequential and currently under-examined. The integrity profile is solid for a workshop paper. The authors use a common experimental design rather than cherry-picking favorable setups for either model family. The contingency table and score decomposition approaches are well-established in forecast verification literature (Brier score decomposition, Murphy diagrams). The limitation is scope: the abstract references "the considered physics-based model" (singular) and unspecified AI models. We don't know how many models, which ones, or what ERA5-based reanalysis data underpins the comparison. The field fight this paper enters is real and consequential: the AI-weather community is split between researchers who see AI models as having already achieved parity on extremes and those who see fundamental deficiencies. This paper argues the disagreement is partly illusory — an artifact of measurement, not of model capability. That's a deflationary but clarifying contribution. The next milestone isn't a better model; it's a community-agreed evaluation protocol for extreme weather that makes model rankings stable across reasonable metric choices.
