Imagine you're a weather forecaster who says there's a 30% chance of rain every day for a year. If it rains on roughly 110 of those days, you're calibrated. But what if you're only calibrated on average — perfectly tuned for Tuesdays, catastrophically wrong for weekends, and wildly overconfident for anything beyond 48 hours out? That's the problem this paper cracks open for time series foundation models: the aggregate scorecard looks fine while the per-horizon, per-condition reality is broken. The core claim is methodological, not algorithmic. Decker and Tresp aren't building a new forecaster — they're building a better ruler. They introduce a taxonomy of how TSFMs represent predictive uncertainty (quantile outputs, parametric distributions, sample-based) and then develop a fine-grained evaluation framework that distinguishes three calibration granularities: pooled (all horizons lumped together), horizon-wise (does the 1-step-ahead confidence interval differ from the 10-step-ahead?), and conditional (does calibration hold when you slice by input features or regimes?). The hierarchy matters because a model can pass pooled calibration while being dangerously miscalibrated at specific horizons or in specific regimes. The empirical results are where this gets uncomfortable for practitioners. The authors evaluate popular TSFMs — the paper names specific foundation models across diverse datasets — and find that current models exhibit "distinct and systematic patterns of miscalibration that standard metrics often obscure." Translation: the models you're deploying in production have uncertainty estimates that look reasonable on dashboards but break down exactly where you need them most — at longer horizons and in distributional tails. Architecturally, this is a benchmarking paper, not a methods paper. The evaluation framework sits on top of existing TSFM architectures (Transformer-based, patch-based, and hybrid designs) and tests their probabilistic outputs against increasingly strict notions of calibration. The key structural insight is that pooled calibration metrics — the ones everyone reports — can mask severe horizon-specific and regime-specific failures. This is the classic Simpsons paradox applied to uncertainty quantification. The integrity story is solid for an evaluation paper. The authors test across diverse datasets (not one cherry-picked domain), compare multiple popular models head-to-head, and apply their framework at multiple granularities rather than picking the one that tells the best story. They further analyze post-hoc recalibration methods (conformal prediction, Platt scaling, isotonic regression variants) and are honest about the limits: some forms of miscalibration respond to post-hoc fixes, others persist stubbornly. This is the kind of finding that's more useful than a SOTA claim. The practical upshot is immediate and concrete. If you're using TSFM uncertainty estimates for downstream decisions — inventory planning, risk management, capacity allocation — you should be evaluating calibration at the horizon level, not pooled. The paper provides the framework to do this. The gap it reveals is not a future problem; it's a current one hiding behind aggregate metrics. The obvious successor work the authors didn't run: training-time calibration objectives. This paper diagnoses the disease but treats it only with post-hoc bandages. Integrating horizon-conditional calibration losses directly into TSFM pre-training is the natural next step, and the honest read is they're saving it — this is a diagnostic paper accepted at NeurIPS Evaluations track, and the intervention paper is almost certainly in progress.
