Imagine you're building a new bathroom scale, and you need to test it. You don't start by weighing people — you start with calibrated metal blocks of known mass. If the scale reads 5.000 kg for the 5 kg block, you trust it on humans. If it doesn't, you fix it before anyone steps on. This paper builds the calibrated metal blocks for probabilistic weather forecast verification. The committed claim: this is the first standardized, idealized 1D dataset designed specifically to probe the behavior and failure modes of verification metrics used for probabilistic forecasts. It doesn't propose a new verification method. It builds the test harness that every verification method should be run against — constant fields, localized events, gradients, fronts, noisy cases, bimodal distributions, and limiting cases. The ambition is diagnostic, not competitive. The context is urgent. AI weather models (GraphCast, Pangu-Weather, GenCast, FourCastNet) are proliferating alongside traditional physics-based ensemble systems. Both produce probabilistic outputs. The verification methods that score these outputs — CRPS, Brier score, reliability diagrams, rank histograms — have known quirks and blind spots, but no shared reference cases to reveal when a metric is misleading you. You can't meaningfully compare two scoring methods if you haven't tested both on cases where the right answer is known by construction. Architecturally, this is a dataset paper, not a methods paper. The underlying mechanism is controlled synthetic generation of 1D probability fields with known ground truth — think of it as a unit-test suite. The code is designed for flexibility: users can customize experiments, adjust parameters, and extend the case library. It belongs to the broader 'Bridging The Gap' project, which aims to build a comparison infrastructure for spatial verification of probabilistic forecasts. Integrity is unusual for this category. Because the dataset is idealized, there's no question of cherry-picked real-world benchmarks or stale baselines — the validation IS the construction. The risk instead is relevance: do 1D synthetic cases transfer lessons to real 2D/3D atmospheric fields with spatial correlations, terrain effects, and non-Gaussian structures? The authors are upfront that this is the 'first building block,' not the whole edifice. The milestone ladder here is about coverage, not accuracy. The current dataset is 1D. The next step — and the one that will determine whether this line of work matters — is extension to 2D spatial fields with realistic correlation structures, where verification metrics genuinely diverge in their recommendations. The gap between 1D toy cases and operationally relevant 2D/3D cases is where the interesting failures live. The experiment not run is the head-to-head: take five leading verification metrics, apply them all to every case in this dataset, and publish a comparison table showing where they agree and where they disagree. The authors built the test harness but stopped short of running the exam. Honest read: they're saving it — this is the infrastructure paper, the comparison paper is next, and it likely constitutes the main deliverable of Bridging The Gap.
