Imagine you're a dispatcher at a taxi company trying to estimate how long each ride will take — but every driver makes hundreds of small, partly random decisions at intersections (go straight? turn? stop for gas?). Even if you know the destination perfectly, the cumulative wobble from all those micro-decisions puts a floor on how precisely you can predict trip duration from the pickup location alone. That's the core mechanism of this paper: autoregressive decoding is a stochastic process where each token is a micro-decision, and the final output length is the sum of hundreds of these — which means, by the central limit theorem, per-prompt decode lengths are approximately normal with a variance that scales affinely with the mean. The committed claim: no prefill-based predictor can beat an irreducible error floor set by decoding stochasticity, and current benchmarks are broken because they don't control for this. The authors derive an explicit bound — the affine variance law σ = αμ + b, where α is the coefficient of variation of response segments — and show that five structurally different predictor architectures (MLP, EGTP, FiLM, soft-label, Mixture-of-Experts) all converge to approximately the same performance near this bound. System error dominates model error. If you're spending engineering cycles making a fancier predictor head, you're optimizing past the noise floor. This isn't the first paper to predict decode lengths from prefill states — that's an established subproblem in LLM inference scheduling. What's new is the theoretical treatment: modeling decoding as a stopped stochastic process, deriving the per-prompt normality result with an explicit error term, and then showing the residual departures (mild right skew, slight excess kurtosis) are exactly what the sampling-noise correction predicts. The theory and the empirics match to within a few percent on per-prompt MAE. On the ladder: the authors compare against the existing prefill-prediction approaches (EGTP being the most prominent) and show they all hit the same wall. This is not a "we beat SOTA by 3%" paper — it's a "we proved why everyone clusters at the same accuracy" paper. The honest framing is that classical approaches aren't losing to neural ones; everyone is losing to physics. The benchmark critique is sharp: reported accuracy collapses when you change the target model, dataset, or sampling configuration, which means published numbers are measuring task hardness, not predictor quality. The integrity picture is solid for a theory-plus-empirics paper. The normal-law predictions are validated against real decode-length distributions, the affine variance law is tested empirically, and five different predictor architectures serve as a quasi-independent replication. Code is released. The main limitation is that all validation uses the same team's experimental setup — no independent group has confirmed the bound or the convergence result. Pre-registration isn't applicable here, but the benchmark protocol they propose (fix model, data, sampling; report relative error) is itself a testable methodology. The practical punchline for systems builders: stop chasing marginal predictor accuracy for decode-length estimation. Instead, invest in scheduling algorithms that are robust to the irreducible prediction error the paper quantifies. The affine variance law gives you an explicit formula for how much uncertainty you'll face as a function of expected output length — use it to size your buffers and batching strategies. The paper doesn't solve scheduling, but it tells you exactly how much information prefill states can and cannot give you, which is the more valuable contribution.
