Imagine two people following the same recipe in the same kitchen with the same ingredients, but one uses a slightly less precise measuring cup. For most steps, the difference is invisible — a fraction of a milliliter here or there. But at one critical moment, the recipe says "add the liquid that's closer to 200ml," and one cook measures 199.8ml in cup A and 200.1ml in cup B, while the other cook's imprecise cup reads them as 200.0ml and 200.1ml respectively. They pick different cups. From that single fork, the rest of the dish unfolds differently. That is what this paper proves is happening inside every large language model when you switch between BF16 and FP16 precision — and the divergence cascades. The committed claim: greedy decoding from LLMs is not precision-invariant. The same model weights, the same prompt, and the same greedy argmax algorithm produce different token sequences in BF16 versus FP16 on identical hardware. Across six models spanning 1.1B to 7B parameters (four architectural families, with additional characterization at 12B) and three benchmarks, 49–100% of prompts produce divergent outputs. This is not a sampling artifact or a temperature effect — greedy decoding is supposed to be deterministic, and it is, within a single precision. But between precisions, the accumulated floating-point rounding errors through ~22 transformer layers create perturbations that, at the final LM head, occasionally flip which token has the highest logit. The mechanism is elegant and well-characterized. The authors develop an empirical error-propagation analysis showing that the accumulated "body error" through the transformer layers does not distinguish steps where a flip occurs from steps where it doesn't. What matters is the margin: the gap between the top-two logit values at the LM head, relative to the directional perturbation between those two candidates. When the margin is tight, even tiny accumulated rounding differences can flip the argmax. Once a single token flips, the autoregressive feedback loop means the entire subsequent trajectory diverges — you're now conditioning on a different prefix. The body error is always there; the margin determines whether it's catastrophic. This analysis generates five testable predictions, and all five match experiment. The most counterintuitive: applying more FP32 compute at broader scope (e.g., recomputing more layers in higher precision) actually makes agreement worse, because it changes the error vector's direction relative to the logit margin rather than uniformly shrinking it. The authors don't just characterize the problem — they propose a mitigation. Selective FP32 LM head recomputation, triggered only when the top-two logit margin falls below a threshold, delivers +22–36 percentage points of exact agreement on A10G GPUs at under 4% latency overhead for low-batch inference. On L4 and A100, gains are +12–21 pp. The integrity picture is solid for an empirical ML paper. Six models across four families (including characterization at 12B), three benchmarks, four batch sizes, and three GPU types. The authors are admirably honest about the limits: the fix works at batch size ≤4 in single-stream inference, but the benefit vanishes at batch size ≥8 and under end-to-end FP8. They hypothesize that training-time precision stability is a determining factor — models trained with tighter precision discipline may be more robust — but don't test this directly. The paper is accepted at TMLR 2026, which means it survived a rigorous review cycle. The practical implications are immediate. Anyone running LLM inference across different hardware configurations, quantization schemes, or precision formats should assume they're getting different outputs — not just numerically different logits, but actually different text. This matters for reproducibility, evaluation, and any system where "same model, same prompt" is expected to mean "same answer." The selective recomputation trick is a useful partial fix, but the deeper lesson is architectural: if your deployment pipeline mixes precisions, you do not have a deterministic system, and no amount of downstream FP32 patching will make it one. The successor question is obvious: does this hold at 70B+ scale, where the body consists of 80+ layers and margins may behave differently? The authors stopped at 7B (with 12B characterization), almost certainly because of compute cost. The training-time precision stability hypothesis is the other dangling thread — a controlled experiment training the same architecture under different precision regimes and measuring downstream divergence rates would be definitive, but requires substantial training budget. Both are natural next-paper territory.
