Stochastic Diffusion Samplers Hit a Theoretical Wall — This Paper Finds a Way Around It Without Climbing Over

Higher-order math can't save stochastic diffusion sampling. Forward-value discretization sidesteps the problem with a lookahead trick that closes the gap with deterministic methods.

Imagine you're navigating fog with a GPS that updates every 30 seconds. You could buy a fancier GPS that predicts your position more precisely between updates — but if someone keeps randomly shoving you off the path every few seconds, the fancier prediction doesn't help much. The shoves dominate. That's the core finding here: stochastic noise injection in diffusion sampling caps your convergence at first-order no matter how clever your higher-order approximation. The authors prove this with matching upper and lower bounds. The committed claim: higher-order discretization is provably futile for improving convergence order in stochastic diffusion samplers, and a training-free "lookahead" trick — forward-value discretization — can nonetheless improve practical sample quality by evaluating the denoising model at a cleaner estimated point. This is two results stapled together: a negative impossibility result (higher-order doesn't help) and a positive engineering contribution (look ahead instead). The impossibility result is the theoretically interesting piece. The authors establish that even with exact score functions, stochastic samplers remain first-order convergent regardless of approximation order. This settles an open question that practitioners have bumped into empirically — people noticed that higher-order ODE solvers helped deterministic sampling (DPMSolver, DPMSolver++) but the same trick didn't translate to stochastic variants. Now we know why: the injected noise at each step creates an irreducible first-order error floor. The engineering contribution builds on this insight. Instead of trying to compute a more accurate Taylor expansion at the current point, F-DDPM evaluates the neural network at a "lookahead" estimate — a cleaner point that anticipates the noise that's about to be injected. Crucially, this reuses the previous model prediction and the previously injected noise, so it costs zero additional neural function evaluations per step. One NFE per iteration, same as DDPM. The deterministic variant (F-DPMSolver) applies the same idea to DDIM-type sampling. Results span CIFAR-10, ImageNet 64×64, FFHQ 64×64, and LSUN-Church 256×256 with pretrained models. F-DDPM consistently beats stochastic baselines (DDPM, analytic-DDPM, extended analytic-DDPM) at matched NFE budgets. On CIFAR-10 unconditional with 100 NFEs, F-DDPM achieves FID 3.65 vs DDPM's 4.73 and analytic-DDPM's 3.78. At 1.5× the NFE budget of deterministic baselines, F-DDPM reaches competitive quality — narrowing but not closing the stochastic-deterministic gap. The FID improvements are consistent but modest: typically 0.1–1.0 FID points over the best stochastic baseline. The validation is simulation-only, using pretrained checkpoints from prior work (VP-SDE on CIFAR-10, EDM on ImageNet/FFHQ/LSUN). The baselines are current and named — DPMSolver, DPMSolver++, DDIM, analytic-DDPM — and the authors are honest that deterministic methods still win at equal NFE. Code is released. The theoretical contribution is proved, not merely demonstrated. However, there's no independent replication, and the benchmark selection post-dates the method design. The practical upshot: if you want stochastic sampling (for diversity, uncertainty quantification, or because your application needs it), this is a free lunch — same cost per step, better FID, no retraining. If you just want the best images fastest, deterministic DPMSolver++ still wins at equal NFE. The real contribution is shifting the conversation: stop trying to make stochastic sampling converge faster through math; instead, be smarter about where you evaluate the model.