Your A/B Test and Your Benchmark Are Measuring Different Things — and They Can Disagree on Which Model Is Better

Formalizes why offline benchmarks and live A/B tests can rank LLMs differently. Users adapt, prompts shift, and the two evaluation regimes silently diverge.

Imagine you're a restaurant reviewer who rates every dish on a fixed menu. You rank Restaurant A above Restaurant B. But when real diners eat at B, they discover the chef riffs on their reactions — adjusting spice, pacing, complexity — and the diners reciprocate by ordering more adventurously. The fixed menu comparison and the live dining experience are measuring fundamentally different things. This paper is about that gap, formalized for LLM evaluation. The committed claim: offline benchmarks and live A/B tests converge to different statistical quantities — what the paper calls the 'common-distribution effect' and the 'induced-distribution effect,' respectively — and these two quantities can rank models in opposite order. This isn't a measurement noise problem you fix with more data. It's a structural divergence baked into the fact that users change their behavior in response to the model they're interacting with. The prompt distribution a model faces in production is partly a product of the model itself. Li defines the two estimands precisely. The common-distribution effect holds the prompt distribution fixed and asks which model performs better on the same tasks. The induced-distribution effect lets each model generate its own prompt distribution through user interaction and compares performance on those respective distributions. Standard difference-in-means estimators in benchmarks converge to the first; standard A/B test estimators converge to the second. The paper provides a sufficient condition — essentially a monotonicity requirement on how user adaptation interacts with model quality — under which the two rankings agree. When that condition fails, you get ranking reversals. The architecture here is statistical methodology, not machine learning engineering. Li works within the potential outcomes framework from causal inference, treating the user-model interaction as a dynamic system where the treatment (model assignment) affects the covariate distribution (prompts). A stylized linear model demonstrates that user adaptivity increases estimator variance — users exploring the model's capabilities add noise to your measurements. This connects to a broader family of endogeneity problems familiar to econometricians but underappreciated in ML evaluation. The integrity picture is mixed in an unusual way. This is a theoretical contribution — the validation is mathematical, not experimental. The proofs are self-contained and the framework is clean. But the practical implications rest on empirical questions (how much do users actually adapt? how often does the monotonicity condition hold?) that aren't tested. The linear model is explicitly stylized. There are no experiments on real user-LLM interaction logs demonstrating that ranking reversals actually occur at decision-relevant magnitudes. What makes this paper matter is the unification move. Reward hacking, novelty effects, and prompt-distribution shift are usually discussed as separate evaluation pathologies. Li shows they're all manifestations of the same mechanism: the model's responses change the distribution of tasks it faces. That reframing is the paper's real contribution — it gives practitioners a single lens for a family of problems they were previously debugging one at a time. The gap everyone should notice: there is no empirical demonstration. The obvious next experiment is to take real A/B test logs from a deployed LLM product, estimate both the common-distribution and induced-distribution effects, and show a ranking reversal in the wild. The honest read is that access to such data requires industry partnerships — this is a data-access constraint, not a conceptual one. The paper reads like the theoretical foundation for an empirical program that needs production-scale interaction data to complete.