Your Standards Are Rising Faster Than Your AI: 1.3M Chatbot Arena Votes Prove Users Are Getting Pickier, Not Models Getting Dumber

Over 1.26M blind votes on fixed-weight models show "both are bad" rising ~1.9 pts/month — while relative rankings hold steady. The goalpost moves with the field.

Imagine you buy a TV in 2020 and love it. In 2023, you visit a friend's house, see a newer display, and suddenly your TV looks washed out. Nothing changed about the panel — your reference point shifted. That's the core mechanism this paper isolates for LLM evaluation, and it matters because the entire RLHF pipeline and every public leaderboard depends on human judges whose internal yardstick is silently drifting. The committed claim: human evaluation standards for LLMs are rising over time, independent of model changes. Using 1,260,967 blind pairwise votes on Chatbot Arena from May 2023 to August 2024, the authors show that for model pairs whose weights are provably frozen, the share of "both are bad" verdicts rises at 1.71 percentage points per month against a baseline mean of 17.1%. In 2024 alone, the rate accelerates to 1.91 points per month. Across 14 pairs observed frequently in both years, the "both are bad" rate rose in every single case — from 15.5% to 23.3% — while the relative ranking between the two models in each pair remained flat. The architecture of this study is observational causal inference, not ML engineering. The authors exploit a natural experiment: Chatbot Arena assigns users random pairs of anonymous models, some of which have version-locked weights (open-source models, dated vendor snapshots). This lets them hold the model constant and measure whether the human side of the equation drifts. They use fixed-effects panel regression, controlling for voter identity, prompt content, interface changes, language, model answer length, and voter arrival cohort. The key structural move is separating absolute judgment ("both are bad") from relative judgment (which model wins), then showing they diverge. Integrity is the paper's strongest suit. The dataset is massive (1.26M votes), the platform is public and widely used, and the authors run an impressive battery of robustness checks: within-voter fixed effects rule out compositional change in the user base, prompt-level controls rule out harder questions over time, cross-model month effects rule out individual model decay. The 14-pair year-over-year comparison is particularly clean — same frozen models, every pair showing the same direction. The weaknesses are that Chatbot Arena users are self-selected tech enthusiasts (not representative of the general public), and the authors cannot directly observe what external AI experiences shifted each voter's standards. The ladder context here is novel. This isn't competing with another measurement paper on the same metric — it's identifying a bias that contaminates the metric itself. Prior work (Chen et al. 2023 on GPT-4 degradation, Liang et al.'s HELM benchmark) documented apparent model declines but couldn't distinguish model change from evaluator change. This paper cleanly separates the two by exploiting frozen weights. The closest methodological precedent is the hedonic treadmill literature in psychology and the reference-dependent utility models in behavioral economics — the authors cite neither, which is a missed connection. The practical implications are sharp. If human preferences drift at ~2 percentage points per month, then RLHF reward models trained on older preference data are calibrated to a standard that no longer matches current users. Leaderboards that aggregate votes across time are mixing incompatible yardsticks. And the widespread user complaint that "ChatGPT got dumber" may be, in substantial part, a measurement artifact rather than a product failure. This doesn't mean models never degrade — but it means you can't tell degradation from standard-raising without version-controlled comparisons. The gap the authors don't address is the causal mechanism behind the drift. They demonstrate the drift convincingly but can only speculate about what drives it — exposure to frontier models, growing familiarity with AI capabilities, changing task expectations. A follow-up that tracks individual users' external model exposure (which services they use, which frontier models they've seen) would nail the mechanism. The honest read is that this data doesn't exist in Chatbot Arena logs, so the authors couldn't run it — it would require a purpose-built longitudinal study with usage tracking consent.