You know when you weigh yourself three times on the same bathroom scale and get three different numbers? Now imagine publishing a ranked list of bathroom scales based on one reading each, calling one "best" and another "worst," and expecting readers to make purchasing decisions from the table. That is, roughly, what most LLM evaluation papers do — and this paper ran the experiment to show exactly how bad it gets. The committed claim: small-sample LLM evaluations produce rankings that look definitive but are not reproducible under basic statistical auditing. Using LLM-based prompt-structure inference as the test case — eight open models from five families spanning 8B to 675B parameters — the author persisted 293 raw intermediate representations and then stress-tested whether the resulting leaderboard holds up. It does not hold up. Mean node-set Jaccard similarity across identical calls ranges from 0.39 to 0.96, and 72% of prompt-model cells never achieve perfect node-set agreement. The phenomenon being measured is itself unstable, which means any single-pass evaluation is sampling noise on top of noise. The bootstrap results are the sharpest contribution. Under a joint cluster bootstrap over prompts, only the bottom of the ranking is firm: the two worst models hold rank in 99% and 86% of replicates. The middle four models hold rank only 27–48% of the time — literally worse than a coin flip. The top two models each hold rank in just 68% of replicates. You can reliably identify the worst performer. You cannot reliably identify the best. This asymmetry is important and under-discussed in the evaluation literature: leaderboards are better at finding losers than winners. The sensitivity analysis deepens the damage. Two equally defensible rules for merging repeated evaluation campaigns change four of eight model rows and shift the study-wide headline metric by 7 percentage points. That is not a rounding error — that is the difference between "promising" and "mediocre" in most evaluation contexts. And ground-truth annotations confirm that reproducibility cannot be read as accuracy: a model that gives the same wrong answer every time scores high on reproducibility but tells you nothing about correctness. The infrastructure decay finding is quietly devastating. Four of the eight API endpoints were withdrawn within ten weeks of measurement. The study as originally specified can no longer be run. This is not an edge case — it is the norm for any evaluation that depends on commercial or semi-commercial model endpoints. The leaderboard has a shelf life shorter than most review cycles. Methodologically, this is a self-audit rather than a new capability paper. The architecture is straightforward: call LLMs to infer prompt structure, persist raw outputs, then apply standard statistical machinery (Jaccard similarity, cluster bootstrap, sensitivity analysis over merge rules) to the accumulated evidence. There is no novel model or algorithm. The contribution is the discipline of turning the audit lens on the evaluation itself, which is rarer than it should be. The paper's limitation is also its strength: it studies one task (prompt-structure inference) on eight models with 293 persisted representations. The sample is small enough that the bootstrap results are themselves somewhat noisy — you are bootstrapping a bootstrap. But that is precisely the point. If even a careful audit of a small evaluation reveals this much instability, larger but less careful evaluations are likely hiding worse. The recommendations — report rank stability, per-cell provenance, sensitivity comparisons, raw outputs, and measurement dates — cost almost nothing to implement and would dramatically improve the field's epistemic hygiene.
