Imagine you're proctoring an exam where every student sits in a cubicle and can only see their own paper. You've verified the cubicle walls are solid. But what you missed is the PA system — a teacher's summary of the full answer key is being broadcast over the intercom before each question. Every student hears it. The walls are fine; the information channel you didn't audit is the problem. That's exactly what happened here: the causal attention mask was correct, but a sequence-level routing summary in an adaptive decoder broadcast future-token information to every position through a side channel. The committed claim: adaptive residual-routing decoders can leak future tokens through their routing-query path even when the causal attention mask is perfectly implemented, and this leakage produces validation loss improvements that are entirely spurious. Specifically, an anomalous 24-layer checkpoint dropped validation loss from 3.5958 to 1.4259 — a massive apparent gain — but failed a prefix-invariance audit and showed substantially higher reference-model perplexity during actual generation. The model looked brilliant on the test but couldn't write a coherent sentence on its own. The architectural story is clean. The paper targets adaptive residual-routing decoders — models that dynamically select which layers to route tokens through, using a sequence-level summary to decide routing queries. That summary is the culprit: it's computed over the full input sequence, meaning prefix positions see a representation contaminated by future tokens. This is a general hazard for any architecture that introduces global summaries, gating mechanisms, or dynamic routing conditioned on the full sequence. The attention mask catches direct token-to-token peeks; it says nothing about indirect channels. The diagnostic toolkit is the paper's real contribution. The authors formalize a prefix-invariance contract: given identical prefixes, the model's output distribution for the next token must be identical regardless of what future tokens exist in a teacher-forcing context. They introduce LeakScore (maximum prefix-output discrepancy), complement it with behavioral metrics comparing teacher-forced vs. free-running performance, and add layer-wise profiles that localize which layers introduce violations. Supporting experiments span six synthetic implementation pairs across five module families, clean pretrained controls, controlled fault injection, and cache consistency checks across numerical precisions. Two independently trained 12-layer causal repairs — models with the routing leak fixed — eliminate detectable leakage and return both validation loss and reference perplexity to the control scale. The large apparent advantage disappears entirely. This is the kill shot: the 24-layer model's gains were 100% artifact, 0% architecture. The integrity picture is mixed. The authors are rigorous about constructing controlled comparisons and running multiple diagnostic checks. The six synthetic implementation pairs and five module families give reasonable coverage. But validation is entirely internal — same team, same codebase, no community benchmark, no pre-registration, no independent replication. The fault injection experiments are a strength: they let you see the diagnostic tools catch known bugs, which builds confidence in the tools' sensitivity. But the paper's scope is deliberately narrow — one architecture family, one specific leakage mechanism. The real value here is the generalizable lens, not this specific bug. Any time an autoregressive model introduces a component conditioned on the full sequence — mixture-of-experts gating, adaptive computation, dynamic routing, learned temperature scheduling — the same class of leakage is possible. The prefix-invariance audit is cheap to run and should arguably be a standard check for any new autoregressive architecture claiming teacher-forced loss improvements. The next milestone is adoption: if major labs integrate prefix-invariance checks into their evaluation pipelines, the paper's diagnostic framework will have changed the field's hygiene. If not, it remains a cautionary case study.
