Imagine you're watching someone solve a math problem on a whiteboard. You can see their handwriting, the scratch marks, the way they pause — and you're pretty sure you can read an intermediate step in the margin. But when you erase that margin note and replace it with someone else's, the final answer doesn't change. The note was there, sure, but the solver wasn't reading it. That's the core finding here: latent representations that probes detect aren't necessarily causally load-bearing. The paper attacks a foundational assumption in mechanistic interpretability — that if a linear probe can read a value from a model's hidden state, the model is 'using' that value to compute the answer. Merin introduces a minimal-pair protocol: take two prompts identical except for one digit, read the last prompt token's activations, and check whether the digit's identity is encoded in a direction shared across different questions. In both Ouro-1.4B (a looped transformer) and Qwen2.5-1.5B (a standard transformer), the answer is yes — cross-question readout accuracy hits ~0.9 using the minimal-pair contrast. But a single-prompt probe, the standard setting in most probing studies, scores only 0.36 (Ouro) and 0.19 (Qwen) even after 5,249 training examples. Why the gap? The paper rules out the obvious explanation — that the digit's signal is simply drowned out by other information. Random noise projected onto equally 'busy' directions shows equivalent magnitude, and a random word substituted in place of the digit moves the state far more and is easily readable. The digit's trace is small, not because the state is crowded, but because the encoding is compact. This is a subtle and important distinction: the information is present in a low-variance, shared code rather than absent. The integrity move is the pre-registered two-step arithmetic study. For problems of the form (a+b)×c, the paper probes for the unwritten intermediate a+b just before the model writes the final digit. A small signal exists, and its sign tracks the operation (positive for addition, negative for subtraction). But here's the deflation: a synthetic null — testing whether generic input-mixing rather than genuine computation produces the same sign pattern — accounts for much of the signal. The paper honestly reports this. The causal test is the sharpest result. Patching the intermediate value from one problem's first loop onto a different problem does not shift the written answer toward the donor problem's result. Patching the same location during the answer phase does. The intermediate is readable but not functionally used — a ghost in the activation, not a gear in the mechanism. This matters because the entire mechanistic interpretability research program leans on probe accuracy as evidence of internal computation. If probe-readable representations can be epiphenomenal — artifacts of input mixing rather than genuine intermediate computations — then a large body of prior probing results needs revisiting with causal controls. The minimal-pair protocol plus synthetic null baseline the paper introduces is a concrete methodological contribution that other groups can adopt immediately. The limitation is scale: two models under 2B parameters on digit-copy and two-step arithmetic. Whether the 'readable but unused' phenomenon generalizes to richer reasoning (multi-hop QA, chain-of-thought, planning) in frontier-scale models is entirely open. The protocol is ready; the coverage is narrow.
