
You know how a metal detector beeps at every bottlecap and pull-tab in the sand? The beep is real — there IS metal down there — but you still have to dig before you know if it's a coin or garbage. This paper is about the dig. The interpretability community has been clustering attention heads by co-activation patterns (the beep) and calling the resulting groups 'circuits.' Xu asks: do these clusters actually function as circuits when you test them causally? The committed claim: co-activation clustering proposes circuit candidates, but only a closure test — ablating the entire discovered community and comparing per-example loss damage to matched-random ablation controls — can confirm them. The paper introduces 'closure' as the validation gate that separates real circuits from statistical phantoms. The method adapts a sparse-autoencoder (SAE) clustering recipe, originally designed for neurons, to attention heads. But where prior work validated clusters by reconstruction quality (can you rebuild the activation from the cluster?), Xu validates by causal intervention: knock out the proposed circuit and measure whether the damage pattern is non-random. Across Pythia 1B and OLMo 1B — two dense transformer models at 1B scale — and two input distributions, the co-activation communities pass closure. The beep was a coin. Then the interesting failure. In OLMoE-1B-7B, a Mixture-of-Experts model, route-conditional clustering finds statistically significant co-activation structure. The beep is loud and clear. But closure kills it: ablating these communities actually improves loss. The model gets better without the 'circuit.' That's not a circuit — that's noise the model was routing around. The MoE architecture's routing mechanism creates co-activation patterns that look structured but are functionally inert or even parasitic. The training-dynamics experiments add a second sobering result. Tracking closure across training checkpoints, the paper finds that attention-target selectivity and participation ratio — two metrics the field uses as proxies for 'this head is doing something specific' — decouple from actual function in both directions. Heads become more selective without becoming more functionally important, and vice versa. The proxies are broken. The paper's real contribution is methodological, not architectural. It doesn't propose a new way to find circuits — it proposes a way to check whether your circuit-finding method is lying to you. The closure test is cheap (matched-random ablation controls), principled (causal rather than correlational), and produces a clear binary signal. The limitation is that closure is necessary but not sufficient: passing closure means the cluster isn't random, but doesn't prove it's the minimal or optimal circuit. For the interpretability field, the takeaway is disciplinary: stop treating co-activation clusters as confirmed circuits. Propose with statistics, confirm with causation. The MoE failure mode is particularly important as the field moves toward sparse architectures — co-activation patterns in routed models may be systematically misleading.
The analogy
A metal detector beeps at every piece of metal in the sand — bottlecaps, pull-tabs, coins alike. The beep (co-activation) is a real signal, but you still have to dig (ablation) to know if you found treasure or trash. This paper is about making the dig mandatory.
Metal detector
Detects buried metal reliably, but can't distinguish coins from bottlecaps without excavation
Closure test
Co-activation clustering detects statistical structure reliably, but can't distinguish functional circuits from correlated noise without causal ablation
Honest assessment
A clean methodological contribution from a single author: co-activation clustering is a proposal mechanism, not a confirmation mechanism, and the closure test is the right gate to add. The MoE negative result is the most interesting finding — it suggests that sparse routing architectures create co-activation mirages that look real statistically but are functionally empty. The limitation is scale: everything here is 1B parameters, and the question of whether closure holds at 7B+ is genuinely open. If you work on mechanistic interpretability, this paper changes how you validate your next result. If you don't, it's a well-executed cautionary tale about correlation vs. causation in a specific technical domain.
What Is CLAIMS?
63
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
2/5
Architecture
4/5
Integrity
4/5
Milestone
2/5
Successor
3/5
What is this paper actually committing to?
Real methodological contribution: closure as validation gate for circuit discovery. Not a new discovery method but a falsification tool — the field needed this distinction stated clearly and tested.
The field fight
Are co-activation statistics sufficient to identify functional circuits in neural networks, or is causal validation required?
Side A
Co-activation clustering with reconstruction validation is sufficient and scalable
Held by: SAE-based circuit discovery, Anthropic's feature visualization pipeline
Side B
Only causal intervention (ablation, patching) confirms functional circuits — correlational evidence is necessary but not sufficient
Held by: Causal abstraction, activation patching, this paper
Milestone tracker
Unlocks: Confidence that co-activation circuits hold (or don't) at production model scale
Current
1B
params
closure validated
Next
7B+
params
scale confirmation
~1-2 years; compute-limited, not method-limited
Verified Facts (5)
Tested on Pythia 1B, OLMo 1B (dense) and OLMoE-1B-7B (Mixture-of-Experts)
Communities pass closure in dense models across two input distributions
In OLMoE-1B-7B, route-conditional clustering recovers statistically real signal that does not survive closure — ablation improves loss
Attention-target selectivity and participation ratio decouple from function in both directions across training
22 pages, 3 figures
Practical Implications (4)
If you're publishing co-activation clusters as 'circuits,' you now need a closure test or reviewers will (rightly) ask for one
MoE interpretability via co-activation is suspect — route-conditional clustering produces statistically significant but functionally meaningless structure
The closure test itself is cheap to implement: matched-random ablation controls against your proposed community, comparing per-example loss damage
Attention-target selectivity and participation ratio are unreliable proxies for head importance across training — stop using them as standalone evidence
More latest news
Read all