
HELD-OUT PROMPTS
+10.3%
COVERAGE REMAINS
+8.7%
FOUR LAYERS
+3.3%
Imagine you work at a massive company with 8 billion employees, and you suspect that most of the real work gets done by a few hundred recurring project teams — small groups who keep showing up together on task after task while most people sit idle. You can't just ask HR; you have to watch millions of projects, track who shows up with whom, and then prove those teams actually matter by seeing what breaks when you fire them and what survives when you fire everyone else. That's the core mechanism of this paper: statistical co-occurrence across ten million prompts, converted into a graph, then carved into communities, then causally stress-tested. The committed claim: within Llama-3.1-8B's MLP layers, there exist small, reusable groups of neurons (capped at 128 across all layers) that recur across prompts and are both causally necessary and sufficient for specific computations — grammatical agreement, arithmetic, fixed phrases, in-context copying, token-probability calibration. This is not a one-off sparsification trick applied per-input. The communities are discovered once, unsupervised, and then validated on held-out prompts they were never trained on. The method pipeline has three stages. First, gradient-based attribution identifies which neurons matter for each of ten million prompts. Second, a co-activation graph connects neurons that are repeatedly attributed together across prompts. Third, standard community detection (think Louvain-family algorithms) carves this graph into candidate neuron groups. The key architectural choice is working in neuron space rather than feature space — no sparse autoencoders, no dictionary learning, just raw neurons and their co-occurrence patterns. This makes the results directly interpretable in terms of the model's existing parameters. The integrity regime is where this paper earns real respect. Three separate causal tests on held-out prompts: necessity (does removing the community break the prediction?), sufficiency within tested layers (does keeping only the community while ablating all other MLP neurons in those layers preserve the prediction?), and layer necessity (do the layers the community spans actually matter?). This triple-gate design means a community can't pass by accident — it has to be both load-bearing and self-contained. The evaluation on held-out data is critical: communities were discovered on one set of ten million prompts and tested on a separate set, eliminating the most obvious overfitting concern. The coverage numbers demand honest framing. With communities capped at 128 neurons, 10.3% of held-out prompts have at least one layer's MLP contribution explained on a neuron-sparse basis. That drops to 8.7% requiring four layers and 3.3% requiring twelve layers. This means roughly 90% of prompts cannot be explained by any discovered 128-neuron community. The authors are admirably upfront about this. The question is whether this 10% represents a floor that scales with more communities and larger graphs, or an asymptotic ceiling reflecting a fundamentally distributed computation for most inputs. The ladder question is tricky because there is no clean SOTA to compare against. The closest prior work is per-input sparsification (showing individual prompts can be post-hoc pruned), but that doesn't establish reusable groups. Mechanistic interpretability work on circuits (Conmy et al., Wang et al.) operates at a different granularity — typically hand-identified circuits for specific tasks, not unsupervised large-scale discovery. The paper's real contribution is methodological: a scalable pipeline that goes from raw co-occurrence to causally validated communities, applied at a scale (10M prompts, 8B-parameter model) that prior circuit-discovery work has not attempted. The successor experiment that wasn't run is the scaling test: does coverage increase with model size, or does it decrease as computation becomes more distributed? Testing on Llama-3.1-70B or 405B would answer whether neuron-sparse communities are a property of all language models or primarily of smaller ones where capacity constraints force more reuse. The honest read is compute budget — running ten million forward passes with gradient attribution on a 70B model is expensive, and the community detection graph would be correspondingly larger. This is almost certainly the next paper.
The analogy
Like discovering that a 8-billion-person company's real work is done by a few hundred recurring project teams — you watch millions of tasks, track who shows up together, then prove those teams matter by firing them (things break) and firing everyone else (things still work).
Project teams
Tracking recurring small groups of employees who co-occur on tasks and proving they're load-bearing by removal experiments
Neuron communities
Tracking recurring small groups of neurons co-attributed across prompts and proving they're causally necessary/sufficient via ablation
Honest assessment
A methodologically clean proof of concept that reusable modular structure exists in LLM neurons at a nontrivial scale. The triple-gate causal validation is genuinely rigorous and the 10M-prompt scale is impressive. But 10.3% coverage means 90% of computation remains unexplained by this method, and the absence of scaling experiments on larger models leaves the central question — is this a fundamental property of LLMs or an artifact of 8B-scale capacity constraints? — wide open. Mechanistic interpretability researchers and anyone working on model compression or steering should read this. Everyone else can wait for the scaling paper.
What Is CLAIMS?
70
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
2/5
Architecture
4/5
Integrity
4/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Genuine methodological first: unsupervised discovery of reusable, causally validated neuron communities at 10M-prompt scale. Not just per-input pruning — persistent modular structure. The claim is real but the coverage is modest.
The field fight
Are language model computations irreducibly distributed across all parameters, or decomposable into reusable modular units?
Side A
Computation is fundamentally distributed; superposition makes clean decomposition impossible at scale
Held by: Anthropic (superposition hypothesis), Elhage et al.
Side B
Modular structure exists and can be found — neurons, circuits, or features form reusable computational units
Held by: Olah et al. (circuits), sparse autoencoders community, this paper
Milestone tracker
Unlocks: Neuron communities as a practical interpretability and compression tool, not just a proof of concept
Current
10.3
% coverage
this paper (128-neuron cap)
Next
50
% coverage
majority-explained threshold
Unknown — depends on scaling with community count and model size
Verified Facts (6)
Applied to Llama-3.1-8B using ten million prompts
Communities capped at 128 neurons across all layers
10.3% of held-out prompts have at least one layer's contribution explained on a neuron-sparse basis
Coverage remains 8.7% when requiring at least four layers and 3.3% when requiring at least twelve
Three causal tests: community necessity, sufficiency within a layer subset, and layer necessity
Discovered communities support functions like grammatical agreement, token-probability calibration, fixed phrases, in-context copying, addition, and multiplication
Practical Implications (3)
If neuron communities generalize across model sizes, they could enable targeted model editing — modifying specific behaviors by intervening on small neuron groups rather than fine-tuning
The 128-neuron sufficiency result suggests a path toward extreme structured pruning for specific capabilities, though 10% coverage is not yet practical
The pipeline (gradient attribution → co-activation graph → community detection → causal validation) is a reusable methodological template for probing any large neural network's modular structure
More latest news
Read all