Imagine you're a manager forced to always assign 100% of a task to someone in the room, even when nobody is qualified. That's softmax attention: the weights must sum to one, so every head must pass something forward, even when the right answer is 'nothing useful here.' Now imagine the people you do assign the task to are each juggling five other jobs simultaneously — their output is contaminated by interference from those other jobs. That's superposition in the residual stream. This paper says these are two separate problems, and the gating mechanisms people bolt onto attention happen to fix both — but each fix has a different scaling curve. The committed claim: gated attention's benefits decompose into (1) abstention — letting a head output zero via a learned sink logit that competes with real keys in the softmax — and (2) noise filtering — a per-value gate that suppresses interference from superposed features. These are not the same thing, they scale in opposite directions, and the best model always uses both. The experimental setup is clean and disciplined. Wang trains matched Transformer language models from 10M to 350M parameters, isolating each primitive independently. At 10M parameters, abstention accounts for nearly all the gain from gating. By 350M, noise filtering dominates. The crossover is the story: as models grow and pack more features into each dimension (higher superposition), the interference problem worsens, and filtering becomes the load-bearing fix. Meanwhile abstention's marginal value shrinks because larger models develop other compensating mechanisms. The architecture choices are minimal and elegant. Abstention adds a single learned scalar per head — a 'sink logit' that competes in the softmax, letting the head effectively say 'I have nothing to contribute.' Noise filtering adds a sigmoid gate on each value vector, element-wise, letting the head suppress specific dimensions contaminated by irrelevant features. Both add negligible parameters and remain compatible with KV caching, which matters enormously for inference deployment. This is not a 'new architecture' paper — it is a diagnostic paper that explains WHY existing gating works and decomposes the mechanism. Integrity is solid for this class of work. The authors run controlled interference injection experiments: they deliberately contaminate the values a head reads and measure whether each gate form removes the contamination. This is the kind of causal experiment most architecture papers skip. They also discover that each gate form has a characteristic blind spot — each fails to filter a specific interference pattern that the other catches — which is the structural reason you need both. The scale sweep from 10M to 350M with matched baselines at each scale is more careful than most ablation studies. The limitation is straightforward: 350M parameters is not frontier scale. The scaling trend is suggestive — filtering's benefit is increasing — but we don't know if it continues monotonically to 7B, 70B, or 400B. The paper also studies only language modeling perplexity; downstream task performance, especially on reasoning benchmarks, remains untested. These are compute-budget limitations, not intellectual ones. The broader significance is for the 'what is attention actually doing?' research program. Prior work on gated attention (GatedLinearAttention, SwiGLU-style gates) showed gains but disagreed on the mechanism. This paper resolves the disagreement: both camps were partially right because gating was bundling two fixes. If the noise-filtering benefit keeps growing with scale, this has implications for how frontier labs design attention heads — and suggests that current large models may be paying a hidden tax from unfiltered superposition interference that could be cheaply eliminated.
