Imagine you have a filing cabinet. You can toss papers in by topic, but the more papers you file under similar labels, the harder it gets to pull out the right one without dragging half the drawer with it. That interference — not the storage itself — is what limits how useful your cabinet is. This post makes the same argument about neural network layers: the bottleneck isn't how much you can write into the weights, it's how cleanly you can read back out. The core claim is that linear maps, MLPs, and self-attention are three points on a single continuum of associative memory stores, each distinguished by how they handle interference during recall. A bare linear layer (outer-product memory) stores key-value pairs as summed outer products in a weight matrix. Querying returns the target value plus interference from every other stored memory whose key is even slightly correlated. For random unit-vector keys in dk dimensions, the interference-to-signal ratio scales as √(M/dk), giving a capacity of roughly dk memories — about 64 for 64-dimensional keys. The first upgrade is the optimal linear store: instead of naively summing outer products, you solve a least-squares regression so that the weight matrix minimizes total recall error across all stored memories simultaneously. This whitens the keys by premultiplying the query with (K^T K)^{-1}, canceling shared directions. Below dk memories you get perfect recall even with non-orthogonal keys; above dk the capacity doubles to 2·dk — about 128. The math is clean linear algebra: the stored weights are the pseudoinverse V^T (K^T)^+, and the interference analysis reduces to properties of projection matrices. MLPs represent the next jump. By passing key similarities through a nonlinearity — effectively a sharpening function — you suppress low-similarity keys that would otherwise contribute noise. The post frames this as replacing the linear similarity ki^T x with a sharpened version, so that only near-matches contribute to recall. This pushes capacity to roughly dk^2 / (2 ln dk), which for dk = 64 gives about 490 memories, or higher with tuned sharpness — the post cites ~2,700 with optimized β. Self-attention is the final form. Instead of baking memories into fixed weights, attention stores keys and values explicitly in the context and computes softmax-weighted retrieval at query time. The softmax acts as the ultimate sharpening function: the winner-take-all dynamic means interference drops exponentially with the gap between the best-matching key and the rest. Capacity becomes effectively unbounded by dk — with 64-dimensional keys, over a million memories can be stored before recall degrades, limited only by context length. The post frames this as the reason transformers can hold so much world knowledge: they get both the parametric memory of MLPs in their feed-forward layers and the near-perfect recall of attention over the context. What makes the exposition work is its consistent use of one quantity — the interference-to-signal ratio — as the lens through which each architecture is evaluated. Every upgrade (outer-product → optimal linear → MLP → attention) is justified by how it reduces this ratio for a given number of stored memories. The interactive visualization lets readers drag a slider past capacity thresholds and watch recall collapse, which is more instructive than any equation. The analysis is pedagogical rather than empirical. No new experiments are run; no benchmarks are compared. The contribution is conceptual unification: showing that the progression from linear layers to self-attention follows logically from the goal of minimizing recall interference. The interference-to-signal framework draws on classical associative memory theory (Hopfield networks, Kanerva's sparse distributed memory) and recent work connecting gradient descent to attention (Irie, Csordás & Schmidhuber). For practitioners, the capacity heuristics — dk for linear, 2·dk for optimal linear, ~dk²/(2 ln dk) for MLP, effectively unbounded for attention — give useful back-of-envelope estimates for model design.
