Imagine you're trying to tune a guitar by ear. You pluck a string and it sounds close enough — you've memorized the approximate pitch. But then you keep tightening and loosening the tuning peg, and the string slowly drifts toward the exact note. The memorization happened fast; the real tuning takes much longer because you're fighting the string's tension the whole way. That tension is weight decay, the tuning peg is the learning rate, and the delay between 'close enough' and 'actually right' is grokking. This paper's committed claim: grokking — the mysterious phenomenon where neural networks memorize training data quickly but don't generalize until thousands or millions of steps later — is quantitatively explained by residual-driven spectral evolution of the neural tangent kernel (NTK) under L₂ weight decay. The grokking timescale is controlled by the product η·λ (learning rate × weight decay), feature learning slows logarithmically near a critical decay value, and above that critical value generalization never arrives. These aren't qualitative stories — they're derived predictions from a reduced dynamical system. The architecture of the argument sits in the NTK / lazy-vs-rich learning family. Most NTK theory assumes the kernel is frozen (lazy regime), which predicts no grokking at all. This paper shows that after memorization, a finite residual remains — larger for target components aligned with small NTK eigenvalues — and this residual feeds back into the kernel dynamics, pushing the network from lazy into rich learning. By projecting onto task-relevant spectral directions, you get a low-dimensional system where residual-driven kernel growth competes against weight decay. The math is for homogeneous networks with squared loss and L₂ regularization, but the empirical tests push beyond those assumptions. On the ladder: prior grokking explanations include Power et al. (2022) who first named the phenomenon, Nanda et al. (2023) who gave mechanistic interpretability accounts for modular addition, and Liu et al. (2023) who connected grokking to representation learning. None offered a quantitative prediction of the generalization timescale as a function of hyperparameters. This paper does. The predicted inverse-product scaling tgrok ∝ 1/(η·λ) is tested across an 84×90 grid of (η, λ) combinations for an MLP and a 42×45 grid for a one-block Transformer, both on modular addition. The phase geometry — regions of memorization-only, grokking, and failed fitting — matches the theory's predicted boundaries. That's a genuine advance: going from 'we can describe grokking after the fact' to 'we can predict where it will and won't happen in hyperparameter space.' Integrity check: the validation is self-consistent simulation, not independent replication. The theory is derived for homogeneous networks, and the MLP experiments match that assumption. The Transformer experiment is more interesting precisely because it violates exact homogeneity — the fact that the same macroscopic phase structure and scaling emerge anyway is either evidence of robustness or a hint that the theory captures the right physics even outside its formal scope. The benchmarks (modular addition, Fourier structure in the NTK) are standard in the grokking literature, not cherry-picked. But the paper does not test on harder tasks — vision, language, anything beyond toy arithmetic — which is where the real question lies. The milestone that matters: can this theory predict grokking timescales for realistic architectures on realistic tasks? Today's result covers modular addition on MLPs and a one-block Transformer. The next concrete target is a multi-layer Transformer on a task like CIFAR or a small language modeling benchmark, where the spectral structure is far richer and the homogeneity assumption breaks hard. If the inverse-product scaling and phase boundaries hold there even approximately, this theory graduates from elegant to load-bearing. The obvious experiment not run: multi-layer Transformers on non-arithmetic tasks. The honest read is (a) compute and complexity — projecting NTK eigendirections for a deep Transformer on ImageNet-scale data is a different engineering problem entirely, and the spectral decomposition that makes the theory tractable becomes intractable. There's also a possibility of (c): the authors know the single-block Transformer result is already a stretch beyond the formal assumptions, and they're positioning the multi-layer case as the natural sequel paper. The gap between 'works on modular addition' and 'works on real tasks' is where this theory will either prove its weight or remain a beautiful special case.
