Word2Vec but for Entire Neural Networks: Meta-Learned Models Organize Their Parameters Like Vocabulary Vectors

Meta-learned networks encode functions as parameter vectors that obey king−man+woman=queen-style arithmetic. Recombined vectors implement intended functions without any direct training.

You know how in Word2Vec, "king" minus "man" plus "woman" gets you something close to "queen"? The vector space learned a decomposition where gender and royalty are separable directions. Now imagine the same trick, but instead of word embeddings, the vectors are entire neural networks — every weight and bias concatenated into one long vector. That is what this paper demonstrates, and the implications for understanding rapid learning are substantial. The core claim: when you meta-learn a neural network (exposing it to many related tasks so it can quickly adapt to new ones), the resulting parameter space develops the same kind of additive compositional structure that word embeddings exhibit. Functions can be decomposed into independent symbolic components, and those components can be algebraically recombined to produce networks that implement never-seen functions — without any gradient descent on those target functions. The network specified by the recombined parameter vector just works. The experimental setup uses MAML-style meta-learning on families of compositional functions. Each function can be described symbolically — for instance, a function might combine an input transformation with an output mapping. After meta-learning, the authors adapt the network to individual functions, then concatenate the adapted parameters into vectors. They then perform arithmetic: take the parameter vector for function A, subtract the component corresponding to one symbolic property, add the component corresponding to a different property, and check whether the resulting parameter vector implements the predicted function. It does, with striking reliability. This is a mechanistic explanation paper, not a performance paper. The authors are not claiming to beat any SOTA benchmark. They are offering a geometric account of why meta-learned networks can learn fast: the meta-learning process organizes parameter space so that functionally meaningful directions are linearly separable. Rapid learning is then just moving along these pre-organized directions. The analogy to word embeddings is not rhetorical — it is structural. The same linear algebra that makes word analogies work makes function analogies work in parameter space. The integrity picture is mixed in the way typical of interpretability work. The functions studied are synthetic and compositional by design, which makes the result clean but raises the question of whether natural task distributions would yield the same structure. The authors acknowledge this. There is no independent replication, and the scale is modest — these are small networks on synthetic tasks. But the experimental logic is sound: they test a specific, falsifiable geometric prediction and it holds. The real value is conceptual. If meta-learning induces compositional structure in parameter space, this gives us a concrete geometric account of inductive bias — not as a vague prior but as a measurable property of how parameter trajectories are organized. This connects the dots between meta-learning, compositionality, and the broader question of what makes some neural network initializations dramatically better than others for rapid learning. The obvious next step is scaling: do these additive relationships hold in larger networks, with richer function families, or with naturalistic (non-synthetic) task distributions? The authors did not run this, and the honest read is that it is computationally expensive and the synthetic setting was necessary to establish the clean result first. Whether the structure degrades gracefully or collapses entirely in messier settings is the load-bearing open question.