Imagine you're a new employee at a company with a massive org chart. Some colleagues you actually understand — you know their role, their team, who they report to. Others you can name when prompted but couldn't place in context if your life depended on it. Now your boss asks you to learn a new coding scheme for departments. The colleagues you truly understood? You can recode them instantly. The ones you merely memorized? Dead weight. This paper proves language models work the same way — and that there's a clean, measurable test to tell the difference. The committed claim: whether a pretrained language model can generalize a newly fine-tuned relation to unseen inputs is strongly predicted by how linearly decodable the underlying original relation was in the model's pretrained representations. Not by recall accuracy, not by training frequency, not by dataset size — by linear readability. Pearson correlations of r = 0.87 to 0.92 across eleven models and seventy relations. That's a remarkably tight relationship for behavioral science of neural networks. The experimental design is elegant. Take a known factual relation (Paris → Europe), assign arbitrary labels to the answers (Europe → crab), creating a composed relation (Paris → crab). Fine-tune on a subset, test on held-out subjects sharing the same original answer. If the model learned Paris → crab, does it generalize to Munich → crab? The answer depends almost entirely on whether the original relation (city → continent) was linearly readable in the pretrained representations. The authors call generalizing facts "active" and non-generalizing ones "inert." The terminology is precise and useful. The controls are unusually thorough. The r = 0.87–0.92 correlation holds after controlling for number of fine-tuning subjects and answers, the model's own accuracy on the original relation, and pretraining frequency of subjects — separately and jointly. They re-randomize label assignments and fine-tuning seeds. They swap animal labels for thirteen other semantic categories. The result doesn't budge. The error analysis is sharp too: when the model assigns the wrong new label, mapping that label back through the relabeling reveals the model's "intended" original answer — and these recovered answers align more with the linear probe's top predictions than with the pretrained model's own output distribution. The geometry is doing the work, not the surface behavior. Architecturally, this is a probing study in the classic representational geometry tradition — linear probes over frozen pretrained transformer representations, followed by parameter-efficient fine-tuning (the paper tests across eleven models spanning different scales and families). The method leans on the linear probe as both a diagnostic instrument and a predictor variable. The key insight is that linear readability is a property of the representation geometry, not of the model's generation capability, and that this geometric property is what matters for downstream compositional generalization. The biggest integrity question is self-grading: the authors construct the benchmark, define the metric, and evaluate on it. There's no pre-existing community benchmark for active-vs-inert factual knowledge, so they had to build one. The mitigation is breadth — seventy relations, eleven models, multiple control variables, multiple random seeds. Code and data are released. But independent replication on different relation types or in different modalities hasn't happened yet. The correlation is strong enough that it would be hard to cherry-pick, but the relation selection process itself could still harbor subtle biases. The practical implication cuts deep for anyone building fine-tuned systems: not all facts in a pretrained model are equally useful for further learning. If you want to fine-tune a model to learn new compositional knowledge, probe the base model's representations first. Facts encoded linearly will generalize; facts encoded nonlinearly won't. This is a concrete, actionable diagnostic — not just interpretability theater. The milestone to watch: whether this linear-readability-predicts-generalization principle extends beyond factual relations to procedural or reasoning tasks, and whether it holds when fine-tuning is full-parameter rather than parameter-efficient.
