Imagine you're studying for an exam. You could painstakingly memorize every possible answer through repetition — that's gradient-based learning. Or you could build a cheat sheet: for every question stem, write down the most common correct answers weighted by how often they appeared in past exams. COUNT is the cheat sheet. It replaces learned n-gram embedding tables with precomputed frequency statistics from the training corpus, stored in ordinary system RAM rather than expensive GPU memory. The committed claim: you can build n-gram embeddings without any learned parameters by counting how often each token follows a given n-gram context in the corpus, then weighting together the model's own unembedding vectors for those successor tokens. The result is a fixed lookup table that lives on host memory, costs nothing to train, and provides the model with a powerful prior about what comes next. At a fixed compute budget, adding COUNT to a 1B-parameter LM improves validation loss by the same amount as giving the baseline 57% more FLOPs. The architectural insight is clean. Existing n-gram scaling methods (like those from the LLEMMA and related lines of work) train embedding tables from scratch, which means those tables must live on GPU memory for fast gradient updates. COUNT sidesteps this entirely: the n-gram statistics are precomputed offline, the resulting embeddings are read-only lookups, and host RAM is orders of magnitude cheaper than GPU HBM. This converts an underutilized resource — the terabytes of system memory sitting idle on most training nodes — into a direct performance lever. The efficiency story has a second act. When GPU memory is constrained, practitioners face a choice: allocate some GPU memory to learned n-gram embeddings, or let the backbone model fill the GPU entirely. COUNT resolves this tension — the backbone gets all the GPU memory it needs, while the n-gram signal arrives from host at negligible cost. The paper demonstrates that this split consistently outperforms the alternative of sharing GPU memory between backbone parameters and learned n-gram embeddings. The most practically interesting feature is training-free domain adaptation. Because the COUNT table is just a frequency distribution, you can swap in a different table computed from a domain-specific corpus at inference time. No fine-tuning, no gradient updates, no additional compute — just a different cheat sheet. This is a genuine capability that learned embeddings cannot match without retraining. Integrity is reasonable but bounded. The experiments center on 1B-parameter models and validation loss as the primary metric. The 57% compute-equivalence number is striking but needs context: it measures how much additional compute a baseline LM would need to match COUNT's validation loss improvement, not a 57% speedup in wall-clock training time. The comparison is against learned n-gram embedding baselines from the same research lineage, which is fair but not exhaustive — we don't see comparisons against retrieval-augmented generation or other memory-augmented approaches that also exploit external storage. The question this paper leaves open is scale. Does the COUNT advantage hold — or grow — at 7B, 13B, 70B parameters? The n-gram statistics are scale-invariant, but the relative contribution of a fixed lookup signal likely diminishes as the backbone model gets larger and learns its own implicit n-gram statistics. The authors almost certainly have intuitions about this trajectory, and either compute budget or strategic publication sequencing explains the gap.
