Imagine you're a chef trying to perfect a spice blend with 288 ingredients. The standard approach — tossing random pinches of everything into a bowl, tasting the result, repeating hundreds of times — is what the field has been doing with data mixing for language models. LinMix says: instead of random blends, test each spice at full strength or leave it out entirely. Binary on-off combinations give you clean signal about what each ingredient actually contributes. The committed claim: Dirichlet-sampled proxy training runs are a statistically poor experimental design for estimating domain contributions, and vertex sampling on the capped simplex — where each domain is either at its maximum weight or absent — provably decorrelates domain effects, enabling a simple linear model whose coefficients directly give you the optimal mix. No hyperparameters, no search over candidate mixes, no nonlinear regression. The optimal mix keeps only domains with positive attribution and discards the rest. The theoretical contribution is genuinely sharp. The authors prove that Dirichlet sampling concentrates mixes near the prior, inflating attribution error for domains far from that center. Vertex sampling achieves D-optimality (maximizing the determinant of the information matrix), which is textbook optimal experimental design applied to a problem the ML community has been solving with brute force. The linear model isn't a simplification — it's the correct model class when your experimental design supports it. On the empirical ladder, LinMix was tested with 1B-parameter OLMo2 and 4B Qwen3.5 models. Against the Dirichlet-regression baseline (DoReMi-style), LinMix shows relative gains of 46% on MBPP (code) and 14% on GSM8K (math). At 288 fine-grained domains — where Dirichlet sampling cannot support a linear fit and practitioners resort to nonlinear models — LinMix improves benchmark accuracy by 11% on average over the best nonlinear baseline. These are not marginal improvements; the method scales where the competition structurally cannot. Integrity is solid for a workshop paper. The authors use standard community benchmarks (MBPP, GSM8K, reasoning tasks), compare against the established Dirichlet-regression paradigm, and provide theoretical proofs for the optimality of their sampling scheme. The main limitation: both models are in the 1B–4B range. We don't yet know if the linear assumption — that domain contributions are approximately additive — holds at 70B or 400B scale, where domain interactions may become more complex. The proofs establish the design is optimal for the linear model; whether the linear model itself is the right model at frontier scale remains open. The successor question is clear. The obvious next experiment is running LinMix at 70B+ scale with the full Llama or Qwen training pipeline. The honest read: this is a compute budget problem. Training hundreds of 70B proxy models, even at small token counts, is expensive. The authors demonstrated the method at the scale they could afford. The theory should transfer, but empirical confirmation at frontier scale is the gap that matters. A secondary gap: the method assumes a fixed set of domains. Real pretraining pipelines involve dynamic domain definitions, deduplication, and quality filtering that change the domain structure mid-flight. What makes this paper worth your time is the reframing. 'Data mixing as domain attribution via experimental design' is a lens that clarifies why the field's current practice works poorly and what to do instead. The vertex sampling idea is simple enough to implement in a weekend and principled enough to trust. For practitioners building pretraining data pipelines, this is immediately actionable at moderate scale. For the field, it's a clean example of classical statistics outperforming ML-on-ML approaches when the problem structure supports it.
