Think about how you learned history in school. If your teacher shuffled every lecture — Renaissance on Monday, World War II on Tuesday, Ancient Rome on Wednesday — you'd still learn the facts eventually, but your sense of when things happened relative to each other would be garbage. You'd know Napoleon existed but not whether he came before or after the telegraph. That's exactly what we do to LLMs: we shuffle the entire training corpus, and then we're surprised when they confidently assert that a CEO who stepped down in 2022 is still running the company. This paper from Kyutai (Pilchen, Fabre, Signe Talla, Perez, and Grave) asks a clean question: what happens if you DON'T shuffle? They pretrain 6B-parameter models on temporally ordered Common Crawl snapshots — feeding the model 2013 data, then 2014, then 2015, and so on — and compare against the standard shuffled regime. The headline result: sequentially trained models match shuffled baselines on general language understanding and common knowledge benchmarks, while consistently exhibiting more up-to-date and temporally precise factual knowledge. The mechanism is intuitive once you see it. Shuffled training exposes the model to the same fact many times across many time periods, which means older formulations of facts (which have had more time to accumulate copies across crawls) get disproportionate weight. The paper calls this 'increased factual repetition' — the shuffled model effectively votes on facts by frequency, and older facts win the election. Sequential training doesn't have this problem: the model sees 2023 data last, so 2023 facts sit closest to the final parameter state. The benchmark contribution matters independently. They introduce over 7,000 temporally grounded questions with an evaluation protocol that tests whether models correctly associate facts with their corresponding time periods. This isn't just 'who is the president?' — it's 'who was the president in 2019 specifically?' The benchmark and evaluation code ship publicly via GitHub (kyutai-labs/kairos) and HuggingFace. The ladder question is where you should be honest-eyed. The paper compares sequential vs. shuffled pretraining at 6B parameters on Common Crawl. It does NOT compare against continual learning approaches (LoRA-based updates, retrieval augmentation, or the various knowledge-editing methods that have become the industry's preferred answer to temporal drift). The claim is narrower than the implication: ordering your pretraining data chronologically is a free lunch for factual freshness at equal general performance. The architecture is standard autoregressive transformer pretraining — the variable under study is data ordering, not model architecture. This is a training-dynamics paper, not a modeling paper. The compute property it leans on is that Common Crawl snapshots are naturally timestamped, so temporal ordering costs nothing extra at data-prep time. The obvious next experiment they didn't run: scaling beyond 6B parameters, and combining sequential pretraining with continual learning or retrieval augmentation. At 6B, you're firmly in the 'research model' regime — nobody ships a 6B model as their production LLM anymore. The question that matters is whether the temporal-ordering benefit persists, grows, or vanishes at 70B+. My read: compute budget (a). Training a 70B model multiple times to compare ordering strategies is expensive, and Kyutai chose breadth of analysis over scale. Fair trade for a first paper, but it means the result is a hypothesis about production-scale training, not a proof.
