Read the paper
Latent Lattice Language Models
on Project Page
REDUCTION OVER
+6.6%
DROP BASE
+11.7%
TRANSFORMER
+4.3%
PURE BASELINES
4x
Imagine a highway interchange where traffic only flows north-south through stacked overpasses. Every car entering at the bottom must climb through every layer to reach the top, and information from earlier cars can only reach later ones indirectly, via the KV-cache rearview mirror. Now imagine you add east-west lanes at every level — persistent streams that carry state forward in time, feeding into the vertical traffic at each layer. That is the Lattice of Latents: horizontal recurrent streams, one per layer, that persist across tokens and directly condition the vertical residual stream's attention, KV-cache writes, MLP computation, and final decoding. The committed claim: persistent per-layer latent state is a complementary axis for scaling language models, orthogonal to depth, width, and existing residual-stream enhancements. At 406M parameters trained on 20B FineWeb-Edu tokens, a Lattice Transformer drops validation perplexity from 15.41 to 14.40 — a 6.6% reduction over a parameter-matched baseline. When composed with hyper-connections and input residuals, it pushes further to 13.61, a 11.7% drop from the base Transformer and a 4.3% improvement over those enhancements alone. What makes this more than a residual-stream tweak is the cross-architecture composability. The authors test on Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids, and the Lattice improves perplexity at every tested native state size. The sharpest result: Lattice-GDN and Lattice-Triadic-GDN variants outperform pure baselines running 4× larger native states. That 4× number is the headline — it means you can get the quality of a much heavier recurrent state by adding lightweight horizontal latents instead of scaling the native state directly. Architecturally, the Lattice sits in the family of augmented residual-stream methods — hyper-connections, mHC, attention residuals, value residuals — but differs structurally. Those methods add or reroute vertical pathways through depth. The Lattice adds horizontal pathways through time. The latent streams are recurrent (they persist across tokens), but unlike hybrid Transformer-SSM designs where recurrent states serve as internal memory for sequence mixing, these latents directly condition the transformer's core operations. The distinction matters: the Lattice is not replacing attention with recurrence; it is giving the attention mechanism richer conditioning context. The integrity picture is solid but bounded. All experiments use a single dataset (FineWeb-Edu, 20B tokens), a single scale (406M parameters), and the same team's implementation. Perplexity is the only reported metric — no downstream task evaluation, no few-shot benchmarks, no generation quality assessment. The baselines are current and named (vanilla Transformer, hyper-connections, Mamba-2, GDN variants), and the comparisons are parameter-matched, which is the right way to do it. But the absence of independent replication and the single-scale, single-dataset regime leaves the generalization question wide open. The milestone question is straightforward: does this scale? The 406M / 20B-token regime is a proof-of-concept scale. The next number that matters is whether Lattice gains hold at 1B-7B parameters on 100B+ tokens, where most architectural innovations either compound or wash out. The authors did not run this experiment, and the honest read is compute budget — training multiple architecture variants at 7B scale is expensive, and this paper is already dense with ablations across five architecture families. The scaling experiment is almost certainly the next paper. The broader field fight here is about where to add compute in language models. The dominant paradigm says: go deeper, go wider, add reasoning steps, or add recurrence for long-context. This paper argues for a neglected axis — persistent per-layer horizontal state — and shows it composes with all the other axes. If the scaling results hold, this becomes a standard ingredient. If they don't, it's a clever 406M-scale trick. The 4× native-state equivalence on SSM hybrids is the number to watch.
The analogy
Imagine a highway interchange where traffic only flows north-south through stacked overpasses. Now add east-west lanes at every level — persistent streams carrying state forward in time, feeding into the vertical flow at each layer. That's the Lattice: horizontal recurrent lanes alongside the vertical residual stream.
Highway interchange
East-west lanes added at every overpass level, carrying traffic forward and merging into north-south flow
Lattice of Latents
Per-layer horizontal latent streams running recurrently across tokens, conditioning the vertical residual stream's attention, KV-cache, and MLP
Honest assessment
A well-executed structural proposal at proof-of-concept scale. The 4× native-state equivalence on SSM hybrids is the genuinely interesting result — it suggests persistent horizontal state isn't just a Transformer trick. But 406M / 20B tokens / perplexity-only is a narrow validation corridor. The paper does exactly what it should at this stage; the question is whether the gains survive scaling. Engineers working on architecture search or efficient inference should read this now. Everyone else should wait for the 7B results.
What Is CLAIMS?
70
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
4/5
Architecture
4/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Real but incremental. Persistent horizontal latent state is a genuinely new structural idea in the residual-stream augmentation space, but it's still an augmentation, not a paradigm shift.
The field fight
Where should we add compute to scale language models — depth/width, reasoning steps, recurrence, or persistent horizontal state?
Side A
Scale depth and width; residual-stream tweaks are marginal at best
Held by: Standard Transformer scaling (Chinchilla, Llama, GPT)
Side B
Augmented state management (recurrence, residual enhancements, persistent latents) is an underexplored axis that composes with scale
Held by: Hyper-connections, Mamba, GDN, this paper
Milestone tracker
Unlocks: Confirmation that persistent latent state is a real scaling axis, not a small-model artifact
Current
406M
params
this paper
Next
1B–7B
params
scaling validation
~6-18 months depending on compute access
Verified Facts (5)
406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 over a parameter-matched Transformer
Trained on 20B FineWeb-Edu tokens
With hyper-connections and input residuals, it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer with the same enhancements) to 13.61
Lattice-GDN and Lattice-Triadic-GDN variants outperforming pure baselines with 4x larger native states
Tested on Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids
Practical Implications (4)
Architecture designers can treat persistent per-layer horizontal state as a new knob to tune alongside depth, width, and native SSM state size
The 4× native-state equivalence suggests Lattice augmentation may be more parameter-efficient than simply scaling recurrent state in SSM/hybrid models
The composability with hyper-connections and input residuals means this can be dropped into existing augmented pipelines without replacing other tricks
At 406M scale, the approach is cheap enough to test in-house — no billion-parameter commitment required to verify on your own data
More latest news
Read all