cd /news/large-language-models/latent-lattice-language-models · home › topics › large-language-models › article
[ARTICLE · art-171392] src=shivamduggal4.github.io ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

Latent Lattice Language Models

A 406M-parameter Lattice of Latents model from MIT and Adobe lowered validation perplexity to 14.40 versus 15.41 for a parameter-matched Transformer when both were trained on 20B FineWeb-Edu tokens, the researchers report. The Lattice adds per-layer horizontal latent streams that persist across tokens and condition the vertical residual stream's attention, KV-cache writes, MLP computation and decoding; combined with hyper-connections and input residuals it reached 13.61 perplexity, against 14.22 for a Transformer with the same enhancements. The approach also improved perplexity on Gated DeltaNet, Mamba-2, Triadic GDN and GDN–Mamba-2 hybrids at every tested native state size, with Lattice-GDN and Lattice-Triadic-GDN variants beating pure baselines that had 4× larger native states.

key takeaway

Adding per-layer horizontal latent streams to a 406M-parameter Transformer cut validation perplexity from 15.41 to 14.40 at equal parameters on 20B FineWeb-Edu tokens, and to 13.61 when stacked with hyper-connections and input residuals.

read3 min views7 publishedNov 17, 2026

<sup>1</sup>MIT<sup>2</sup>Adobe Abstract

Language models typically increase per-token computation through deeper or looped layers and latent reasoning steps, while richer residual connections improve information flow across layers. We instead increase computational capacity: the number of active computational states a model maintains. We introduce the Lattice of Latents, a set of per-layer horizontal latent streams that run recurrently along time, alongside the vertical residual stream that flows through depth. Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream, but rather than adding or rerouting residual pathways through depth, it provides companion computational states that persist across tokens. Unlike hybrid Transformer–SSM architectures, whose recurrent states primarily serve as internal memory for sequence mixing, these latents directly condition the vertical stream’s attention, KV-cache writes, MLP computation, and decoding. On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 over a parameter-matched Transformer. These gains extend across architectural variants. The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals, it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer with same enhancements) to 13.61. On Gated DeltaNet, Mamba-2, Triadic GDN, and GDN–Mamba-2 hybrids, it improves perplexity at every tested native state size, with Lattice-GDN & Lattice-Triadic-GDN variants outperforming pure baselines with 4× larger native states. These results identify persistent per-layer state as a complementary axis for scaling language-model computation.

Vertical residual streams flow through depth; horizontal latents persist across tokens. Switch between training and inference, and compare models.

Interactive versions of the main results. Hover over any bar or point for details.

LLLM (d<sub>model</sub>, d<sub>r</sub>) has a vertical residual-stream width d<sub>model</sub> and horizontal-latent width d<sub>r</sub>. GFLOP/token: floating-point operations per token, excluding attention scores (which add only ≈ 0.1 GFLOP/token) and the embedding lookup.

  J3 runs three sweeps; its sequential rollout uses a single pass at inference (click *Show full table* for results).

FineWeb-Edu validation perplexity as training continues. Circle area is proportional to parameter count.

Both LLLMs use only 22 layers vs. 40 for the Transformer 40L, so with the same attention width (1024) they keep a 45% smaller KV cache.

400M models, 20B FineWeb-Edu tokens. Lattice models narrow the vertical width from 1024 to 932 to stay iso-parameter. The Transformer and its single enhancements appear first, then the Lattice and its single enhancements, then two-enhancement combinations.

HC: hyper-connections; mHC: manifold-constrained HC; value: value residual (ResFormer; learned weights for the single +value runs, fixed 0.5 in x0 + value); x0: nanochat-style input residual. A hard regime: the vertical residual stream is squeezed to 256 dimensions. Press play to add the horizontal-update enhancements one at a time and watch them recover most of the gap to a full-width Transformer. 10B-token pretraining.

  • adds a component; ↻ replaces one. Parameter counts are shown under each bar.

Pure vs. Lattice recurrent models (∼405M parameters, 10B FineWeb-Edu tokens) across native-state sizes per mixer. Each Lattice model is iso-parameter with its pure counterpart. Bars appear from highest to lowest perplexity, each at its state-size position.

Native-state size counts recurrent-memory entries per mixer (1K = 1024 scalars), excluding the horizontal latents and convolution states. Lattice models use input injection + vstream.

── more in #large-language-models 4 stories · sorted by recency
── more on @mit 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/latent-lattice-langu…] indexed:0 read:3min 2026-11-17 · —