{"slug": "latent-lattice-language-models", "title": "Latent Lattice Language Models", "summary": "A 406M-parameter Lattice of Latents model from MIT and Adobe lowered validation perplexity to 14.40 versus 15.41 for a parameter-matched Transformer when both were trained on 20B FineWeb-Edu tokens, the researchers report. The Lattice adds per-layer horizontal latent streams that persist across tokens and condition the vertical residual stream's attention, KV-cache writes, MLP computation and decoding; combined with hyper-connections and input residuals it reached 13.61 perplexity, against 14.22 for a Transformer with the same enhancements. The approach also improved perplexity on Gated DeltaNet, Mamba-2, Triadic GDN and GDN–Mamba-2 hybrids at every tested native state size, with Lattice-GDN and Lattice-Triadic-GDN variants beating pure baselines that had 4× larger native states.", "body_md": "<sup>1</sup>MIT<sup>2</sup>Adobe\n\nAbstract\n\nLanguage models typically increase per-token computation through deeper or looped layers and latent reasoning steps,\n  while richer residual connections improve information flow across layers.\n  We instead increase *computational capacity*: the number of active computational states a model maintains.\n  We introduce the **Lattice of Latents**, a set of per-layer *horizontal latent streams* that run\n  recurrently along time, alongside the vertical residual stream that flows through depth.\n  Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream, but rather than\n  adding or rerouting residual pathways through depth, it provides *companion computational states* that persist across tokens.\n  Unlike hybrid Transformer–SSM architectures, whose recurrent states primarily serve as internal memory for sequence mixing,\n  these latents directly condition the vertical stream’s attention, KV-cache writes, MLP computation, and decoding.\n  On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40\n  over a parameter-matched Transformer.\n  These gains extend across architectural variants.\n  The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals,\n  it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer with same enhancements) to 13.61.\n  On Gated DeltaNet, Mamba-2, Triadic GDN, and GDN–Mamba-2 hybrids, it improves perplexity at every tested native state size,\n  with Lattice-GDN & Lattice-Triadic-GDN variants outperforming pure baselines with 4× larger native states.\n  These results identify persistent per-layer state as a complementary axis for scaling language-model computation.\n\nVertical residual streams flow through depth; horizontal latents persist across tokens. Switch between training and inference, and compare models.\n\nInteractive versions of the main results. Hover over any bar or point for details.\n\nLLLM (d<sub>model</sub>, d<sub>r</sub>) has a vertical residual-stream width d<sub>model</sub> and horizontal-latent width d<sub>r</sub>.\n\n**GFLOP/token**: floating-point operations per token, excluding attention scores (which add only ≈ 0.1 GFLOP/token) and the embedding lookup.\n\n      J3 runs three sweeps; its sequential rollout uses a single pass at inference (click *Show full table* for results).\n\nFineWeb-Edu validation perplexity as training continues. Circle area is proportional to parameter count.\n\nBoth LLLMs use only **22 layers** vs. 40 for the Transformer 40L, so with the same attention width (1024) they keep a 45% smaller KV cache.\n\n400M models, 20B FineWeb-Edu tokens. Lattice models narrow the vertical width from 1024 to 932 to stay iso-parameter. The Transformer and its single enhancements appear first, then the Lattice and its single enhancements, then two-enhancement combinations.\n\nHC: hyper-connections; mHC: manifold-constrained HC; value: value residual (ResFormer; learned weights for the single +value runs, fixed 0.5 in x0 + value); x0: nanochat-style input residual.\n\nA hard regime: the vertical residual stream is squeezed to 256 dimensions. Press play to add the horizontal-update enhancements one at a time and watch them recover most of the gap to a full-width Transformer. 10B-token pretraining.\n\n+ adds a component; ↻ replaces one. Parameter counts are shown under each bar.\n\nPure vs. Lattice recurrent models (∼405M parameters, 10B FineWeb-Edu tokens) across native-state sizes per mixer. Each Lattice model is iso-parameter with its pure counterpart. Bars appear from highest to lowest perplexity, each at its state-size position.\n\nNative-state size counts recurrent-memory entries per mixer (1K = 1024 scalars), excluding the horizontal latents and convolution states. Lattice models use input injection + vstream.", "url": "https://wpnews.pro/news/latent-lattice-language-models", "canonical_source": "https://shivamduggal4.github.io/latent-lattice-language-models/", "published_at": "2026-11-17 22:17:18+00:00", "updated_at": "2026-11-17 22:45:34.461218+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "natural-language-processing"], "entities": ["MIT", "Adobe", "Lattice of Latents", "FineWeb-Edu", "Gated DeltaNet", "Mamba-2", "Triadic GDN", "Transformer"], "also_reported_by": [], "takeaway": "Adding per-layer horizontal latent streams to a 406M-parameter Transformer cut validation perplexity from 15.41 to 14.40 at equal parameters on 20B FineWeb-Edu tokens, and to 13.61 when stacked with hyper-connections and input residuals.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/latent-lattice-language-models", "markdown": "https://wpnews.pro/news/latent-lattice-language-models.md", "text": "https://wpnews.pro/news/latent-lattice-language-models.txt", "jsonld": "https://wpnews.pro/news/latent-lattice-language-models.jsonld"}}