# Latent Lattice Language Models

> Source: <https://shivamduggal4.github.io/latent-lattice-language-models/>
> Published: 2026-11-17 22:17:18+00:00

**Key takeaway:** Adding per-layer horizontal latent streams to a 406M-parameter Transformer cut validation perplexity from 15.41 to 14.40 at equal parameters on 20B FineWeb-Edu tokens, and to 13.61 when stacked with hyper-connections and input residuals.

<sup>1</sup>MIT<sup>2</sup>Adobe

Abstract

Language models typically increase per-token computation through deeper or looped layers and latent reasoning steps,
  while richer residual connections improve information flow across layers.
  We instead increase *computational capacity*: the number of active computational states a model maintains.
  We introduce the **Lattice of Latents**, a set of per-layer *horizontal latent streams* that run
  recurrently along time, alongside the vertical residual stream that flows through depth.
  Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream, but rather than
  adding or rerouting residual pathways through depth, it provides *companion computational states* that persist across tokens.
  Unlike hybrid Transformer–SSM architectures, whose recurrent states primarily serve as internal memory for sequence mixing,
  these latents directly condition the vertical stream’s attention, KV-cache writes, MLP computation, and decoding.
  On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40
  over a parameter-matched Transformer.
  These gains extend across architectural variants.
  The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals,
  it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer with same enhancements) to 13.61.
  On Gated DeltaNet, Mamba-2, Triadic GDN, and GDN–Mamba-2 hybrids, it improves perplexity at every tested native state size,
  with Lattice-GDN & Lattice-Triadic-GDN variants outperforming pure baselines with 4× larger native states.
  These results identify persistent per-layer state as a complementary axis for scaling language-model computation.

Vertical residual streams flow through depth; horizontal latents persist across tokens. Switch between training and inference, and compare models.

Interactive versions of the main results. Hover over any bar or point for details.

LLLM (d<sub>model</sub>, d<sub>r</sub>) has a vertical residual-stream width d<sub>model</sub> and horizontal-latent width d<sub>r</sub>.

**GFLOP/token**: floating-point operations per token, excluding attention scores (which add only ≈ 0.1 GFLOP/token) and the embedding lookup.

      J3 runs three sweeps; its sequential rollout uses a single pass at inference (click *Show full table* for results).

FineWeb-Edu validation perplexity as training continues. Circle area is proportional to parameter count.

Both LLLMs use only **22 layers** vs. 40 for the Transformer 40L, so with the same attention width (1024) they keep a 45% smaller KV cache.

400M models, 20B FineWeb-Edu tokens. Lattice models narrow the vertical width from 1024 to 932 to stay iso-parameter. The Transformer and its single enhancements appear first, then the Lattice and its single enhancements, then two-enhancement combinations.

HC: hyper-connections; mHC: manifold-constrained HC; value: value residual (ResFormer; learned weights for the single +value runs, fixed 0.5 in x0 + value); x0: nanochat-style input residual.

A hard regime: the vertical residual stream is squeezed to 256 dimensions. Press play to add the horizontal-update enhancements one at a time and watch them recover most of the gap to a full-width Transformer. 10B-token pretraining.

+ adds a component; ↻ replaces one. Parameter counts are shown under each bar.

Pure vs. Lattice recurrent models (∼405M parameters, 10B FineWeb-Edu tokens) across native-state sizes per mixer. Each Lattice model is iso-parameter with its pure counterpart. Bars appear from highest to lowest perplexity, each at its state-size position.

Native-state size counts recurrent-memory entries per mixer (1K = 1024 scalars), excluding the horizontal latents and convolution states. Lattice models use input injection + vstream.
