What If a Transformer Never Had to Forget? Meet the Recurrent Looped Transformer (RLT)

You ask a language model a one-line question — it processes it through 48 layers. You paste a 10,000-word document — it still processes it through 48 layers. Same depth. Same ceiling. Every single time.

That's the structural limit of every decoder-only Transformer in production today.

A technical report published September 12, 2026 by Princeton researcher Yifan Zhang — Recurrent Looped Transformer (RLT) — proposes closing that loop. In most decoder-only LLMs, nothing computed at the last layer of token t feeds the first layer of token t+1; positions communicate only through attention over cached keys and values. RLT changes this: the decoder's final hidden state and its layerwise sliding-window attention (SWA) cache are carried into the next token, across both prompt and response, with no reset at the boundary.

⚠️ Important: RLT is an architectural specification, not a trained system. The report explicitly states that no measured efficiency, reasoning quality, or scaling results are reported.

Recurrent Looped Transformer (RLT) pairs a causal encoder with a recurrent decoder.

The encoder processes tokens in parallel under a causal mask and produces representations for each position. These representations are projected into key-value memory that the decoder can query later. Memory groups can be:

This memory is encoder-derived — it depends only on input tokens, not on decoder states.

The decoder holds the recurrence. Its complete state has two components:

For each token, a gated merge combines the current encoder representation with the previous decoder output. The gate controls how much previous information is carried forward. Then each decoder block runs: The next-token distribution is read from the final decoder output. Initialization happens once before the beginning-of-sequence token with a learned start state and an empty cache.

💡 Tip: RLT separates two memory stores because they serve different roles:

Encoder memory is immutable for a fixed prefix — think of it as a "textbook index" of what was said Decoder SWA cache changes every step and keeps only recent activations — think of it as the "last 2 pages of notes" Neither can replace the other.

The reference configuration uses:

Each token executes 96 logical blocks (48 encoder + 48 decoder). Zhang calls this parameter reuse, not activation copying. Two logical passes do not imply equal per-block compute — the decoder block adds cross-attention that the encoder doesn't have.

After processing t tokens, the state path traverses t × 48 decoder blocks in the reference configuration.

Per-token work stays fixed at 96 blocks, while the path's structural depth grows with the sequence. "Unbounded" means no fixed architectural upper bound on the recurrent computation path — it does not mean infinite computation per input.

The report warns that gates and contraction may suppress long paths. Structural depth is not a reasoning guarantee — it's a structural possibility that requires experimental validation.

💡 Tip — Snowball Analogy: The same amount of snow is added during every rotation (fixed compute per token), but the snowball grows because information accumulates. After 100 rotations, you have a massive snowball — without needing a bigger hill.

Encoder features use token-parallel kernels — they can process known tokens in parallel. Decoder transitions stay sequential within a sequence, but independent sequences can be batched together:

Sequence A: Token 1 → Token 2 → Token 3 → Token 4

Sequence B: Token 1 → Token 2 → Token 3 → Token 4

Sequence C: Token 1 → Token 2 → Token 3 → Token 4

↓

BATCHED ON GPU

The report states plainly:

💡 Tip for Practitioners: RLT's decoder is inherently sequential per sequence. Throughput optimization comes from batching independent sequences, not from parallelizing within one sequence. This is closer to RNN-style serving than standard transformer serving.

Pretraining, supervised fine-tuning (SFT), sampling, and reinforcement learning (RL) replay share one state transition.

During sampling:

During training:

The trainer computes the current policy probability and forms an importance ratio comparing it to the behavior policy probability. Proposition 3.1 formalizes the payoff: moving the prompt-response split leaves the conditional distribution unchanged for a fixed token history. This is mathematical equivalence — different kernels and numerical precision can still cause numerical discrepancies.

💡 Tip for ML Engineers: If your architecture has any recurrent component, old states become stale after parameter updates. Reconstruct states from the sequence start under current parameters. Record behavior log-probabilities including all sampling transformations — metadata alone cannot restore missing support from truncated sampling.

Pretraining is full-sequence next-token prediction with full backpropagation through time (BPTT). Independent documents reset:

Loss is masked to assistant targets only, but state updates are never masked. Gradients from assistant losses backpropagate through user and tool tokens. The state is not reset at an assistant boundary.

💡 Tip — The SFT Insight: The model learns how to "think about" the user's message, not just how to respond to it. Loss masking removes the loss term, not the computation or the gradient path.

Appendix B shows why partial detaching is risky. The state-to-state Jacobian has cross terms through the decoder key-value cache. The recurrent state has two components:

Detaching only the final decoder output leaves gradient paths through the SWA cache. Therefore, any truncated-BPTT scheme must explicitly identify every detached tensor.

💡 Tip: Detaching the recurrent hidden state alone is NOT sufficient to cut temporal gradient dependencies. The decoder SWA cache also carries gradient information. A complete detach requires stop-gradient on both components of the recurrent state.

An exact prefix snapshot includes:

A fixed-weight snapshot can be reused because the state is independent of the serving split. Weight updates invalidate old states. Editing a prefix forces recomputation from an earlier checkpoint. External tokens in multi-turn RL update the state but get no importance-ratio factors.

Encoder-derived memory follows: RLT keeps the encoder-derived memory but drops prompt-wide decoder skipping — every prompt token gets its full decoder update.

Temporal feedback builds on:

RLT instead feeds the previous final decoder output into the next decoder input and runs recurrence over the prompt too.

Depth-wise reuse connects to: The RL replay argument extends Zhang's prefill-decode kernel mismatch note (2026).

RLT carries the full decoder state — final output plus layerwise SWA cache — across every prompt and response token with no boundary reset

Reference configuration: 48 tied encoder + 48 tied decoder layers = 96 logical blocks per token. State path has structural depth of 48t blocks after t tokens

Hardware opportunities: Encoder parallelism and batching across sequences. No parallel scan or reduced-prefill speedup is claimed

RL replay rebuilds all states under current parameters while keeping recorded behavior log-probabilities as ratio denominators

No measured results: Reasoning quality, efficiency, and RL scaling remain open validation targets

More rounds mean more reasoning passes through the model's layer stack on accumulated context. Instead of one massive prompt, build context iteratively through back-and-forth conversation.

When AI contradicts earlier context or forgets constraints, don't just rephrase. Instead:

If building RL training pipelines, check: Reconstruct states from the sequence start under current parameters — never reuse old states after parameter updates.

Check out the Technical Report, GitHub repository, and Project Page. All credit goes to the researcher of this project.