# What If a Transformer Never Had to Forget? Meet the Recurrent Looped Transformer (RLT)

> Source: <https://dev.to/neha_maurya/what-if-a-transformer-never-had-to-forget-meet-the-recurrent-looped-transformer-rlt-43oh>
> Published: 2026-09-15 18:34:13+00:00

You ask a language model a one-line question — it processes it through **48 layers.** You paste a 10,000-word document — it still processes it through **48 layers.** Same depth. Same ceiling. Every single time.

That's the structural limit of every decoder-only Transformer in production today.

A technical report published September 12, 2026 by Princeton researcher Yifan Zhang — **Recurrent Looped Transformer (RLT)** — proposes closing that loop. In most decoder-only LLMs, **nothing computed at the last layer of token t feeds the first layer of token t+1**; positions communicate only through attention over cached keys and values. RLT changes this: **the decoder's final hidden state and its layerwise sliding-window attention (SWA) cache are carried into the next token, across both prompt and response, with no reset at the boundary.**

⚠️ **Important:** RLT is an architectural specification, not a trained system. The report explicitly states that **no measured efficiency, reasoning quality, or scaling results are reported.**

**Recurrent Looped Transformer (RLT)** pairs a **causal encoder** with a **recurrent decoder.**

The encoder processes tokens **in parallel** under a causal mask and produces representations for each position. These representations are projected into **key-value memory** that the decoder can query later. Memory groups can be:

This memory is **encoder-derived** — it depends only on input tokens, not on decoder states.

The decoder holds the recurrence. Its complete state has **two components:**

For each token, a **gated merge** combines the current encoder representation with the previous decoder output. The gate controls how much previous information is carried forward. Then each decoder block runs:

The next-token distribution is read from the final decoder output. Initialization happens once before the beginning-of-sequence token with a **learned start state** and an **empty cache.**

💡 **Tip:** RLT separates two memory stores because they serve different roles:

**Encoder memory** is immutable for a fixed prefix — think of it as a **"textbook index"** of what was said
**Decoder SWA cache** changes every step and keeps only recent activations — think of it as the **"last 2 pages of notes"**
Neither can replace the other.

The reference configuration uses:

Each token executes **96 logical blocks** (48 encoder + 48 decoder). Zhang calls this **parameter reuse, not activation copying.** Two logical passes do not imply equal per-block compute — the decoder block adds cross-attention that the encoder doesn't have.

After processing **t tokens**, the state path traverses **t × 48 decoder blocks** in the reference configuration.

| Tokens Processed | Normal Transformer Depth | RLT State Path Depth | 
|---|---|---|
| 1 | 48 | 48 | 
| 10 | 48 | 480 | 
| 100 | 48 | 4,800 | 
| 1,000 | 48 | 48,000 | 

**Per-token work stays fixed at 96 blocks,** while the path's structural depth grows with the sequence. "Unbounded" means **no fixed architectural upper bound on the recurrent computation path** — it does not mean infinite computation per input.

The report warns that **gates and contraction may suppress long paths.** Structural depth is **not a reasoning guarantee** — it's a structural possibility that requires experimental validation.

💡 **Tip — Snowball Analogy:** The same amount of snow is added during every rotation (fixed compute per token), but the snowball grows because information accumulates. After 100 rotations, you have a massive snowball — without needing a bigger hill.

**Encoder features** use token-parallel kernels — they can process known tokens in parallel. **Decoder transitions** stay sequential within a sequence, but **independent sequences can be batched together:**

Sequence A: Token 1 → Token 2 → Token 3 → Token 4

Sequence B: Token 1 → Token 2 → Token 3 → Token 4

Sequence C: Token 1 → Token 2 → Token 3 → Token 4

↓

BATCHED ON GPU

The report states plainly:

💡 **Tip for Practitioners:** RLT's decoder is inherently sequential per sequence. Throughput optimization comes from **batching independent sequences**, not from parallelizing within one sequence. This is closer to RNN-style serving than standard transformer serving.

Pretraining, supervised fine-tuning (SFT), sampling, and reinforcement learning (RL) replay **share one state transition.**

**During sampling:**

**During training:**

The trainer computes the current policy probability and forms an **importance ratio** comparing it to the behavior policy probability. **Proposition 3.1** formalizes the payoff: moving the prompt-response split leaves the conditional distribution unchanged for a fixed token history. This is mathematical equivalence — different kernels and numerical precision can still cause numerical discrepancies.

💡 **Tip for ML Engineers:** If your architecture has any recurrent component, **old states become stale after parameter updates.** Reconstruct states from the sequence start under current parameters. Record behavior log-probabilities including all sampling transformations — metadata alone cannot restore missing support from truncated sampling.

Pretraining is **full-sequence next-token prediction** with **full backpropagation through time (BPTT).** Independent documents reset:

Loss is **masked to assistant targets only,** but **state updates are never masked.** Gradients from assistant losses **backpropagate through user and tool tokens.** The state is **not reset** at an assistant boundary.

💡 **Tip — The SFT Insight:** The model learns how to **"think about"** the user's message, not just how to respond to it. Loss masking removes the loss term, not the computation or the gradient path.

Appendix B shows why partial detaching is risky. The state-to-state Jacobian has **cross terms through the decoder key-value cache.** The recurrent state has two components:

**Detaching only the final decoder output leaves gradient paths through the SWA cache.** Therefore, any truncated-BPTT scheme must explicitly identify **every detached tensor.**

💡 **Tip:** Detaching the recurrent hidden state alone is **NOT sufficient** to cut temporal gradient dependencies. The decoder SWA cache also carries gradient information. A complete detach requires stop-gradient on **both components** of the recurrent state.

An exact prefix snapshot includes:

A fixed-weight snapshot can be reused because the state is **independent of the serving split.** Weight updates invalidate old states. Editing a prefix forces recomputation from an earlier checkpoint. External tokens in multi-turn RL update the state but **get no importance-ratio factors.**

**Encoder-derived memory** follows:

RLT keeps the encoder-derived memory but **drops prompt-wide decoder skipping** — every prompt token gets its full decoder update.

**Temporal feedback** builds on:

RLT instead feeds the **previous final decoder output** into the next decoder input and runs recurrence over the prompt too.

**Depth-wise reuse** connects to:

The RL replay argument extends Zhang's **prefill-decode kernel mismatch** note (2026).

| Feature | Standard Transformer | YOCO (2024) | Feedback Transformer | Recurrent Transformer | **RLT (2026)** | 
|---|---|---|---|---|---|
| Per-token depth | Fixed | Fixed | Fixed | Fixed | Fixed ✅ | 
| Depth grows with sequence? | ❌ | ❌ | ❌ | ❌ | **✅ Yes (t × decoder depth)** | 
| Memory type | KV cache (one type) | Encoder KV reuse | Layer feedback | Layerwise KV | **Encoder memory (global) + SWA cache (local) + Recurrent state** | 
| Prompt-response boundary | Different phases | Different | Different | Different | **Same transition, no reset** | 
| RL training consistency | Mismatch (stale states) | Not addressed | Not addressed | Not addressed | **Exact replay under current params** | 
| Hardware co-design | Generic | KV reuse | Not addressed | Tiling schedule | **Explicit parallel/sequential split** | 
| Prompt-wide decoder skip? | No | Yes (early exit) | No | No | **No — full recurrence** | 
| Empirical validation | ✅ Extensive | ✅ Some | ✅ Some | ✅ Some | **❌ Not yet** | 

**RLT carries the full decoder state** — final output plus layerwise SWA cache — across every prompt and response token **with no boundary reset**

**Reference configuration:** 48 tied encoder + 48 tied decoder layers = **96 logical blocks per token.** State path has structural depth of **48t blocks** after t tokens

**Hardware opportunities:** Encoder parallelism and batching across sequences. **No parallel scan or reduced-prefill speedup is claimed**

**RL replay** rebuilds all states under current parameters while keeping recorded behavior log-probabilities as ratio denominators

**No measured results:** Reasoning quality, efficiency, and RL scaling remain **open validation targets**

More rounds mean more reasoning passes through the model's layer stack on accumulated context. Instead of one massive prompt, build context iteratively through back-and-forth conversation.

When AI contradicts earlier context or forgets constraints, don't just rephrase. Instead:

If building RL training pipelines, check:

Reconstruct states from the sequence start under current parameters — never reuse old states after parameter updates.

**Check out the [Technical Report](https://github.com/yifanzhang-pro/recurrent-looped-tranformer), [GitHub repository](https://github.com/yifanzhang-pro/recurrent-looped-tranformer), and Project Page. All credit goes to the researcher of this project.**