{"slug": "what-if-a-transformer-never-had-to-forget-meet-the-recurrent-looped-transformer", "title": "What If a Transformer Never Had to Forget? Meet the Recurrent Looped Transformer (RLT)", "summary": "Princeton researcher Yifan Zhang published a technical report on September 12, 2026 proposing the Recurrent Looped Transformer (RLT), an architecture that carries the decoder's final hidden state and sliding-window attention cache into the next token instead of resetting at each position. RLT pairs a causal encoder with a recurrent decoder, giving the state path a structural depth that grows with sequence length while per-token work stays fixed at 96 logical blocks. The report explicitly notes that RLT is an architectural specification only, with no measured efficiency, reasoning quality, or scaling results reported.", "body_md": "You ask a language model a one-line question — it processes it through **48 layers.** You paste a 10,000-word document — it still processes it through **48 layers.** Same depth. Same ceiling. Every single time.\n\nThat's the structural limit of every decoder-only Transformer in production today.\n\nA technical report published September 12, 2026 by Princeton researcher Yifan Zhang — **Recurrent Looped Transformer (RLT)** — proposes closing that loop. In most decoder-only LLMs, **nothing computed at the last layer of token t feeds the first layer of token t+1**; positions communicate only through attention over cached keys and values. RLT changes this: **the decoder's final hidden state and its layerwise sliding-window attention (SWA) cache are carried into the next token, across both prompt and response, with no reset at the boundary.**\n\n⚠️ **Important:** RLT is an architectural specification, not a trained system. The report explicitly states that **no measured efficiency, reasoning quality, or scaling results are reported.**\n\n**Recurrent Looped Transformer (RLT)** pairs a **causal encoder** with a **recurrent decoder.**\n\nThe encoder processes tokens **in parallel** under a causal mask and produces representations for each position. These representations are projected into **key-value memory** that the decoder can query later. Memory groups can be:\n\nThis memory is **encoder-derived** — it depends only on input tokens, not on decoder states.\n\nThe decoder holds the recurrence. Its complete state has **two components:**\n\nFor each token, a **gated merge** combines the current encoder representation with the previous decoder output. The gate controls how much previous information is carried forward. Then each decoder block runs:\n\nThe next-token distribution is read from the final decoder output. Initialization happens once before the beginning-of-sequence token with a **learned start state** and an **empty cache.**\n\n💡 **Tip:** RLT separates two memory stores because they serve different roles:\n\n**Encoder memory** is immutable for a fixed prefix — think of it as a **\"textbook index\"** of what was said\n**Decoder SWA cache** changes every step and keeps only recent activations — think of it as the **\"last 2 pages of notes\"**\nNeither can replace the other.\n\nThe reference configuration uses:\n\nEach token executes **96 logical blocks** (48 encoder + 48 decoder). Zhang calls this **parameter reuse, not activation copying.** Two logical passes do not imply equal per-block compute — the decoder block adds cross-attention that the encoder doesn't have.\n\nAfter processing **t tokens**, the state path traverses **t × 48 decoder blocks** in the reference configuration.\n\n| Tokens Processed | Normal Transformer Depth | RLT State Path Depth | \n|---|---|---|\n| 1 | 48 | 48 | \n| 10 | 48 | 480 | \n| 100 | 48 | 4,800 | \n| 1,000 | 48 | 48,000 | \n\n**Per-token work stays fixed at 96 blocks,** while the path's structural depth grows with the sequence. \"Unbounded\" means **no fixed architectural upper bound on the recurrent computation path** — it does not mean infinite computation per input.\n\nThe report warns that **gates and contraction may suppress long paths.** Structural depth is **not a reasoning guarantee** — it's a structural possibility that requires experimental validation.\n\n💡 **Tip — Snowball Analogy:** The same amount of snow is added during every rotation (fixed compute per token), but the snowball grows because information accumulates. After 100 rotations, you have a massive snowball — without needing a bigger hill.\n\n**Encoder features** use token-parallel kernels — they can process known tokens in parallel. **Decoder transitions** stay sequential within a sequence, but **independent sequences can be batched together:**\n\nSequence A: Token 1 → Token 2 → Token 3 → Token 4\n\nSequence B: Token 1 → Token 2 → Token 3 → Token 4\n\nSequence C: Token 1 → Token 2 → Token 3 → Token 4\n\n↓\n\nBATCHED ON GPU\n\nThe report states plainly:\n\n💡 **Tip for Practitioners:** RLT's decoder is inherently sequential per sequence. Throughput optimization comes from **batching independent sequences**, not from parallelizing within one sequence. This is closer to RNN-style serving than standard transformer serving.\n\nPretraining, supervised fine-tuning (SFT), sampling, and reinforcement learning (RL) replay **share one state transition.**\n\n**During sampling:**\n\n**During training:**\n\nThe trainer computes the current policy probability and forms an **importance ratio** comparing it to the behavior policy probability. **Proposition 3.1** formalizes the payoff: moving the prompt-response split leaves the conditional distribution unchanged for a fixed token history. This is mathematical equivalence — different kernels and numerical precision can still cause numerical discrepancies.\n\n💡 **Tip for ML Engineers:** If your architecture has any recurrent component, **old states become stale after parameter updates.** Reconstruct states from the sequence start under current parameters. Record behavior log-probabilities including all sampling transformations — metadata alone cannot restore missing support from truncated sampling.\n\nPretraining is **full-sequence next-token prediction** with **full backpropagation through time (BPTT).** Independent documents reset:\n\nLoss is **masked to assistant targets only,** but **state updates are never masked.** Gradients from assistant losses **backpropagate through user and tool tokens.** The state is **not reset** at an assistant boundary.\n\n💡 **Tip — The SFT Insight:** The model learns how to **\"think about\"** the user's message, not just how to respond to it. Loss masking removes the loss term, not the computation or the gradient path.\n\nAppendix B shows why partial detaching is risky. The state-to-state Jacobian has **cross terms through the decoder key-value cache.** The recurrent state has two components:\n\n**Detaching only the final decoder output leaves gradient paths through the SWA cache.** Therefore, any truncated-BPTT scheme must explicitly identify **every detached tensor.**\n\n💡 **Tip:** Detaching the recurrent hidden state alone is **NOT sufficient** to cut temporal gradient dependencies. The decoder SWA cache also carries gradient information. A complete detach requires stop-gradient on **both components** of the recurrent state.\n\nAn exact prefix snapshot includes:\n\nA fixed-weight snapshot can be reused because the state is **independent of the serving split.** Weight updates invalidate old states. Editing a prefix forces recomputation from an earlier checkpoint. External tokens in multi-turn RL update the state but **get no importance-ratio factors.**\n\n**Encoder-derived memory** follows:\n\nRLT keeps the encoder-derived memory but **drops prompt-wide decoder skipping** — every prompt token gets its full decoder update.\n\n**Temporal feedback** builds on:\n\nRLT instead feeds the **previous final decoder output** into the next decoder input and runs recurrence over the prompt too.\n\n**Depth-wise reuse** connects to:\n\nThe RL replay argument extends Zhang's **prefill-decode kernel mismatch** note (2026).\n\n| Feature | Standard Transformer | YOCO (2024) | Feedback Transformer | Recurrent Transformer | **RLT (2026)** | \n|---|---|---|---|---|---|\n| Per-token depth | Fixed | Fixed | Fixed | Fixed | Fixed ✅ | \n| Depth grows with sequence? | ❌ | ❌ | ❌ | ❌ | **✅ Yes (t × decoder depth)** | \n| Memory type | KV cache (one type) | Encoder KV reuse | Layer feedback | Layerwise KV | **Encoder memory (global) + SWA cache (local) + Recurrent state** | \n| Prompt-response boundary | Different phases | Different | Different | Different | **Same transition, no reset** | \n| RL training consistency | Mismatch (stale states) | Not addressed | Not addressed | Not addressed | **Exact replay under current params** | \n| Hardware co-design | Generic | KV reuse | Not addressed | Tiling schedule | **Explicit parallel/sequential split** | \n| Prompt-wide decoder skip? | No | Yes (early exit) | No | No | **No — full recurrence** | \n| Empirical validation | ✅ Extensive | ✅ Some | ✅ Some | ✅ Some | **❌ Not yet** | \n\n**RLT carries the full decoder state** — final output plus layerwise SWA cache — across every prompt and response token **with no boundary reset**\n\n**Reference configuration:** 48 tied encoder + 48 tied decoder layers = **96 logical blocks per token.** State path has structural depth of **48t blocks** after t tokens\n\n**Hardware opportunities:** Encoder parallelism and batching across sequences. **No parallel scan or reduced-prefill speedup is claimed**\n\n**RL replay** rebuilds all states under current parameters while keeping recorded behavior log-probabilities as ratio denominators\n\n**No measured results:** Reasoning quality, efficiency, and RL scaling remain **open validation targets**\n\nMore rounds mean more reasoning passes through the model's layer stack on accumulated context. Instead of one massive prompt, build context iteratively through back-and-forth conversation.\n\nWhen AI contradicts earlier context or forgets constraints, don't just rephrase. Instead:\n\nIf building RL training pipelines, check:\n\nReconstruct states from the sequence start under current parameters — never reuse old states after parameter updates.\n\n**Check out the [Technical Report](https://github.com/yifanzhang-pro/recurrent-looped-tranformer), [GitHub repository](https://github.com/yifanzhang-pro/recurrent-looped-tranformer), and Project Page. All credit goes to the researcher of this project.**", "url": "https://wpnews.pro/news/what-if-a-transformer-never-had-to-forget-meet-the-recurrent-looped-transformer", "canonical_source": "https://dev.to/neha_maurya/what-if-a-transformer-never-had-to-forget-meet-the-recurrent-looped-transformer-rlt-43oh", "published_at": "2026-09-15 18:34:13+00:00", "updated_at": "2026-09-15 18:49:10.457834+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "neural-networks"], "entities": ["Yifan Zhang", "Princeton University", "Recurrent Looped Transformer", "RLT"], "alternates": {"html": "https://wpnews.pro/news/what-if-a-transformer-never-had-to-forget-meet-the-recurrent-looped-transformer", "markdown": "https://wpnews.pro/news/what-if-a-transformer-never-had-to-forget-meet-the-recurrent-looped-transformer.md", "text": "https://wpnews.pro/news/what-if-a-transformer-never-had-to-forget-meet-the-recurrent-looped-transformer.txt", "jsonld": "https://wpnews.pro/news/what-if-a-transformer-never-had-to-forget-meet-the-recurrent-looped-transformer.jsonld"}}