{"slug": "unified-radix-cache-explained-one-tree-for-full-attention-sliding-window-and", "title": "Unified Radix Cache Explained: One Tree for Full Attention, Sliding Window and Mamba", "summary": "SGLang introduced the Unified Radix Cache (URC) in August 2026, replacing separate cache classes such as RadixCache, MambaRadixCache and SWARadixCache with one radix tree plus pluggable components for prefix caching across hybrid models that mix full attention, sliding window attention and Mamba layers. The LMSYS team's design addresses the fact that each layer type has a different reusable boundary — full attention can reuse any prefix length, sliding window attention needs a contiguous trailing window of W tokens, and Mamba's fixed-size recurrent state cannot be rewound to an earlier position — so forcing one boundary on all layers either discards valid reuse or produces wrong outputs. URC collapses the combinatorial matrix of cache classes (which multiplied further when orthogonal features like HiCache were added) into a single tree where a new model family selects a combination instead of requiring a new tree.", "body_md": "If you serve LLMs, you already know the cheapest token is the one you never recompute. Prefix caching makes that possible: when two requests start with the same text, the second one reuses the work done for the first.\n\nThat trick is easy for classic transformers. It gets tricky for **hybrid models** that mix different kinds of layers. SGLang’s answer is the **Unified Radix Cache (URC)**, introduced by the LMSYS team in August 2026. This post explains the problem, the design, and how it behaves in practice, using small examples you can follow by hand.\n\n**What you’ll learn**\n\n***A note on the numbers.*** *Token counts in the examples are small round numbers I chose for readability. The rules and the benchmark results come from the* [*LMSYS announcement*](https://www.lmsys.org/blog/2026-08-11-unified-radix-cache) *and the SGLang docs. I haven’t verified every internal detail against the source code, and I flag the spots where I’m inferring.*\n\nWhen an LLM processes a prompt, it computes a **KV cache**: key/value tensors for every token that later tokens attend to. If a new request begins with the same tokens as an earlier one (a system prompt, a chat history, a code repository), those tensors don’t need to be recomputed.\n\nSGLang tracks which prefixes are cached with a **radix tree** (a compressed prefix tree). Each edge is labeled with a run of tokens. To serve a new request, SGLang walks the tree, finds the longest cached prefix, and only prefills the rest.\n\n```\n              root               │        [You are a helpful assistant ...]        <- shared system prompt          ┌────┴─────┐   [User: hi ...]  [User: fix my bug ...]        <- two different conversations\n```\n\nFor a plain full-attention transformer this works perfectly, because cached KV for a prefix stays valid no matter what is appended later.\n\nMany recent models mix layer types in one network. A request now carries several kinds of cached state, and each has its own notion of “reusable”:\n\nAll three share the same token prefix, but **not the same reusable boundary**. If you force one boundary on all of them, you either throw away valid reuse or, worse, allow invalid reuse and get wrong outputs.\n\nEarly on, SGLang handled this with separate cache classes, such as RadixCache, MambaRadixCache and SWARadixCache. Each re-implemented matching, insertion, locking and eviction. Add an orthogonal feature like HiCache (hierarchical caching) and the number of combinations multiplies.\n\n```\n               Full only     + SWA       + Mamba      + all three  no HiCache   Cache A      Cache B      Cache C      Cache D  + HiCache    Cache E      Cache F      Cache G      Cache H     <- the \"class matrix\"\n```\n\nURC replaces this matrix with **one tree** and **pluggable components**. A new model family picks a combination instead of requiring a new tree.\n\nBefore looking at the design, here is what each layer type actually needs to remember, and why.\n\nIn full attention, each new token looks back at **all** earlier tokens. So the cache holds a key/value pair for every token so far, and it grows linearly with context length.\n\nBecause attention is causal (a token only depends on tokens before it), the KV for tokens 1..N stays correct no matter what comes next. That is why any prefix length is a valid reuse point.\n\nIn sliding window attention, each token only looks at the last **W** tokens. Older keys and values will never be read again by these layers, so only a **trailing window** is kept.\n\nTo resume from position *p*, you need the KV for tokens *p-W+1 … p*, and they must be **contiguous**. Memory is bounded by W instead of growing with the sequence.\n\nRecurrent layers don’t store per-token keys and values. They fold the entire history into one fixed-size **state**, updated token by token. That makes memory constant regardless of length.\n\nThe catch: the state at position 550 is a blend of all 550 tokens. You can’t “subtract” tokens to get the state at position 300. So a Mamba state is only reusable at the **exact position where it was saved**. That position is called a checkpoint.\n\nHybrid models combine these. According to the LMSYS post, DeepSeek-V4 uses full attention plus SWA, Kimi-K3 uses full attention plus a recurrent (KDA) state, and Inkling uses all three.\n\nURC is built on one separation of concerns:\n\nHere is the picture. One tree, three kinds of attached data:\n\none token-keyed radix tree (the shared topology)\n\n```\n    root ──► n1 ──► n2 ──► n3 ──► n4              │      │      │      │   FULL       ●      ●      ●      ●     KV for every token on the path   SWA        ○      ○      ○      ●     trailing window only (○ = tombstone)   MAMBA      ○      ○      ○      ●     checkpoint only where saved (○ = none)\n● = data present      ○ = empty slot, node still in the tree\n```\n\nEach tree node stores **references** (indices) into separate memory pools: one for full-attention KV, one for SWA KV, one for Mamba state. The tensors live on the GPU in those pools, and the node only records where they are. This is my reading of the design; the older MambaRadixCache documented the same pattern of independent pools with independent eviction.\n\nURC doesn’t make KV smaller. Full KV is still stored for every cached token. The win is architectural: one implementation of the tree plus small components, so new combinations don’t need new trees. Benefits like tiered storage and smarter eviction come from what that unified design makes possible.\n\nThe FULL component is always present. It keeps KV for every token in the matched prefix and protects the whole **path** of ancestor nodes while a request uses it.\n\nSWA needs a **contiguous trailing window** before the boundary. Older SWA slots can be freed. Their nodes stay in the tree as **tombstones**: empty slots that keep the topology intact.\n\nMAMBA needs **one recurrent checkpoint** at the boundary. Because a request will keep updating the state as it runs, URC **copies** the shared checkpoint into a private slot for that request before use. Otherwise one request would corrupt the state another one depends on.\n\nThe tree core stays generic because every component implements the same hooks across the node lifecycle:\n\nRemoving one component’s data doesn’t necessarily remove the node. Other components may still use the node as an anchor.\n\nThe SGLang docs describe how eviction cascades between components:\n\nIn short, Mamba state is the cheapest thing to lose and FULL is the most structural.\n\nNot every memory pool deserves to be a component. URC distinguishes:\n\nDeepSeek-V4 is the worked example in the announcement. FULL and SWA are components, and they use independent index spaces (the allocator maps tail slots in the FULL pool onto slots in the SWA pool). Compressed KV pools, indexer buffers and compressor states are sidecars, with three following FULL and two following SWA.\n\n```\n   components (vote on reuse)         sidecars (just follow)   ┌───────┐                          ┌──────────────────────┐   │ FULL  │◄─────────────────────────│ 3 auxiliary pools    │   └───────┘                          └──────────────────────┘   ┌───────┐                          ┌──────────────────────┐   │ SWA   │◄─────────────────────────│ 2 auxiliary pools    │   └───────┘                          └──────────────────────┘\n```\n\nGPU memory is fast but small. **HiCache** adds slower, larger tiers so reusable prefixes survive longer:\n\n```\n   ┌────────────────────────────┐   │ L1  GPU memory             │   fastest, smallest   └─────────────┬──────────────┘                 │  demote ▼      ▲ promote   ┌─────────────┴──────────────┐   │ L2  Host (CPU) memory      │   slower, larger   └─────────────┬──────────────┘                 │  demote ▼      ▲ promote   ┌─────────────┴──────────────┐   │ L3  External store         │   slowest, largest (e.g. Mooncake)   └────────────────────────────┘\n```\n\nThink of a long-running chat. Early turns stop fitting in GPU memory, but the user may come back to them. Without lower tiers, those prefixes would be recomputed.\n\nComponents decide **what** can be reused. HiCache decides **where** the payload lives. Since URC carries the same component identity across all three tiers, moving data down or up never changes its prefix identity or reuse rule. Components describe the transfers they need, and a controller (HybridCacheController) performs the actual I/O.\n\nThe LMSYS team compared three setups on multi-turn chat workloads: GPU only, GPU plus host memory, and GPU plus host plus an external L3 store (Mooncake).\n\nTwo reading tips. First, “effective input tokens per second” credits tokens served from cache, so it measures serving progress under reuse rather than raw compute. Second, the two models used different hardware and workloads, so compare tiers **within** a model, not across models.\n\nThis is the core algorithm. When a request arrives:\n\nIn pseudocode (my paraphrase, not the real source):\n\n``` python\ndef match_prefix(tokens):    validators = [c.create_match_validator() for c in components]    best = root    for node in walk_full_path(tokens):      # every node is a candidate        if all(v.accepts(node) for v in validators):            best = node                      # deepest safe boundary so far    result = MatchResult(best)    for c in components:        c.finalize(result)                   # e.g. copy a Mamba checkpoint    return result                            # reuse len(best) tokens\n```\n\nA node’s checkpoint or window is judged **only against that node’s own boundary**. A tombstone on n1 doesn’t hurt a candidate at n4.\n\nEach node holds 128 tokens, and the SWA window is 128 tokens. The cumulative length is shown in brackets.\n\nThe walk reaches n4, but the deepest node every component accepts is **n2**. The request reuses 256 tokens and prefills the remaining ones. This matches the announcement’s own figure, where traversal depth and reusable depth are different things.\n\nBecause the walk keeps going, n4 is accepted and **all 512 tokens are reused**. This is common in practice: after a request finishes, typically only the deepest node has a fresh SWA window and a Mamba checkpoint, while shallower nodes keep just their full KV.\n\nn4 has a Mamba checkpoint, but there’s no valid SWA window. The best accepted boundary is the root, so **zero** tokens are reused even though all the full KV is cached. A boundary must satisfy every component.\n\nIf a node holds fewer tokens than the window W, the window spans the node and part of its parent. If the parent’s SWA slots are tombstoned, the window isn’t complete. I expect the validator to reject in that case, since the announcement requires a contiguous trailing window, but I haven’t confirmed how the code tracks this.\n\nLRU evicts whatever was used least recently. But recency isn’t the same as likelihood of reuse. Consider a coding agent: it sends a request, then spends 30 seconds running tools. Its cached prefix looks “cold”, yet the next request is about to need it. Under memory pressure, LRU can evict exactly that prefix while keeping unrelated one-off entries that were touched more recently.\n\nPer the docs, FULL orders candidates by whether they are referenced, then by session reference count, then by base eviction priority. SWA and MAMBA do two passes: unreferenced first, then referenced if more room is needed. The policy currently covers GPU (L1) and host (L2) tiers, not external L3.\n\n```\nMemory is full. Who gets evicted? Plain LRU                           Session-aware ─────────                           ───────────── oldest-touched first                unreferenced first, then referenced [A idle 30s]  ← evicted!            [one-off C]    ← evicted first [B active]                          [A idle 30s]   ← kept (referenced) [one-off C]                         [B active]     ← kept (referenced)\n```\n\nOn SWE-bench agent workloads, the LMSYS team reported 2.9% to 16.6% lower time-to-first-token (TTFT) than ordinary HiRadixCache with LRU. Device cache hit ratio improved too, for example about 5% to 34% for Qwen3.5–397B at batch size 32, and about 42% to 51% for DeepSeek-V4-Pro at batch size 128.\n\nAn important caveat from the authors: the comparison changes both the cache implementation and the eviction policy, so it isn’t an isolated test of session awareness alone.\n\n```\nSGLANG_ENABLE_UNIFIED_RADIX_TREE=1 python3 -m sglang.launch_server ... --enable-session-radix-cache\n```\n\nThen send the same session_id on every request in a session, and call /close_session when it ends, including on error and cancellation paths.\n\nLet’s follow two agent sessions on a model with **FULL + SWA + MAMBA**. Assumptions for the example: the SWA window is **100 tokens**, one system prompt **S** is 300 tokens, and the GPU cache is small enough to fill up. Shallow SWA windows and Mamba checkpoints are freed after later turns to save memory, which is an assumption about typical behavior.\n\nRequest: S (300) + question A1 (200) = 500 tokens. The model generates 50 more.\n\n```\nroot ──► n1 [tokens 1–550]          FULL ● KV for 550 tokens          SWA  ● window = tokens 451–550          MAMBA● checkpoint at 550\n```\n\nCache is cold, so all 500 prompt tokens are prefilled. Session A is registered on n1.\n\nNew prompt = previous 550 + 100 new tokens = 650.\n\nn1 is accepted, so **550 tokens are reused** and only 100 are prefilled. The Mamba checkpoint is copied into a private slot first. After the turn (50 more generated), a child node is added:\n\n```\nroot ──► n1 [1–550] ──► n2 [551–700]          FULL ●          FULL ●          SWA  ○ (freed)  SWA  ● window = 601–700          MAMBA○ (freed)  MAMBA● checkpoint at 700\n```\n\nSession B uses the same system prompt S but a different question B1 (150 tokens), for a prompt of 450. The walk goes into n1, matches the first 300 tokens (S), then diverges. n1 **splits** into [S: 1–300] and [301–550]:\n\n```\nroot ──► nS [1–300] ──► nA1 [301–550] ──► n2 [551–700]                    └─► (B1 inserted after B finishes)\n```\n\nEach component’s split hook decides where its data goes. FULL KV for the first 300 tokens moves to the parent nS. The SWA window and the Mamba checkpoint belonged to the end of the sequence, so they stay with the children. That leaves **nS with FULL only**.\n\nNow the match for B at boundary nS:\n\nRejected, so B reuses nothing in this simplest case and prefills all 450 tokens. This is the honest cost of hybrid models: a shared prefix is only reusable where a window and a checkpoint were saved. Text matching alone isn’t enough.\n\nThe GPU pool fills up. Entries in the cache:\n\nEviction also follows the cascade rules. Dropping a Mamba checkpoint frees only that checkpoint. Dropping an internal FULL node would also take its SWA and Mamba data with it, so those are the last things to go.\n\nA’s tool call returns and it sends turn 3. Because its prefix survived, the walk accepts n2 (700 tokens), and almost the whole prompt is a cache hit. With LRU, that prefix might have been evicted and fully recomputed.\n\nWhen A’s agent finishes, the app calls /close_session. The references are removed, but the KV stays. It returns to normal eviction order and can still serve a future request with the same prefix. If memory is tight, it becomes a natural eviction candidate.\n\n**The key ideas**\n\n**On the roadmap**, according to the announcement: an experimental Rust tree core to cut scheduler overhead (L1-only for now, with up to 42% lower TTFT on one SWA benchmark), direct GPU-to-L3 connections, and cache identity shared across routers and prefill/decode workers for agentic workloads.\n\n[Unified Radix Cache Explained: One Tree for Full Attention, Sliding Window and Mamba](https://pub.towardsai.net/unified-radix-cache-explained-one-tree-for-full-attention-sliding-window-and-mamba-be2035121ddc) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/unified-radix-cache-explained-one-tree-for-full-attention-sliding-window-and", "canonical_source": "https://pub.towardsai.net/unified-radix-cache-explained-one-tree-for-full-attention-sliding-window-and-mamba-be2035121ddc?source=rss----98111c9905da---4", "published_at": "2026-11-18 03:06:23+00:00", "updated_at": "2026-11-18 03:09:44.198926+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-research"], "entities": ["SGLang", "LMSYS", "Unified Radix Cache", "RadixCache", "MambaRadixCache", "SWARadixCache", "HiCache"], "also_reported_by": [], "takeaway": "SGLang's Unified Radix Cache lets hybrid full-attention, sliding-window and Mamba models share one radix tree with pluggable per-layer reuse rules, so a new model family picks a component combination instead of requiring a new cache class.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/unified-radix-cache-explained-one-tree-for-full-attention-sliding-window-and", "markdown": "https://wpnews.pro/news/unified-radix-cache-explained-one-tree-for-full-attention-sliding-window-and.md", "text": "https://wpnews.pro/news/unified-radix-cache-explained-one-tree-for-full-attention-sliding-window-and.txt", "jsonld": "https://wpnews.pro/news/unified-radix-cache-explained-one-tree-for-full-attention-sliding-window-and.jsonld"}}