Unified Radix Cache Explained: One Tree for Full Attention, Sliding Window and Mamba SGLang introduced the Unified Radix Cache (URC) in August 2026, replacing separate cache classes such as RadixCache, MambaRadixCache and SWARadixCache with one radix tree plus pluggable components for prefix caching across hybrid models that mix full attention, sliding window attention and Mamba layers. The LMSYS team's design addresses the fact that each layer type has a different reusable boundary — full attention can reuse any prefix length, sliding window attention needs a contiguous trailing window of W tokens, and Mamba's fixed-size recurrent state cannot be rewound to an earlier position — so forcing one boundary on all layers either discards valid reuse or produces wrong outputs. URC collapses the combinatorial matrix of cache classes (which multiplied further when orthogonal features like HiCache were added) into a single tree where a new model family selects a combination instead of requiring a new tree. If you serve LLMs, you already know the cheapest token is the one you never recompute. Prefix caching makes that possible: when two requests start with the same text, the second one reuses the work done for the first. That trick is easy for classic transformers. It gets tricky for hybrid models that mix different kinds of layers. SGLang’s answer is the Unified Radix Cache URC , introduced by the LMSYS team in August 2026. This post explains the problem, the design, and how it behaves in practice, using small examples you can follow by hand. What you’ll learn A note on the numbers. Token counts in the examples are small round numbers I chose for readability. The rules and the benchmark results come from the LMSYS announcement https://www.lmsys.org/blog/2026-08-11-unified-radix-cache and the SGLang docs. I haven’t verified every internal detail against the source code, and I flag the spots where I’m inferring. When an LLM processes a prompt, it computes a KV cache : key/value tensors for every token that later tokens attend to. If a new request begins with the same tokens as an earlier one a system prompt, a chat history, a code repository , those tensors don’t need to be recomputed. SGLang tracks which prefixes are cached with a radix tree a compressed prefix tree . Each edge is labeled with a run of tokens. To serve a new request, SGLang walks the tree, finds the longest cached prefix, and only prefills the rest. root │ You are a helpful assistant ... <- shared system prompt ┌────┴─────┐ User: hi ... User: fix my bug ... <- two different conversations For a plain full-attention transformer this works perfectly, because cached KV for a prefix stays valid no matter what is appended later. Many recent models mix layer types in one network. A request now carries several kinds of cached state, and each has its own notion of “reusable”: All three share the same token prefix, but not the same reusable boundary . If you force one boundary on all of them, you either throw away valid reuse or, worse, allow invalid reuse and get wrong outputs. Early on, SGLang handled this with separate cache classes, such as RadixCache, MambaRadixCache and SWARadixCache. Each re-implemented matching, insertion, locking and eviction. Add an orthogonal feature like HiCache hierarchical caching and the number of combinations multiplies. Full only + SWA + Mamba + all three no HiCache Cache A Cache B Cache C Cache D + HiCache Cache E Cache F Cache G Cache H <- the "class matrix" URC replaces this matrix with one tree and pluggable components . A new model family picks a combination instead of requiring a new tree. Before looking at the design, here is what each layer type actually needs to remember, and why. In full attention, each new token looks back at all earlier tokens. So the cache holds a key/value pair for every token so far, and it grows linearly with context length. Because attention is causal a token only depends on tokens before it , the KV for tokens 1..N stays correct no matter what comes next. That is why any prefix length is a valid reuse point. In sliding window attention, each token only looks at the last W tokens. Older keys and values will never be read again by these layers, so only a trailing window is kept. To resume from position p , you need the KV for tokens p-W+1 … p , and they must be contiguous . Memory is bounded by W instead of growing with the sequence. Recurrent layers don’t store per-token keys and values. They fold the entire history into one fixed-size state , updated token by token. That makes memory constant regardless of length. The catch: the state at position 550 is a blend of all 550 tokens. You can’t “subtract” tokens to get the state at position 300. So a Mamba state is only reusable at the exact position where it was saved . That position is called a checkpoint. Hybrid models combine these. According to the LMSYS post, DeepSeek-V4 uses full attention plus SWA, Kimi-K3 uses full attention plus a recurrent KDA state, and Inkling uses all three. URC is built on one separation of concerns: Here is the picture. One tree, three kinds of attached data: one token-keyed radix tree the shared topology root ──► n1 ──► n2 ──► n3 ──► n4 │ │ │ │ FULL ● ● ● ● KV for every token on the path SWA ○ ○ ○ ● trailing window only ○ = tombstone MAMBA ○ ○ ○ ● checkpoint only where saved ○ = none ● = data present ○ = empty slot, node still in the tree Each tree node stores references indices into separate memory pools: one for full-attention KV, one for SWA KV, one for Mamba state. The tensors live on the GPU in those pools, and the node only records where they are. This is my reading of the design; the older MambaRadixCache documented the same pattern of independent pools with independent eviction. URC doesn’t make KV smaller. Full KV is still stored for every cached token. The win is architectural: one implementation of the tree plus small components, so new combinations don’t need new trees. Benefits like tiered storage and smarter eviction come from what that unified design makes possible. The FULL component is always present. It keeps KV for every token in the matched prefix and protects the whole path of ancestor nodes while a request uses it. SWA needs a contiguous trailing window before the boundary. Older SWA slots can be freed. Their nodes stay in the tree as tombstones : empty slots that keep the topology intact. MAMBA needs one recurrent checkpoint at the boundary. Because a request will keep updating the state as it runs, URC copies the shared checkpoint into a private slot for that request before use. Otherwise one request would corrupt the state another one depends on. The tree core stays generic because every component implements the same hooks across the node lifecycle: Removing one component’s data doesn’t necessarily remove the node. Other components may still use the node as an anchor. The SGLang docs describe how eviction cascades between components: In short, Mamba state is the cheapest thing to lose and FULL is the most structural. Not every memory pool deserves to be a component. URC distinguishes: DeepSeek-V4 is the worked example in the announcement. FULL and SWA are components, and they use independent index spaces the allocator maps tail slots in the FULL pool onto slots in the SWA pool . Compressed KV pools, indexer buffers and compressor states are sidecars, with three following FULL and two following SWA. components vote on reuse sidecars just follow ┌───────┐ ┌──────────────────────┐ │ FULL │◄─────────────────────────│ 3 auxiliary pools │ └───────┘ └──────────────────────┘ ┌───────┐ ┌──────────────────────┐ │ SWA │◄─────────────────────────│ 2 auxiliary pools │ └───────┘ └──────────────────────┘ GPU memory is fast but small. HiCache adds slower, larger tiers so reusable prefixes survive longer: ┌────────────────────────────┐ │ L1 GPU memory │ fastest, smallest └─────────────┬──────────────┘ │ demote ▼ ▲ promote ┌─────────────┴──────────────┐ │ L2 Host CPU memory │ slower, larger └─────────────┬──────────────┘ │ demote ▼ ▲ promote ┌─────────────┴──────────────┐ │ L3 External store │ slowest, largest e.g. Mooncake └────────────────────────────┘ Think of a long-running chat. Early turns stop fitting in GPU memory, but the user may come back to them. Without lower tiers, those prefixes would be recomputed. Components decide what can be reused. HiCache decides where the payload lives. Since URC carries the same component identity across all three tiers, moving data down or up never changes its prefix identity or reuse rule. Components describe the transfers they need, and a controller HybridCacheController performs the actual I/O. The LMSYS team compared three setups on multi-turn chat workloads: GPU only, GPU plus host memory, and GPU plus host plus an external L3 store Mooncake . Two reading tips. First, “effective input tokens per second” credits tokens served from cache, so it measures serving progress under reuse rather than raw compute. Second, the two models used different hardware and workloads, so compare tiers within a model, not across models. This is the core algorithm. When a request arrives: In pseudocode my paraphrase, not the real source : python def match prefix tokens : validators = c.create match validator for c in components best = root for node in walk full path tokens : every node is a candidate if all v.accepts node for v in validators : best = node deepest safe boundary so far result = MatchResult best for c in components: c.finalize result e.g. copy a Mamba checkpoint return result reuse len best tokens A node’s checkpoint or window is judged only against that node’s own boundary . A tombstone on n1 doesn’t hurt a candidate at n4. Each node holds 128 tokens, and the SWA window is 128 tokens. The cumulative length is shown in brackets. The walk reaches n4, but the deepest node every component accepts is n2 . The request reuses 256 tokens and prefills the remaining ones. This matches the announcement’s own figure, where traversal depth and reusable depth are different things. Because the walk keeps going, n4 is accepted and all 512 tokens are reused . This is common in practice: after a request finishes, typically only the deepest node has a fresh SWA window and a Mamba checkpoint, while shallower nodes keep just their full KV. n4 has a Mamba checkpoint, but there’s no valid SWA window. The best accepted boundary is the root, so zero tokens are reused even though all the full KV is cached. A boundary must satisfy every component. If a node holds fewer tokens than the window W, the window spans the node and part of its parent. If the parent’s SWA slots are tombstoned, the window isn’t complete. I expect the validator to reject in that case, since the announcement requires a contiguous trailing window, but I haven’t confirmed how the code tracks this. LRU evicts whatever was used least recently. But recency isn’t the same as likelihood of reuse. Consider a coding agent: it sends a request, then spends 30 seconds running tools. Its cached prefix looks “cold”, yet the next request is about to need it. Under memory pressure, LRU can evict exactly that prefix while keeping unrelated one-off entries that were touched more recently. Per the docs, FULL orders candidates by whether they are referenced, then by session reference count, then by base eviction priority. SWA and MAMBA do two passes: unreferenced first, then referenced if more room is needed. The policy currently covers GPU L1 and host L2 tiers, not external L3. Memory is full. Who gets evicted? Plain LRU Session-aware ───────── ───────────── oldest-touched first unreferenced first, then referenced A idle 30s ← evicted one-off C ← evicted first B active A idle 30s ← kept referenced one-off C B active ← kept referenced On SWE-bench agent workloads, the LMSYS team reported 2.9% to 16.6% lower time-to-first-token TTFT than ordinary HiRadixCache with LRU. Device cache hit ratio improved too, for example about 5% to 34% for Qwen3.5–397B at batch size 32, and about 42% to 51% for DeepSeek-V4-Pro at batch size 128. An important caveat from the authors: the comparison changes both the cache implementation and the eviction policy, so it isn’t an isolated test of session awareness alone. SGLANG ENABLE UNIFIED RADIX TREE=1 python3 -m sglang.launch server ... --enable-session-radix-cache Then send the same session id on every request in a session, and call /close session when it ends, including on error and cancellation paths. Let’s follow two agent sessions on a model with FULL + SWA + MAMBA . Assumptions for the example: the SWA window is 100 tokens , one system prompt S is 300 tokens, and the GPU cache is small enough to fill up. Shallow SWA windows and Mamba checkpoints are freed after later turns to save memory, which is an assumption about typical behavior. Request: S 300 + question A1 200 = 500 tokens. The model generates 50 more. root ──► n1 tokens 1–550 FULL ● KV for 550 tokens SWA ● window = tokens 451–550 MAMBA● checkpoint at 550 Cache is cold, so all 500 prompt tokens are prefilled. Session A is registered on n1. New prompt = previous 550 + 100 new tokens = 650. n1 is accepted, so 550 tokens are reused and only 100 are prefilled. The Mamba checkpoint is copied into a private slot first. After the turn 50 more generated , a child node is added: root ──► n1 1–550 ──► n2 551–700 FULL ● FULL ● SWA ○ freed SWA ● window = 601–700 MAMBA○ freed MAMBA● checkpoint at 700 Session B uses the same system prompt S but a different question B1 150 tokens , for a prompt of 450. The walk goes into n1, matches the first 300 tokens S , then diverges. n1 splits into S: 1–300 and 301–550 : root ──► nS 1–300 ──► nA1 301–550 ──► n2 551–700 └─► B1 inserted after B finishes Each component’s split hook decides where its data goes. FULL KV for the first 300 tokens moves to the parent nS. The SWA window and the Mamba checkpoint belonged to the end of the sequence, so they stay with the children. That leaves nS with FULL only . Now the match for B at boundary nS: Rejected, so B reuses nothing in this simplest case and prefills all 450 tokens. This is the honest cost of hybrid models: a shared prefix is only reusable where a window and a checkpoint were saved. Text matching alone isn’t enough. The GPU pool fills up. Entries in the cache: Eviction also follows the cascade rules. Dropping a Mamba checkpoint frees only that checkpoint. Dropping an internal FULL node would also take its SWA and Mamba data with it, so those are the last things to go. A’s tool call returns and it sends turn 3. Because its prefix survived, the walk accepts n2 700 tokens , and almost the whole prompt is a cache hit. With LRU, that prefix might have been evicted and fully recomputed. When A’s agent finishes, the app calls /close session. The references are removed, but the KV stays. It returns to normal eviction order and can still serve a future request with the same prefix. If memory is tight, it becomes a natural eviction candidate. The key ideas On the roadmap , according to the announcement: an experimental Rust tree core to cut scheduler overhead L1-only for now, with up to 42% lower TTFT on one SWA benchmark , direct GPU-to-L3 connections, and cache identity shared across routers and prefill/decode workers for agentic workloads. Unified Radix Cache Explained: One Tree for Full Attention, Sliding Window and Mamba https://pub.towardsai.net/unified-radix-cache-explained-one-tree-for-full-attention-sliding-window-and-mamba-be2035121ddc was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.