cd /news/ai-infrastructure/unified-radix-cache-explained-one-tr… Β· home β€Ί topics β€Ί ai-infrastructure β€Ί article
[ARTICLE Β· art-171485] src=pub.towardsai.net β†— pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Unified Radix Cache Explained: One Tree for Full Attention, Sliding Window and Mamba

SGLang introduced the Unified Radix Cache (URC) in August 2026, replacing separate cache classes such as RadixCache, MambaRadixCache and SWARadixCache with one radix tree plus pluggable components for prefix caching across hybrid models that mix full attention, sliding window attention and Mamba layers. The LMSYS team's design addresses the fact that each layer type has a different reusable boundary β€” full attention can reuse any prefix length, sliding window attention needs a contiguous trailing window of W tokens, and Mamba's fixed-size recurrent state cannot be rewound to an earlier position β€” so forcing one boundary on all layers either discards valid reuse or produces wrong outputs. URC collapses the combinatorial matrix of cache classes (which multiplied further when orthogonal features like HiCache were added) into a single tree where a new model family selects a combination instead of requiring a new tree.

key takeaway

SGLang's Unified Radix Cache lets hybrid full-attention, sliding-window and Mamba models share one radix tree with pluggable per-layer reuse rules, so a new model family picks a component combination instead of requiring a new cache class.

by read12 min views8 publishedNov 18, 2026

If you serve LLMs, you already know the cheapest token is the one you never recompute. Prefix caching makes that possible: when two requests start with the same text, the second one reuses the work done for the first.

That trick is easy for classic transformers. It gets tricky for hybrid models that mix different kinds of layers. SGLang’s answer is the Unified Radix Cache (URC), introduced by the LMSYS team in August 2026. This post explains the problem, the design, and how it behaves in practice, using small examples you can follow byΒ hand.

What you’llΒ learn

A note on the numbers. Token counts in the examples are small round numbers I chose for readability. The rules and the benchmark results come from the LMSYS announcement and the SGLang docs. I haven’t verified every internal detail against the source code, and I flag the spots where I’m inferring.

When an LLM processes a prompt, it computes a KV cache: key/value tensors for every token that later tokens attend to. If a new request begins with the same tokens as an earlier one (a system prompt, a chat history, a code repository), those tensors don’t need to be recomputed.

SGLang tracks which prefixes are cached with a radix tree (a compressed prefix tree). Each edge is labeled with a run of tokens. To serve a new request, SGLang walks the tree, finds the longest cached prefix, and only prefills theΒ rest.

              root               β”‚        [You are a helpful assistant ...]        <- shared system prompt          β”Œβ”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”   [User: hi ...]  [User: fix my bug ...]        <- two different conversations

For a plain full-attention transformer this works perfectly, because cached KV for a prefix stays valid no matter what is appended later.

Many recent models mix layer types in one network. A request now carries several kinds of cached state, and each has its own notion of β€œreusable”:

All three share the same token prefix, but not the same reusable boundary. If you force one boundary on all of them, you either throw away valid reuse or, worse, allow invalid reuse and get wrongΒ outputs.

Early on, SGLang handled this with separate cache classes, such as RadixCache, MambaRadixCache and SWARadixCache. Each re-implemented matching, insertion, locking and eviction. Add an orthogonal feature like HiCache (hierarchical caching) and the number of combinations multiplies.

               Full only     + SWA       + Mamba      + all three  no HiCache   Cache A      Cache B      Cache C      Cache D  + HiCache    Cache E      Cache F      Cache G      Cache H     <- the "class matrix"

URC replaces this matrix with one tree and pluggable components. A new model family picks a combination instead of requiring a newΒ tree.

Before looking at the design, here is what each layer type actually needs to remember, and why.

In full attention, each new token looks back at all earlier tokens. So the cache holds a key/value pair for every token so far, and it grows linearly with contextΒ length.

Because attention is causal (a token only depends on tokens before it), the KV for tokens 1..N stays correct no matter what comes next. That is why any prefix length is a valid reuse point.

In sliding window attention, each token only looks at the last W tokens. Older keys and values will never be read again by these layers, so only a trailing window isΒ kept.

To resume from position p, you need the KV for tokens p-W+1 … p, and they must be contiguous. Memory is bounded by W instead of growing with the sequence.

Recurrent layers don’t store per-token keys and values. They fold the entire history into one fixed-size state, updated token by token. That makes memory constant regardless ofΒ length.

The catch: the state at position 550 is a blend of all 550 tokens. You can’t β€œsubtract” tokens to get the state at position 300. So a Mamba state is only reusable at the exact position where it was saved. That position is called a checkpoint.

Hybrid models combine these. According to the LMSYS post, DeepSeek-V4 uses full attention plus SWA, Kimi-K3 uses full attention plus a recurrent (KDA) state, and Inkling uses all three.

URC is built on one separation of concerns:

Here is the picture. One tree, three kinds of attached data:

one token-keyed radix tree (the shared topology)

    root ──► n1 ──► n2 ──► n3 ──► n4              β”‚      β”‚      β”‚      β”‚   FULL       ●      ●      ●      ●     KV for every token on the path   SWA        β—‹      β—‹      β—‹      ●     trailing window only (β—‹ = tombstone)   MAMBA      β—‹      β—‹      β—‹      ●     checkpoint only where saved (β—‹ = none)
● = data present      β—‹ = empty slot, node still in the tree

Each tree node stores references (indices) into separate memory pools: one for full-attention KV, one for SWA KV, one for Mamba state. The tensors live on the GPU in those pools, and the node only records where they are. This is my reading of the design; the older MambaRadixCache documented the same pattern of independent pools with independent eviction.

URC doesn’t make KV smaller. Full KV is still stored for every cached token. The win is architectural: one implementation of the tree plus small components, so new combinations don’t need new trees. Benefits like tiered storage and smarter eviction come from what that unified design makes possible.

The FULL component is always present. It keeps KV for every token in the matched prefix and protects the whole path of ancestor nodes while a request usesΒ it.

SWA needs a contiguous trailing window before the boundary. Older SWA slots can be freed. Their nodes stay in the tree as tombstones: empty slots that keep the topologyΒ intact.

MAMBA needs one recurrent checkpoint at the boundary. Because a request will keep updating the state as it runs, URC copies the shared checkpoint into a private slot for that request before use. Otherwise one request would corrupt the state another one dependsΒ on.

The tree core stays generic because every component implements the same hooks across the node lifecycle:

Removing one component’s data doesn’t necessarily remove the node. Other components may still use the node as an anchor.

The SGLang docs describe how eviction cascades between components:

In short, Mamba state is the cheapest thing to lose and FULL is the most structural.

Not every memory pool deserves to be a component. URC distinguishes:

DeepSeek-V4 is the worked example in the announcement. FULL and SWA are components, and they use independent index spaces (the allocator maps tail slots in the FULL pool onto slots in the SWA pool). Compressed KV pools, indexer buffers and compressor states are sidecars, with three following FULL and two following SWA.

   components (vote on reuse)         sidecars (just follow)   β”Œβ”€β”€β”€β”€β”€β”€β”€β”                          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”‚ FULL  │◄─────────────────────────│ 3 auxiliary pools    β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”˜                          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β”Œβ”€β”€β”€β”€β”€β”€β”€β”                          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”‚ SWA   │◄─────────────────────────│ 2 auxiliary pools    β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”˜                          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

GPU memory is fast but small. HiCache adds slower, larger tiers so reusable prefixes surviveΒ longer:

   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”‚ L1  GPU memory             β”‚   fastest, smallest   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                 β”‚  demote β–Ό      β–² promote   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”‚ L2  Host (CPU) memory      β”‚   slower, larger   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                 β”‚  demote β–Ό      β–² promote   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”‚ L3  External store         β”‚   slowest, largest (e.g. Mooncake)   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Think of a long-running chat. Early turns stop fitting in GPU memory, but the user may come back to them. Without lower tiers, those prefixes would be recomputed.

Components decide what can be reused. HiCache decides where the payload lives. Since URC carries the same component identity across all three tiers, moving data down or up never changes its prefix identity or reuse rule. Components describe the transfers they need, and a controller (HybridCacheController) performs the actualΒ I/O.

The LMSYS team compared three setups on multi-turn chat workloads: GPU only, GPU plus host memory, and GPU plus host plus an external L3 store (Mooncake).

Two reading tips. First, β€œeffective input tokens per second” credits tokens served from cache, so it measures serving progress under reuse rather than raw compute. Second, the two models used different hardware and workloads, so compare tiers within a model, not acrossΒ models.

This is the core algorithm. When a request arrives:

In pseudocode (my paraphrase, not the real source):

def match_prefix(tokens):    validators = [c.create_match_validator() for c in components]    best = root    for node in walk_full_path(tokens):      # every node is a candidate        if all(v.accepts(node) for v in validators):            best = node                      # deepest safe boundary so far    result = MatchResult(best)    for c in components:        c.finalize(result)                   # e.g. copy a Mamba checkpoint    return result                            # reuse len(best) tokens

A node’s checkpoint or window is judged only against that node’s own boundary. A tombstone on n1 doesn’t hurt a candidate atΒ n4.

Each node holds 128 tokens, and the SWA window is 128 tokens. The cumulative length is shown in brackets.

The walk reaches n4, but the deepest node every component accepts is n2. The request reuses 256 tokens and prefills the remaining ones. This matches the announcement’s own figure, where traversal depth and reusable depth are different things.

Because the walk keeps going, n4 is accepted and all 512 tokens are reused. This is common in practice: after a request finishes, typically only the deepest node has a fresh SWA window and a Mamba checkpoint, while shallower nodes keep just their fullΒ KV.

n4 has a Mamba checkpoint, but there’s no valid SWA window. The best accepted boundary is the root, so zero tokens are reused even though all the full KV is cached. A boundary must satisfy every component.

If a node holds fewer tokens than the window W, the window spans the node and part of its parent. If the parent’s SWA slots are tombstoned, the window isn’t complete. I expect the validator to reject in that case, since the announcement requires a contiguous trailing window, but I haven’t confirmed how the code tracks this.

LRU evicts whatever was used least recently. But recency isn’t the same as likelihood of reuse. Consider a coding agent: it sends a request, then spends 30 seconds running tools. Its cached prefix looks β€œcold”, yet the next request is about to need it. Under memory pressure, LRU can evict exactly that prefix while keeping unrelated one-off entries that were touched more recently.

Per the docs, FULL orders candidates by whether they are referenced, then by session reference count, then by base eviction priority. SWA and MAMBA do two passes: unreferenced first, then referenced if more room is needed. The policy currently covers GPU (L1) and host (L2) tiers, not external L3.

Memory is full. Who gets evicted? Plain LRU                           Session-aware ─────────                           ───────────── oldest-touched first                unreferenced first, then referenced [A idle 30s]  ← evicted!            [one-off C]    ← evicted first [B active]                          [A idle 30s]   ← kept (referenced) [one-off C]                         [B active]     ← kept (referenced)

On SWE-bench agent workloads, the LMSYS team reported 2.9% to 16.6% lower time-to-first-token (TTFT) than ordinary HiRadixCache with LRU. Device cache hit ratio improved too, for example about 5% to 34% for Qwen3.5–397B at batch size 32, and about 42% to 51% for DeepSeek-V4-Pro at batch size 128.

An important caveat from the authors: the comparison changes both the cache implementation and the eviction policy, so it isn’t an isolated test of session awareness alone.

SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 python3 -m sglang.launch_server ... --enable-session-radix-cache

Then send the same session_id on every request in a session, and call /close_session when it ends, including on error and cancellation paths.

Let’s follow two agent sessions on a model with FULL + SWA + MAMBA. Assumptions for the example: the SWA window is 100 tokens, one system prompt S is 300 tokens, and the GPU cache is small enough to fill up. Shallow SWA windows and Mamba checkpoints are freed after later turns to save memory, which is an assumption about typical behavior.

Request: S (300) + question A1 (200) = 500 tokens. The model generates 50 more.

root ──► n1 [tokens 1–550]          FULL ● KV for 550 tokens          SWA  ● window = tokens 451–550          MAMBA● checkpoint at 550

Cache is cold, so all 500 prompt tokens are prefilled. Session A is registered on n1.

New prompt = previous 550 + 100 new tokens = 650.

n1 is accepted, so 550 tokens are reused and only 100 are prefilled. The Mamba checkpoint is copied into a private slot first. After the turn (50 more generated), a child node isΒ added:

root ──► n1 [1–550] ──► n2 [551–700]          FULL ●          FULL ●          SWA  β—‹ (freed)  SWA  ● window = 601–700          MAMBAβ—‹ (freed)  MAMBA● checkpoint at 700

Session B uses the same system prompt S but a different question B1 (150 tokens), for a prompt of 450. The walk goes into n1, matches the first 300 tokens (S), then diverges. n1 splits into [S: 1–300] and [301–550]:

root ──► nS [1–300] ──► nA1 [301–550] ──► n2 [551–700]                    └─► (B1 inserted after B finishes)

Each component’s split hook decides where its data goes. FULL KV for the first 300 tokens moves to the parent nS. The SWA window and the Mamba checkpoint belonged to the end of the sequence, so they stay with the children. That leaves nS with FULLΒ only.

Now the match for B at boundary nS:

Rejected, so B reuses nothing in this simplest case and prefills all 450 tokens. This is the honest cost of hybrid models: a shared prefix is only reusable where a window and a checkpoint were saved. Text matching alone isn’t enough.

The GPU pool fills up. Entries in the cache:

Eviction also follows the cascade rules. Dropping a Mamba checkpoint frees only that checkpoint. Dropping an internal FULL node would also take its SWA and Mamba data with it, so those are the last things to go.

A’s tool call returns and it sends turn 3. Because its prefix survived, the walk accepts n2 (700 tokens), and almost the whole prompt is a cache hit. With LRU, that prefix might have been evicted and fully recomputed.

When A’s agent finishes, the app calls /close_session. The references are removed, but the KV stays. It returns to normal eviction order and can still serve a future request with the same prefix. If memory is tight, it becomes a natural eviction candidate.

The keyΒ ideas

On the roadmap, according to the announcement: an experimental Rust tree core to cut scheduler overhead (L1-only for now, with up to 42% lower TTFT on one SWA benchmark), direct GPU-to-L3 connections, and cache identity shared across routers and prefill/decode workers for agentic workloads.

Unified Radix Cache Explained: One Tree for Full Attention, Sliding Window and Mamba was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

── more in #ai-infrastructure 4 stories Β· sorted by recency
── more on @sglang 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/unified-radix-cache-…] indexed:0 read:12min 2026-11-18 Β· β€”