If you serve LLMs, you already know the cheapest token is the one you never recompute. Prefix caching makes that possible: when two requests start with the same text, the second one reuses the work done for the first.
That trick is easy for classic transformers. It gets tricky for hybrid models that mix different kinds of layers. SGLangβs answer is the Unified Radix Cache (URC), introduced by the LMSYS team in August 2026. This post explains the problem, the design, and how it behaves in practice, using small examples you can follow byΒ hand.
What youβllΒ learn
A note on the numbers. Token counts in the examples are small round numbers I chose for readability. The rules and the benchmark results come from the LMSYS announcement and the SGLang docs. I havenβt verified every internal detail against the source code, and I flag the spots where Iβm inferring.
When an LLM processes a prompt, it computes a KV cache: key/value tensors for every token that later tokens attend to. If a new request begins with the same tokens as an earlier one (a system prompt, a chat history, a code repository), those tensors donβt need to be recomputed.
SGLang tracks which prefixes are cached with a radix tree (a compressed prefix tree). Each edge is labeled with a run of tokens. To serve a new request, SGLang walks the tree, finds the longest cached prefix, and only prefills theΒ rest.
root β [You are a helpful assistant ...] <- shared system prompt ββββββ΄ββββββ [User: hi ...] [User: fix my bug ...] <- two different conversations
For a plain full-attention transformer this works perfectly, because cached KV for a prefix stays valid no matter what is appended later.
Many recent models mix layer types in one network. A request now carries several kinds of cached state, and each has its own notion of βreusableβ:
All three share the same token prefix, but not the same reusable boundary. If you force one boundary on all of them, you either throw away valid reuse or, worse, allow invalid reuse and get wrongΒ outputs.
Early on, SGLang handled this with separate cache classes, such as RadixCache, MambaRadixCache and SWARadixCache. Each re-implemented matching, insertion, locking and eviction. Add an orthogonal feature like HiCache (hierarchical caching) and the number of combinations multiplies.
Full only + SWA + Mamba + all three no HiCache Cache A Cache B Cache C Cache D + HiCache Cache E Cache F Cache G Cache H <- the "class matrix"
URC replaces this matrix with one tree and pluggable components. A new model family picks a combination instead of requiring a newΒ tree.
Before looking at the design, here is what each layer type actually needs to remember, and why.
In full attention, each new token looks back at all earlier tokens. So the cache holds a key/value pair for every token so far, and it grows linearly with contextΒ length.
Because attention is causal (a token only depends on tokens before it), the KV for tokens 1..N stays correct no matter what comes next. That is why any prefix length is a valid reuse point.
In sliding window attention, each token only looks at the last W tokens. Older keys and values will never be read again by these layers, so only a trailing window isΒ kept.
To resume from position p, you need the KV for tokens p-W+1Β β¦ p, and they must be contiguous. Memory is bounded by W instead of growing with the sequence.
Recurrent layers donβt store per-token keys and values. They fold the entire history into one fixed-size state, updated token by token. That makes memory constant regardless ofΒ length.
The catch: the state at position 550 is a blend of all 550 tokens. You canβt βsubtractβ tokens to get the state at position 300. So a Mamba state is only reusable at the exact position where it was saved. That position is called a checkpoint.
Hybrid models combine these. According to the LMSYS post, DeepSeek-V4 uses full attention plus SWA, Kimi-K3 uses full attention plus a recurrent (KDA) state, and Inkling uses all three.
URC is built on one separation of concerns:
Here is the picture. One tree, three kinds of attached data:
one token-keyed radix tree (the shared topology)
root βββΊ n1 βββΊ n2 βββΊ n3 βββΊ n4 β β β β FULL β β β β KV for every token on the path SWA β β β β trailing window only (β = tombstone) MAMBA β β β β checkpoint only where saved (β = none)
β = data present β = empty slot, node still in the tree
Each tree node stores references (indices) into separate memory pools: one for full-attention KV, one for SWA KV, one for Mamba state. The tensors live on the GPU in those pools, and the node only records where they are. This is my reading of the design; the older MambaRadixCache documented the same pattern of independent pools with independent eviction.
URC doesnβt make KV smaller. Full KV is still stored for every cached token. The win is architectural: one implementation of the tree plus small components, so new combinations donβt need new trees. Benefits like tiered storage and smarter eviction come from what that unified design makes possible.
The FULL component is always present. It keeps KV for every token in the matched prefix and protects the whole path of ancestor nodes while a request usesΒ it.
SWA needs a contiguous trailing window before the boundary. Older SWA slots can be freed. Their nodes stay in the tree as tombstones: empty slots that keep the topologyΒ intact.
MAMBA needs one recurrent checkpoint at the boundary. Because a request will keep updating the state as it runs, URC copies the shared checkpoint into a private slot for that request before use. Otherwise one request would corrupt the state another one dependsΒ on.
The tree core stays generic because every component implements the same hooks across the node lifecycle:
Removing one componentβs data doesnβt necessarily remove the node. Other components may still use the node as an anchor.
The SGLang docs describe how eviction cascades between components:
In short, Mamba state is the cheapest thing to lose and FULL is the most structural.
Not every memory pool deserves to be a component. URC distinguishes:
DeepSeek-V4 is the worked example in the announcement. FULL and SWA are components, and they use independent index spaces (the allocator maps tail slots in the FULL pool onto slots in the SWA pool). Compressed KV pools, indexer buffers and compressor states are sidecars, with three following FULL and two following SWA.
components (vote on reuse) sidecars (just follow) βββββββββ ββββββββββββββββββββββββ β FULL ββββββββββββββββββββββββββββ 3 auxiliary pools β βββββββββ ββββββββββββββββββββββββ βββββββββ ββββββββββββββββββββββββ β SWA ββββββββββββββββββββββββββββ 2 auxiliary pools β βββββββββ ββββββββββββββββββββββββ
GPU memory is fast but small. HiCache adds slower, larger tiers so reusable prefixes surviveΒ longer:
ββββββββββββββββββββββββββββββ β L1 GPU memory β fastest, smallest βββββββββββββββ¬βββββββββββββββ β demote βΌ β² promote βββββββββββββββ΄βββββββββββββββ β L2 Host (CPU) memory β slower, larger βββββββββββββββ¬βββββββββββββββ β demote βΌ β² promote βββββββββββββββ΄βββββββββββββββ β L3 External store β slowest, largest (e.g. Mooncake) ββββββββββββββββββββββββββββββ
Think of a long-running chat. Early turns stop fitting in GPU memory, but the user may come back to them. Without lower tiers, those prefixes would be recomputed.
Components decide what can be reused. HiCache decides where the payload lives. Since URC carries the same component identity across all three tiers, moving data down or up never changes its prefix identity or reuse rule. Components describe the transfers they need, and a controller (HybridCacheController) performs the actualΒ I/O.
The LMSYS team compared three setups on multi-turn chat workloads: GPU only, GPU plus host memory, and GPU plus host plus an external L3 store (Mooncake).
Two reading tips. First, βeffective input tokens per secondβ credits tokens served from cache, so it measures serving progress under reuse rather than raw compute. Second, the two models used different hardware and workloads, so compare tiers within a model, not acrossΒ models.
This is the core algorithm. When a request arrives:
In pseudocode (my paraphrase, not the real source):
def match_prefix(tokens): validators = [c.create_match_validator() for c in components] best = root for node in walk_full_path(tokens): # every node is a candidate if all(v.accepts(node) for v in validators): best = node # deepest safe boundary so far result = MatchResult(best) for c in components: c.finalize(result) # e.g. copy a Mamba checkpoint return result # reuse len(best) tokens
A nodeβs checkpoint or window is judged only against that nodeβs own boundary. A tombstone on n1 doesnβt hurt a candidate atΒ n4.
Each node holds 128 tokens, and the SWA window is 128 tokens. The cumulative length is shown in brackets.
The walk reaches n4, but the deepest node every component accepts is n2. The request reuses 256 tokens and prefills the remaining ones. This matches the announcementβs own figure, where traversal depth and reusable depth are different things.
Because the walk keeps going, n4 is accepted and all 512 tokens are reused. This is common in practice: after a request finishes, typically only the deepest node has a fresh SWA window and a Mamba checkpoint, while shallower nodes keep just their fullΒ KV.
n4 has a Mamba checkpoint, but thereβs no valid SWA window. The best accepted boundary is the root, so zero tokens are reused even though all the full KV is cached. A boundary must satisfy every component.
If a node holds fewer tokens than the window W, the window spans the node and part of its parent. If the parentβs SWA slots are tombstoned, the window isnβt complete. I expect the validator to reject in that case, since the announcement requires a contiguous trailing window, but I havenβt confirmed how the code tracks this.
LRU evicts whatever was used least recently. But recency isnβt the same as likelihood of reuse. Consider a coding agent: it sends a request, then spends 30 seconds running tools. Its cached prefix looks βcoldβ, yet the next request is about to need it. Under memory pressure, LRU can evict exactly that prefix while keeping unrelated one-off entries that were touched more recently.
Per the docs, FULL orders candidates by whether they are referenced, then by session reference count, then by base eviction priority. SWA and MAMBA do two passes: unreferenced first, then referenced if more room is needed. The policy currently covers GPU (L1) and host (L2) tiers, not external L3.
Memory is full. Who gets evicted? Plain LRU Session-aware βββββββββ βββββββββββββ oldest-touched first unreferenced first, then referenced [A idle 30s] β evicted! [one-off C] β evicted first [B active] [A idle 30s] β kept (referenced) [one-off C] [B active] β kept (referenced)
On SWE-bench agent workloads, the LMSYS team reported 2.9% to 16.6% lower time-to-first-token (TTFT) than ordinary HiRadixCache with LRU. Device cache hit ratio improved too, for example about 5% to 34% for Qwen3.5β397B at batch size 32, and about 42% to 51% for DeepSeek-V4-Pro at batch size 128.
An important caveat from the authors: the comparison changes both the cache implementation and the eviction policy, so it isnβt an isolated test of session awareness alone.
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 python3 -m sglang.launch_server ... --enable-session-radix-cache
Then send the same session_id on every request in a session, and call /close_session when it ends, including on error and cancellation paths.
Letβs follow two agent sessions on a model with FULL + SWA + MAMBA. Assumptions for the example: the SWA window is 100 tokens, one system prompt S is 300 tokens, and the GPU cache is small enough to fill up. Shallow SWA windows and Mamba checkpoints are freed after later turns to save memory, which is an assumption about typical behavior.
Request: S (300) + question A1 (200) = 500 tokens. The model generates 50 more.
root βββΊ n1 [tokens 1β550] FULL β KV for 550 tokens SWA β window = tokens 451β550 MAMBAβ checkpoint at 550
Cache is cold, so all 500 prompt tokens are prefilled. Session A is registered on n1.
New prompt = previous 550 + 100 new tokens = 650.
n1 is accepted, so 550 tokens are reused and only 100 are prefilled. The Mamba checkpoint is copied into a private slot first. After the turn (50 more generated), a child node isΒ added:
root βββΊ n1 [1β550] βββΊ n2 [551β700] FULL β FULL β SWA β (freed) SWA β window = 601β700 MAMBAβ (freed) MAMBAβ checkpoint at 700
Session B uses the same system prompt S but a different question B1 (150 tokens), for a prompt of 450. The walk goes into n1, matches the first 300 tokens (S), then diverges. n1 splits into [S: 1β300] and [301β550]:
root βββΊ nS [1β300] βββΊ nA1 [301β550] βββΊ n2 [551β700] βββΊ (B1 inserted after B finishes)
Each componentβs split hook decides where its data goes. FULL KV for the first 300 tokens moves to the parent nS. The SWA window and the Mamba checkpoint belonged to the end of the sequence, so they stay with the children. That leaves nS with FULLΒ only.
Now the match for B at boundary nS:
Rejected, so B reuses nothing in this simplest case and prefills all 450 tokens. This is the honest cost of hybrid models: a shared prefix is only reusable where a window and a checkpoint were saved. Text matching alone isnβt enough.
The GPU pool fills up. Entries in the cache:
Eviction also follows the cascade rules. Dropping a Mamba checkpoint frees only that checkpoint. Dropping an internal FULL node would also take its SWA and Mamba data with it, so those are the last things to go.
Aβs tool call returns and it sends turn 3. Because its prefix survived, the walk accepts n2 (700 tokens), and almost the whole prompt is a cache hit. With LRU, that prefix might have been evicted and fully recomputed.
When Aβs agent finishes, the app calls /close_session. The references are removed, but the KV stays. It returns to normal eviction order and can still serve a future request with the same prefix. If memory is tight, it becomes a natural eviction candidate.
The keyΒ ideas
On the roadmap, according to the announcement: an experimental Rust tree core to cut scheduler overhead (L1-only for now, with up to 42% lower TTFT on one SWA benchmark), direct GPU-to-L3 connections, and cache identity shared across routers and prefill/decode workers for agentic workloads.
Unified Radix Cache Explained: One Tree for Full Attention, Sliding Window and Mamba was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.