Rudy Pei (NVIDIA)| August 25, 2026
In large-scale LLM serving, the orchestration layer is critical to the overall system efficiency. How effectively the orchestration layer distributes heterogeneous request streams across a fleet of LLM engine replicas directly impacts serving TTFT, TPOT, and throughput. When operating a cluster of vLLM replicas, the most critical decision your router makes is simple: Who gets the next request? In this blog, we’ll unpack the misconception of optimizing solely for KV cache reuse, introduce token-load-aware routing, and explain why it is critical for efficient LLM serving.
LinkChallenges of LLM Request Routing #
LLM serving differs fundamentally from traditional microservice routing due to the autoregressive nature of inference and the high variability of LLM requests. Traditional microservices typically process requests independently, with little benefit from routing related requests to the same replica. LLM serving, in contrast, introduces statefulness, high heterogeneity, and unpredictable execution costs.
Stateful execution: An engine can reuse KV caches computed by previous requests. Routing a request to a replica that already holds its prefix can significantly reduce prefill computation and TTFT.High heterogeneity: Requests vary widely in input and output sequence lengths, leading to large differences in GPU resource usage and execution time.Non-deterministic generation: Even for the same input, the number of generated output tokens can vary, making the total cost of a request difficult to predict in advance.
Together, these properties create a unique challenge for the orchestration layer: how to take advantage of KV cache reuse while balancing heterogeneity across engine replicas.
LinkNaive Optimization: Maximize KV Cache Reuse #
Maximizing KV cache reuse is the most naive approach to reduce TTFT and optimize serving efficiency. There are three common ways to maximize KV cache reuse: session affinity, prefix affinity, and KV cache affinity.
LinkSession Affinity
Ray Serve LLM supports session affinity through consistent hashing, which requires clients to associate requests from the same session with a session ID. For multi-turn workloads, this routes subsequent turns to the same replica, allowing them to reuse KV caches computed by earlier turns.
An added benefit of consistent hashing for session affinity is session-level load balancing: with enough virtual nodes, it distributes sessions evenly across replicas.
session_affinity_config = LLMConfig(
model__config=...,
deployment_config=(
"request_router_config": RequestRouterConfig(
request_router_class=(
"ray.serve.experimental.",
"consistent_hash_router.ConsistentHashRouter"
)
),
engine_kwargs={...},
)
LinkPrefix Affinity
Based on incoming requests, Ray Serve LLM’s prefix affinity router builds a prefix tree that tracks which prefixes have been sent to each replica. Later requests walk the tree to find the replica with the largest prefix overlap.
Approximate: The prefix tree tracks request strings, not the actual KV cache. KV caching operates at the token level, so the router’s view may differ from the engine’s actual cache state.Not eviction-aware: The router assumes previously seen prefixes remain cached. Under high concurrency or long running services, eviction can make this view stale.KV cache offreduces cache misses, but the router still treats all cache tiers equally, ignoring the cost of re offloaded blocks back to GPU memory.
prefix_affinity_config = LLMConfig(
model__config=...,
deployment_config=(
"request_router_config": RequestRouterConfig(
request_router_class=(
"ray.serve.llm.",
"request_router.PrefixCacheAffinityRouter",
),
engine_kwargs={...},
)
LinkKV Cache Affinity
Ray Serve LLM’s KV aware router ingests KV cache events from the engines to build a global radix tree of KV cache blocks. vLLM emits events as cache blocks are created and removed, giving the router an up-to-date view of KV cache state across replicas and allowing it to compute overlap at the KV cache block level for each request.
We collaborate with the NVIDIA Dynamo team to expose Dynamo’s KV indexer through modular interfaces to external libraries. The event, request, and data planes remain Ray-native, while Dynamo’s KV indexer computes KV cache overlap and contributes to request scoring. The resulting scores are then used by Ray Serve’s request plane to select the target replica.
kv_affinity_config = LLMConfig(
model__config=...,
deployment_config=(
"request_router_config": RequestRouterConfig(
request_router_class=(
"ray.serve.llm.",
"request_router.KVAwareRouter",
),
engine_kwargs={...},
)
Beyond KV cache overlap, Ray Serve LLM’s KV cache affinity router also accounts for token load when making routing decisions, which we’ll explore in the following sections.
LinkWhy Isn’t Maximizing KV Cache Reuse Alone Enough? #
Optimizing KV cache reuse does not necessarily maximize serving performance. Consider an asynchronous multi-turn RL rollout workload with stragglers, where requests vary significantly in input and output sequence lengths.
Each step runs eight rollouts, with 10 turns per rollout.
Each rollout starts with a 2K-token input, and each turn generates 1K tokens, except for two stragglers whose final turn generates 8K tokens.
For asynchronous RL rollouts, the goal is typically to minimize step time or maximize throughput at a fixed concurrency. We use p99 end-to-end rollout latency as a proxy for step time. We compare three router variants under concurrency=16 as shown in Figure 4:
PureKVCacheAffinityRouter
, which modifiesKVAwareRouter
to optimizesolely for KV cache overlap.ConsistentHashRouter
, which also maximizes KV cache reuse for multi-turn conversations while providing session-level load balancing. We assign each multi-turn rollout a session ID.KVAwareRouter
, which considers both KV cache overlap and token load.
KVAwareRouter
has the best p99 rollout end-to-end latency despite lower prefix cache hit rate. Taking a deeper look into the KV cache hit rate and token load* which estimates the active KV cache footprint of decoding requests on a replica, KVAwareRouter
trades off KV cache hit rate for more balanced token load.
LinkRequest Herding
Pure KV cache affinity supported by PureKVCacheAffinityRouter
can cause request herding. Once a replica caches a shared prefix, such as the system prompt, subsequent requests see cache overlap and preferentially route to that replica. Requests from many different conversations collide on the same replica and create token load imbalance.
LinkLoad Balancing
In consistent hashing, with enough virtual nodes, it balances the number of sessions across replicas, but it perceives those sessions uniformly. In LLM serving, they are not: one session may contain short requests, while another may become a long-running straggler. Multiple stragglers can land on the same replica, creating load imbalance despite an even request distribution.
While session affinity does an excellent job of preserving KV cache locality, request-level balance does not guarantee actual load balance.
LinkBeyond KV Cache Reuse: Token Load Awareness Matters More #
Let’s consider the work an engine actually performs for each request. A request goes through two phases:
Prefill: The engine first reuses any overlapping KV cache, then computes the KV cache for the remaining uncached input tokens.Decode: Once prefill completes, the request starts generating output tokens. At every decode step, the engine accesses the KV cache for all tokens generated so far and performs another forward pass.
This reveals the limitation of using KV cache overlap as the routing objective. KV cache overlap tells us how much prefill work can be saved, but not how much work the engine still has to do. That remaining load comes from two sources:
Prefill load: active prefill tokens plus incoming tokens that are not already cached.Decode load: ongoing decode work from active requests.
Token load combines both into a single metric, capturing the compute-bound prefill load and memory-bound decode load on each replica.
KV cache affinity is still important, but its role changes: it is a proxy for load estimation, not the routing objective itself. KV cache overlap tells us how much of the incoming prefill can be skipped. We use that to estimate the remaining prefill load, then combine it with ongoing decode load to estimate the overall token load.
KVAwareRouter
is the routing policy with token-load awareness in Ray Serve LLM. It also provides some additional benefits over session affinity with consistent hashing:
Automatic KV cache reuse: Session affinity through consistent hashing requires clients to provide session IDs, so applications need to explicitly define which requests belong to the same session.KVAwareRouter
instead discovers and reuses available KV cache automatically, without requiring session IDs.Flexibility: Based on your workloads, you can adaptcoefficientsfrom the scoring function.
Ray Serve LLM tracks each request throughout its lifecycle: when it is admitted, when the first token is generated, as decode progresses, and when the request completes. At each phase, the corresponding token load is updated at the router, maintaining an up-to-date view of the load across all engine replicas. Ray Serve LLM provides the end-to-end request and control planes, while integrating Dynamo’s selection service module for request scoring.
From a scalability perspective, Ray Serve LLM replicates the router and maintains an eventually consistent view of token load across router replicas. KVAwareRouter
makes routing decisions locally, then asynchronously shares load updates with peer routers. This keeps routing and token streaming off the synchronization path while allowing all routers to converge on the same load view.
Nevertheless, session affinity with consistent hashing remains a strong choice for workloads where:
Cache misses are particularly expensive: For long multi-turn conversations, keeping all turns on the same replica guarantees KV cache locality.KVAwareRouter
, depending on its scoring coefficients, may trade some cache overlap for better token-load balance and therefore requires tuning.Cross-session KV cache reuse is limited: If conversations diverge after a common system prompt, there is little additional cache reuse forKVAwareRouter
to discover across sessions.Sessions have similar workloads: When sessions do not vary significantly in input/output lengths or execution time, consistent hashing already provides reasonable load balance at the session level.
LinkCase Study: Replaying Claude Code Traces #
Coding agents are one of the most popular LLM inference workloads today, so let’s take some inspiration from real Claude Code traces. We evaluate different routing strategies on a filtered subset of the Weka Claude Code trace corpus that fits within the context window of
gpt-oss-120b
.First, let’s take a look at the workload shape. Two things stand out: sessions are highly heterogeneous, and there are significant opportunities for KV cache reuse within each session. Individual requests already vary widely in input and output lengths, but the differences become even larger at the session level. Some sessions have only a single turn, while others span dozens of turns and accumulate millions of input tokens.
This sets up an interesting routing tradeoff: do we maximize KV cache reuse by keeping a session on the same replica, or do we spread the token load more evenly across replicas?
Ray Serve LLM’s KVAwareRouter
considers both KV cache overlap and token load when scoring replicas. As shown in the results, it achieves better TTFT, TPOT, and throughput than session affinity with consistent hashing.
To understand why, let’s look at the active decode-block coefficient of variation (CV) and prefix cache hit rate*. Session affinity with consistent hashing preserves KV cache reuse well because requests from the same session stay on the same replica. But it only balances at the session level. When one session can be much larger or longer than another, evenly distributing sessions does not mean evenly distributing token load.
KVAwareRouter
instead trades some prefix cache hit rate for better token load balance as concurrency increases. The result shows that for heterogeneous LLM workloads, **balancing KV cache reuse with token load **leads to better overall serving performance than maximizing KV cache reuse alone.
** Note: Prefix-cache hit rate is the token-weighted ratio of vLLM-reported cached prompt tokens to total prompt tokens, computed over requests with both usage fields available. Mean active decode-block CV is an offline reconstruction of per-replica decoding KV-block imbalance: in 0.5-second bins, each request contributes an estimated active KV-block footprint from first token until completion; lower CV means more balanced load.*
LinkFuture Work #
Revisiting the routing challenges outlined at the beginning of this blog post, KVAwareRouter addresses stateful execution and workload heterogeneity, while non-deterministic generation remains an open problem. Because output lengths are unknown at admission time, requests initially expected to be lightweight may end up with long decode phases and collide on the same replica, creating unexpected token load imbalance. Accurately accounting for this uncertainty remains an open challenge across the industry. One approach is for agentic harnesses to provide hints about expected generation behavior.
In the near future, we plan to extend KVAwareRouter
support to prefill-decode disaggregated deployments, data-parallel deployments, and multimodal workloads. Routing is still an evolving problem: choosing where and at what level to route requires a deep understanding of the workload and deployment architecture. Stay tuned for more updates!
LinkConclusion #
The optimal routing policy depends on the workload, and Ray Serve LLM provides configurable routing policies that you can plug and play. Session affinity with consistent hashing provides strong KV cache reuse for multi-turn conversations while balancing sessions across replicas. KVAwareRouter
goes a step further by balancing both KV cache overlap and token load, enabling finer-grained load balancing and improving TTFT, TPOT and throughput for heterogeneous workloads.
The key takeaway is that KV cache overlap should not be the sole routing objective. Token load, which captures both remaining prefill work and ongoing decode load, better reflects the actual work on each engine. Balancing this work, while still taking advantage of KV cache reuse, leads to more efficient LLM serving.
LinkAcknowledgment #
Special thanks to the NVIDIA Dynamo team for modularizing their KV indexer and request scoring modules, making them available for integration with Ray Serve LLM.
LinkReproduction Notes #
Benchmark code.
Table of contents
- Challenges of LLM Request Routing
- Naive Optimization: Maximize KV Cache Reuse
- Session Affinity
- Prefix Affinity
- KV Cache Affinity
- Why Isn’t Maximizing KV Cache Reuse Alone Enough?
- Request Herding
- Load Balancing
- Beyond KV Cache Reuse: Token Load Awareness Matters More
- Case Study: Replaying Claude Code Traces
- Future Work
- Conclusion
- Acknowledgment
- Reproduction Notes