Route to Where the KV Cache Is, Not Where It Was

Ranvier Systems introduced a load-balancing technique for LLM serving that routes requests based on where the KV cache currently resides rather than historical prefix matches, reducing P99 time-to-first-token by 57% under load. The system uses gossiped cache residency weights to avoid routing to backends that have evicted the needed cache blocks, preventing costly recomputation for tail-latency requests.

Route to Where the KV Cache Is, Not Where It Was We’ve argued here before that a load balancer which ignores the KV cache leaves GPU throughput on the floor, first in Why Your Load Balancer Is Wasting Your GPUs https://ranvier.systems/2026/03/16/why-your-load-balancer-is-wasting-your-gpus.html and then in KV Cache Locality https://ranvier.systems/2026/04/30/kv-cache-locality-the-hidden-variable-in-your-llm-serving-cost.html . The short version: Ranvier tokenizes each prompt, looks the prefix up in an adaptive radix tree ART that maps prefixes to backends, and sends the request to the GPU most likely to already hold those KV blocks. Reuse the cache, skip the recompute. This post is about what happens next, when the backend that had your prefix doesn’t have it anymore. Routing by where a prefix still resides, rather than where it was last served, cut our P99 time-to-first-token by 57% under load. The median didn’t budge, and that turns out to be the point. A Cache Hit in the Router Can Be a Miss at the Engine KV cache is finite, and backends evict under pressure. The ART says “backend 7 served prefix X.” But that was ten minutes and ten thousand requests ago, and 7 has long since evicted X. You route to 7 expecting a hit; the engine takes a miss and recomputes the whole prefix. The request that should have been the fastest is now one of the slowest. The routing table remembers history. The cache only knows the present. The busier the system gets, the faster the two drift apart. That’s a tail problem. Median requests are fine. The unlucky ones that bet on an evicted prefix pay full recompute and collect in your P99. We flagged exactly this case in the learning-router post https://ranvier.systems/2026/06/10/what-happens-when-your-llm-load-balancer-learns.html : a learned route is a prediction, and the backend’s own memory pressure decides whether it still holds. This post closes that gap. Track Residency, Not Just History Ranvier nodes already gossip routing state to each other it runs as a cluster of peer nodes . We extended that gossip with a small amount of cache state. Each node now shares its cache utilization and a per-prefix residency weight : an estimate of how likely a prefix is still resident on a given backend, derived from utilization and recency. When the router finds a prefix match, it discounts the route by the residency weight. A match on a backend whose cache is cold or thrashing gets downgraded, and the router prefers a backend where the prefix is still likely to be, or routes for load instead of betting on a stale hit. A few notes for the systems-minded: The decision is inline, in C++. The ART lookup, the residency weighting, and the load comparison fold into a single weighted score that runs in microseconds, in-process, on the Seastar data plane. The residency check is a local lookup. There is no per-request hop to a separate routing service. This signal is an estimate by default, and on purpose. The gossiped residency weight is a probabilistic retention model built from utilization and recency, not an exact account of the engine’s cache. It’s cheap, engine-agnostic, and enough to catch the stale-route case that owns the tail. On vLLM you can tighten it: Ranvier can subscribe to the engine’s native KV-cache event stream and route on verified block-level residency, where present means resident and absent means evicted, with no guessing. It falls back to the probabilistic signal for engines that don’t emit those events. More on the verified path below. The wire format won’t need a flag day. The cache-state packet tolerates trailing bytes, so new fields can ride along later, and an unknown-packet guard keeps mixed-version clusters safe through rolling upgrades. You can watch it work. A counter, ranvier router residency route downgrades total , records how often residency weighting actually changed a routing decision. That counter is the whole story of the benchmark below. It mostly doesn’t care which engine you run. The base routing is on raw token prefixes, with no engine cooperation required, so it works across vLLM, SGLang, TensorRT-LLM, Ollama, and LM Studio. The verified-residency path above is the one place it leans on a vLLM-specific feature, degrading to the estimate everywhere else. The gossip-and-estimate mechanism shipped in ranvier-core 527 https://github.com/Ranvier-Systems/ranvier-core/pull/527 ; verified residency from vLLM’s native events landed in 549 https://github.com/Ranvier-Systems/ranvier-core/pull/549 . The Benchmark Residency weighting on vs. off, with everything else held identical. The setup: - 8x A100-40GB, Llama 3.1 8B, vLLM 0.15.1 - 60 concurrent users, 400-token outputs, target GPU utilization ~0.50 - A churn workload: prefixes rotate enough that backends continuously evict and refill, the regime where history and residency drift apart - 30 minutes per leg, identical traffic to both; errors on either leg would invalidate the run | Metric | OFF | ON | Δ | |---|---|---|---| | TTFT P99 | 2800 ms | 1200 ms | −57.1% | | TTFT P50 | baseline | unchanged | ~0 | | Throughput | baseline | +1.2% | +1.2% | | Errors | 0 | 0 | — | | Residency downgrades | 0 | 39 0.2% of requests | — | The honest read. This is a tail win, not a median win. P50 didn’t move and throughput barely did, which is what the mechanism predicts. Residency weighting does nothing for the typical request that was going to hit a warm cache anyway. It catches the request that was about to bet on an evicted prefix and eat a full recompute, and those requests are what live in your P99. The number that tells the story is 0.2%. Only 39 requests on the ON leg were rerouted by residency weighting, and those were the ones headed for a cache miss. Rerouting them more than halved the P99. The low count matters: if residency weighting were firing on 20% of traffic, the model would be over-discounting and stealing warm hits from requests that were fine. A fraction of a percent of reroutes with the tail cut in half is the shape you want. Zero errors on both legs, and a little throughput back from the recompute work avoided. The harness and full results are in ranvier-core 546 https://github.com/Ranvier-Systems/ranvier-core/pull/546 . What’s Next: Verified Residency Beyond vLLM The benchmark above runs on the probabilistic estimate, and the estimate is enough to own the tail. On vLLM, Ranvier already goes further: it subscribes to the engine’s native KV-cache event stream and keeps a block-exact mirror of what’s resident, so a route is confirmed or discounted against ground truth instead of a guess. Those same events also create routes. A freshly stored prefix is inserted at its block boundaries, so cache computed by engine warm-up or another frontend becomes routable before Ranvier has sent a single request there. What’s left is reach and proof. The verified path rides vLLM’s event stream today, and every other engine falls back to the estimate, so we want verified residency anywhere an engine will emit the events for it. And the numbers above are the estimate’s; the verified path should sharpen them and push the win into lower-churn regimes, where the estimate has less to catch. That A/B is the next benchmark to publish. Try It Ranvier is a single binary: no Kubernetes, no sidecars, no separate control plane. The code, including everything described above, is open source at github.com/Ranvier-Systems/ranvier-core https://github.com/Ranvier-Systems/ranvier-core . Ranvier is a project of Minds Aspire, LLC.