{"slug": "the-kv-cache-is-just-session-state-disaggregated-inference-for-backend-engineers", "title": "The KV Cache Is Just Session State: Disaggregated Inference for Backend Engineers", "summary": "A vLLM blog post dated 12 October 2026 describes bi-directional KV cache transfer for multi-turn and agentic workloads, in which a decode node retains a conversation's KV cache and a router maps a conversation_id to that node's transfer metadata so the prefill node pulls the existing state back and computes only new tokens. The post builds on vLLM's 29 September 2026 disaggregated serving guide, which reports Llama-3.1-70B in BF16 at 320 KiB of KV cache per token and measured, on two L40S GPUs with 8k-token prompts, colocated p99 inter-token latency jumping from 23 ms to 169 ms at 0.4 requests per second while disaggregated stayed between 25 and 52 ms, though median time to first token was worse at 2.2 s versus 0.7 s at 0.2 requests per second because a roughly 1.3 s transfer landed on every first token. The guide advises that colocated serving remains right for low, bursty or TTFT-bound traffic, and DistServe lists fault propagation and a weak fit for resource-constrained setups as limits.", "body_md": "*Disclosure: I drafted parts of this article with AI writing tools. I edited all of it, checked the claims against the sources linked below, and ran the code myself.*\n\nIf you run backend systems, you already know most of what you need to reason about LLM serving. Sticky sessions, session stores, hot keys and moving state between nodes all carry over; only the vocabulary is new.\n\nThis came up after I read a vLLM post from 12 October 2026 about routing by conversation ID. Its numbers are easy to over-read, so I go through them below, after the mental model.\n\nServing one LLM request has two phases. Prefill is where the model reads your whole prompt. All the input tokens can be processed in parallel, so this phase is compute-bound, and how long it takes sets the time to first token (TTFT), the pause before the user sees anything.\n\nDecode is where the model writes the answer one token at a time. Each new token needs the results already computed for every earlier token, so the work is dominated by reading memory rather than by arithmetic. It is memory-bandwidth-bound, and it sets the gap between tokens (ITL).\n\nThe vLLM disaggregated serving guide (29 September 2026) says the same. When both phases share one GPU they interfere, and a long prefill stalls everyone’s decode. It’s the noisy-neighbour problem inside one box.\n\nThose “results already computed for every earlier token” are the KV cache. The useful reading for a backend engineer is that it’s per-conversation state that makes the next request cheap. Three properties matter.\n\nIt costs memory, and a lot of it. The same vLLM guide gives Llama-3.1–70B in BF16 at 320 KiB of KV cache per token, so a 10,000-token prompt is about 3 GB. A hundred such conversations held at once is on the order of 300 GB (my arithmetic).\n\nA miss is expensive. If the cache isn’t on the node that receives turn five, that node redoes the prefill for the entire history, so the user pays in TTFT and you pay in GPU time.\n\nRouting therefore matters. Send turn five to the node holding the state and it only processes the new message; send it elsewhere and it reprocesses everything. That’s the reason you used sticky sessions in the first place, and chat makes it worse because, as the Splitwise paper (arXiv 2311.18677) notes, a chat client resends the whole conversation each turn.\n\nDisaggregated inference splits the phases onto separate pools. Prefill nodes read prompts, decode nodes hold the state and stream tokens, and the KV cache travels between them. From the papers and the vLLM guide, the gains are these:\n\nThese come from specific setups, and none belongs in a business case for your workload. The costs matter just as much.\n\nYou run three or four services instead of one, and KV transfer becomes a new failure mode. The transfer can also eat the gain. On two L40S GPUs with 8k-token prompts, the guide measured colocated p99 ITL jumping from 23 ms to 169 ms at 0.4 requests per second, while disaggregated stayed between 25 and 52 ms. Median TTFT was worse, though, at 2.2 s against 0.7 s at 0.2 requests per second, because a slow transfer of about 1.3 s (no peer-to-peer or RDMA on that box) landed on every first token. DistServe lists fault propagation and a weak fit for resource-constrained setups as limits. The guide’s advice is that colocated stays right for low, bursty or TTFT-bound traffic.\n\nThe post is about “bi-directional KV transfer” for multi-turn and agentic workloads (https://vllm.ai/blog/2026-10-12-bidirectional-kvxfer-multiturn-agentic-workload). In a plain disaggregated setup, KV moves one way, from prefill to decode. Here the decode node keeps the conversation’s KV for a while, and on the next turn a router sends the request to the prefill node with a pointer to that state. The prefill node pulls the existing KV back and computes only the new tokens.\n\nThe router is a small stateful service that maps a conversation_id to the decode node's transfer metadata, while the tensors move node to node. Two defaults are worth knowing. The decode node holds state for 480 seconds (fixed, not renewed per turn), and below a 64-token threshold the system recomputes. The post doesn't say it in one sentence, but I infer that without a conversation_id every turn is a cache miss and gets recomputed.\n\nThe headline is average TTFT up to 1.6x lower for Qwen3–32B and up to 3.3x lower for GLM-5-FP8. The details matter here.\n\nMy reading: on a fast interconnect, reusing session state beats recomputing it, and the benefit grows with prompt length and model size. It doesn’t say a router makes everything 3x faster, and the figures compare reuse against full recompute on H200 nodes with EFA, so they aren’t a promise for your stack.\n\nThe obvious backend answer is to hash the conversation ID onto a ring. You get affinity without a lookup table, and it survives adding nodes better than modulo does, but it falls short here. One reason is visible in Ray Serve’s LLM docs, and the others are my own reasoning.\n\nRay ships a ConsistentHashRouter that hashes an x-session-id header, and its docs state the cost plainly: a session with more turns or longer sequences can saturate its replica. From here on it's my reasoning, not the docs. The hash key is the conversation, but the cost of a conversation isn't uniform. A request's price grows with its context length, so one long-running agent can be worth a hundred short chats, and the ring can't see that.\n\nThe same docs describe an alpha KVAwareRouter that routes on token load, meaning the prefill work remaining after cache overlap plus the work in flight on each replica. It learns where state lives from vLLM's KV events. The caveats are that it has no prefill-decode disaggregation support yet, and its overhead may not pay off on uniform workloads.\n\nIn my view a good router needs three things a hash can’t give: which node holds the state, how loaded it is, and what a miss would cost. The conversation_id table in the vLLM post is the simplest version of the first.\n\nTo see the tension between affinity and balance I wrote a toy model (router_sim.py). It isn't a benchmark of vLLM, Ray or anything else, and it has no queueing, no decode and no network. It generates multi-turn conversations with a heavy-tailed number of turns, gives each of 8 replicas an LRU cache of conversation prefixes, and counts prefill tokens computed. Results are averages over 20 seeds.\n\nWith the default setup (2,000 conversations, 300,000-token cache per replica):\n\nBalance looks fine at that scale, with the busiest replica’s prefill work (tokens computed, not request count) at 1.16 times the mean for hashing. It changes when conditions tighten. With only 200 conversations the busiest replica under hashing was 1.37 times the mean (1.30 for the pinned table). With 2,000 conversations but a small cache per replica, hashing reached 2.10 times the mean, the pinned table 1.56, and the load-aware scorer 1.03, at the price of a lower hit rate (56.9% against 65.1% for hashing and 69.5% for the table).\n\nDirectional only: affinity gives most of the win, hashing leaves some replicas hotter, and balancing harder costs cache hits.\n\nIn-house models for KYC review, disputes or support are multi-turn, with long documents and policy text up front, which is where reusing state matters most. If your prompts are short and one-shot, little of this applies.\n\nA KV cache holds a derivative of the customer’s conversation in GPU memory for as long as your TTL says, so your retention and residency policies should name it. Take it to security and compliance, and ask any vendor the same.\n\nAWS EC2 on-demand, Linux, shared tenancy, Asia Pacific (Mumbai), p5en.48xlarge costs $75.9552 per hour per instance, from the AWS price list file for ap-south-1 (publication date 12 October 2026, [https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/ap-south-1/index.json).](<https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/ap-south-1/index.json).>) A setup like the one in the vLLM post needs two (plus a small node for the proxy), so $151.9104 per hour for the GPU pair before storage, networking, redundancy and people.\n\nThe Anthropic API price for Sonnet 5.5 is $2 per million input tokens and $10 per million output tokens, standard tier, from [https://www.anthropic.com/pricing,](https://www.anthropic.com/pricing,) checked 15 October 2026.\n\nArithmetic on those two: $151.91 an hour buys about 76 million input tokens or about 15 million output tokens from that API. That’s a reference point, not a verdict. The models and their quality differ, and a self-hosted pair has to stay busy and redundant to compete. An idle GPU costs what a busy one does.\n\nMy rule of thumb for the decision:\n\n*Follow me for more on system design and the trade-offs behind it.*\n\n[The KV Cache Is Just Session State: Disaggregated Inference for Backend Engineers](https://pub.towardsai.net/the-kv-cache-is-just-session-state-disaggregated-inference-for-backend-engineers-1c9303939865) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/the-kv-cache-is-just-session-state-disaggregated-inference-for-backend-engineers", "canonical_source": "https://pub.towardsai.net/the-kv-cache-is-just-session-state-disaggregated-inference-for-backend-engineers-1c9303939865?source=rss----98111c9905da---4", "published_at": "2026-10-19 21:01:01+00:00", "updated_at": "2026-10-19 21:15:12.679896+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-agents"], "entities": ["vLLM", "Llama-3.1-70B", "DistServe", "Splitwise", "L40S"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-kv-cache-is-just-session-state-disaggregated-inference-for-backend-engineers", "markdown": "https://wpnews.pro/news/the-kv-cache-is-just-session-state-disaggregated-inference-for-backend-engineers.md", "text": "https://wpnews.pro/news/the-kv-cache-is-just-session-state-disaggregated-inference-for-backend-engineers.txt", "jsonld": "https://wpnews.pro/news/the-kv-cache-is-just-session-state-disaggregated-inference-for-backend-engineers.jsonld"}}