Amazon SageMaker AI real-time endpoints now support KV cache aware routing Amazon SageMaker AI added KV cache-aware routing for real-time endpoints, a routing strategy that uses cache state reported by the model container to send each request to an instance holding a long cached match for its prompt that isn't too busy. The feature is enabled by adding an environment variable so the container reports cache contents and setting RoutingStrategy to KV_AWARE in the endpoint configuration, while clients keep calling the endpoint with the same APIs. Amazon says the approach addresses two limits of its earlier prefix-aware routing: instances evicting KV cache entries the router still assumes are warm, and short multi-turn conversations landing on different instances before requests grow past the configured PrefixLength. Artificial Intelligence https://aws.amazon.com/blogs/machine-learning/ Amazon SageMaker AI real-time endpoints now support KV cache aware routing Today, Amazon SageMaker AI introduces KV cache-aware routing KV-aware routing , a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models LLMs with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set RoutingStrategy to KV AWARE in the endpoint configuration. Your clients call the endpoint with the same APIs as before. Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation RAG , and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In part 1 of this series https://aws.amazon.com/blogs/machine-learning/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference/ , we introduced prefix-aware routing for Amazon SageMaker AI, which sends requests that start the same way to the same instance. Prefix-aware routing infers where a prefix is cached from the request alone, so it can’t tell when an instance has evicted that cache. Prefix-aware routing picks an instance from the request alone. It hashes the start of the request, up to the PrefixLength you configure bytes of the InvokeEndpoint payload, or characters of the message text for the OpenAI-compatible Chat Completions API , and sends requests that start the same way to the same instance. It never sees what an instance actually has cached. This works well when every request starts with the same shared text, and less well in two cases: The cache on an instance changes over time. An instance evicts older KV cache entries when its GPU memory fills up, but the router keeps sending matching requests there as if the cache were still warm. A conversation starts out short. In a multi-turn chat, each turn adds to the history. While the request is still shorter than PrefixLength , every new turn changes the text the router hashes, so the next turn can land on a different instance than the one that cached the earlier turns. The turns of a conversation stay on one instance only after the request grows past PrefixLength . In this post, we explain how KV-aware routing works, when to use it, and how to enable, invoke, and verify it on single model endpoints and inference component endpoints. How KV-aware routing works KV-aware routing has three parts: the model container reports what it has cached, SageMaker AI keeps a managed cache index of that state for the endpoint, and the endpoint uses the index to choose an instance for each request. When you turn events on with SM VLLM KV EVENTS CONFIG , the container publishes a KV cache event each time it stores or evicts cache entries, and SageMaker AI uses these events to keep the index current. The index holds only pointers: one-way, truncated hashes of cached content, each mapped to the instance or inference component copy that holds it. It never stores prompts, token IDs, request payloads, or KV tensors. When a request arrives, the endpoint tokenizes the prompt and looks up the tokens in the index. The index returns the instances or inference component copies that hold part of the prompt in their cache, and how much of it each one holds. For example, if instance A holds the cached start of a conversation, the next turn of that conversation goes to A. Because the index matches on tokens rather than a fixed number of bytes, a multi-turn history can match in full as it grows. A request that shares only part of its prompt still goes to the instance with the longest match. The endpoint then picks among these candidates by weighing how much of the prompt each one has cached against how busy it is. A candidate that is too busy is skipped, so if the instance with the longest match is busy, the request goes to the next-best one that is available. If no candidate has a useful match or all of them are busy, the request falls back to load-aware routing, as described in Routing overhead and fallbacks. You don’t configure a concurrency threshold for this, as you do with prefix-aware routing. The index also follows the fleet. When you scale in, entries for removed instances are dropped. When you scale out or an update replaces instances, the new instances start with empty caches, so latency can be higher for a short time. In Figure 1, the endpoint has three instances, A, B, and C. Each instance runs the same model in its own model container the AWS vLLM Deep Learning Container listed in Requirements and limits and has its own KV cache in GPU memory, so each instance holds different cached content: A holds turns 1 to 3 of conversation 1, B holds conversations 2 and 3, and C holds document 7. When turn 4 of conversation 1 arrives step 1 , the router tokenizes it and looks it up in the managed cache index step 2 . The index shows that A holds the first 6,000 of the request’s 6,200 tokens, so the router sends the request to A step 3 , and A only has to compute the remaining tokens. If A is busy, the router sends the request to the least busy instance instead step 4 . KV-aware routing needs a choice of destinations, so it is active only with at least two instances, or at least two copies of an inference component. With a single instance, every request goes to that instance and the setting has no effect. Routing strategies on SageMaker AI With this launch, SageMaker AI offers four routing strategies for real-time endpoints. You set the strategy per production variant in the RoutingConfig of your endpoint configuration. | Strategy | How it routes | Recommended for | | RANDOM default | Sends each request to a randomly chosen instance. | General-purpose workloads, non-LLM models, and requests that are interchangeable. | | LEAST OUTSTANDING REQUESTS | Sends each request to the instance with the fewest in-flight requests. | Workloads where processing time varies and you want to keep all instances equally busy. | | PREFIX AWARE | Sends requests that share the same prompt prefix, up to PrefixLength, to the same instance. Overflows to another instance at ConcurrencyThreshold. | LLM workloads with shared text at a fixed position at the start of the request, on any serving framework that supports prefix caching. | | KV AWARE new | Sends each request to an instance that holds a matching cached prefix, based on cache state reported by the container, weighing match length against load. Falls back to load-aware routing. | LLM workloads on a supported container with multi-turn conversations, frequent cache eviction, or requests that share only part of their start. | Prefix-aware or KV-aware Both strategies send requests to where their prefix is likely to be cached. They differ in how they know where that is, and in what they need from your container. | | Prefix-aware routing | KV-aware routing | | Source of cache location | Inferred from the request prefix | Cache state reported by the container | | Added latency per request | 1.3–1.9 milliseconds in part 1 | A few milliseconds for short prompts, more for long ones, mostly for tokenization | | Matching | First PrefixLength bytes Invoke API or characters OpenAI-compatible API | Tokens, longest cached match | | Parameters | PrefixLength, ConcurrencyThreshold | None needed | | Container | Any container with prefix caching on | A supported container, with prefix caching and KV cache events on see Requirements and limits | | Request payload | Any | OpenAI-compatible Completions or Chat Completions format | Performance We measured KV-aware routing against random routing in two ways: a large-scale benchmark on production-sized instances, and a smaller, reproducible comparison you can follow in Verify that KV-aware routing is working. Benchmark results In our benchmarks with Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances, we ran 30-minute tests with 20 or 100 simulated users holding multi-turn chats and document Q&A sessions, with prompts of up to about 64,000 tokens. Compared with random routing, and with long prompts under load, KV-aware routing reduced p50 time to first token TTFT by up to 97 percent and increased throughput by up to 234 percent. It raised the KV cache hit rate from 3–48 percent to 80–93 percent. Long prompts gained even at lower load. The gain was smallest with short prompts, where there is less prefill to save. Results depend on your workload. To measure them for your own traffic, use the checks in Verify that KV-aware routing is working. Routing overhead and fallbacks Before it routes a request, the endpoint tokenizes the prompt and looks it up in the managed cache index. The lookup takes a few milliseconds for short prompts and longer for long ones, mostly for tokenization, but it stays small next to the time a cache hit saves. For text-only requests that use standard options, the tokens it computes are passed to the model, so the prompt isn’t tokenized twice. If the lookup is slow, finds no useful match, or all matching instances are busy, the endpoint routes the request with load-aware routing, the same behavior as the LEAST OUTSTANDING REQUESTS strategy. The request is still served, without the cache benefit. The same happens when the endpoint can’t tokenize a request, except in the two error cases described in Invoking the endpoint. These fallbacks aren’t reported in the SageMaker AI console or metrics at launch, so use the checks in Verify that KV-aware routing is working to confirm that routing is effective for your traffic. With very short prompts there is less prefill to save, so prefix-aware routing can be all you need. When to use KV-aware routing Prefix-aware routing needs no changes to your container, but it can’t see what an instance actually holds. KV-aware routing asks the container what it holds, which makes it more precise. It pays off when there is a lot of cached context to reuse and that cache is under pressure: long prompts, high load, and instances that run short of GPU memory for the KV cache. When that happens, the container evicts the least recently used parts of the cache, including parts of shared prompts and conversation history. For example, a support assistant runs on three instances, and each conversation begins with the same 6,000-token knowledge-base excerpt. Conversation 1 starts on instance A, which caches the excerpt. Traffic rises, A runs short of memory, and the container evicts most of that excerpt. Instance B recently served another conversation over the same excerpt and still holds its first 4,000 tokens. When the next turn of conversation 1 arrives, prefix-aware routing sends it to A again, and A recomputes the excerpt. With KV-aware routing, A’s container has already reported the eviction, so the index shows that B holds the longest match, and the request goes to B, which recomputes only the remaining tokens. At low load, A wouldn’t have evicted anything, and both strategies would have sent the request to A with the cache intact. KV-aware routing helps most when requests reuse cached context in ways a fixed-length prefix can’t capture: Multi-turn conversations: Each turn sends the full conversation history, so the cached prefix grows with every turn. KV-aware routing matches the whole cached history and keeps each conversation on the instance that holds it. This matters most for long conversations. After the managed cache index reflects an eviction, the router doesn’t keep sending the conversation to an instance that has already dropped its cache. High-throughput inference with shared context: RAG applications prepend retrieved documents, and assistants send long system prompts with every request. KV-aware routing fits when these prompts are long and instances are under memory pressure. With a stable shared prefix and moderate load, prefix-aware routing also fits. Put the shared content first and in the same order: if the same documents appear in a different order, the prompts no longer share a start, so they don’t match. Workloads where prefix length is hard to tune: With prefix-aware routing, you choose a PrefixLength that covers your shared text without scattering requests that should stay together. KV-aware routing has no prefix length to tune, because it matches on what is actually cached. Prefix-aware routing is a good fit, with no container changes, if your requests share a stable system prompt or document at the start, your prompts are short, or load is moderate. In these cases, instances have enough GPU memory to keep the shared text cached, so both strategies find it. Prefix-aware routing is also the choice if your container isn’t supported by KV-aware routing, because it works with any container that has prefix caching turned on. If your requests are short and interchangeable, or your model isn’t an LLM, there is little cached context to reuse, so use RANDOM or LEAST OUTSTANDING REQUESTS . If you aren’t sure which fits, start with prefix-aware routing, run your own traffic against a KV AWARE endpoint, and compare time to first token, throughput, and hit rate as described in Verify that KV-aware routing is working. Requirements and limits Before you enable KV-aware routing, check that your container, endpoint, and requests meet these requirements: Container: At launch, KV-aware routing supports the AWS vLLM Deep Learning Container for SageMaker AI: images with vLLM 0.33.0 or later, or the Amazon Linux build server-sagemaker-cuda-v2.8.0 or later. Prefix caching must be on, which is the default, and KV cache events must be turned on with the SM VLLM KV EVENTS CONFIG environment variable, as shown in How to enable it. SageMaker AI doesn’t set it for you. Endpoints and instances: Real-time single model