{"slug": "what-are-kv-caches-really", "title": "What are KV caches really?", "summary": "KV caching is a core mechanism inside transformer inference that splits the process into two stages: prefill, which runs the user's prefix through the model and computes keys and values for each token, and decode, which loops token-by-token using all prior keys and values to predict the next output. The prefill stage determines time to first token, while the decode loop determines time between tokens (TBT), and keys and values are unique to each token and never shared between tokens.", "body_md": "### What are KV caches really?\n\nKV caching lies sufficiently deep within the guts of how [transformers](https://www.glennklockwood.com/garden/transformer)\nwork that really understanding how it can be used goes beyond any simple\nanalogy. This is why the Internet is full of explanations of KV caching\nthat are either grossly oversimplified (and therefore imply that KV\ncaching is more useful than it really is) or dive straight into\nmathematical formalism (and therefore are incomprehensible to dumb-dumbs\nlike me).\n\nSo I will try to thread that needle and explain how KV caching works in terms of how dumb-dumbs like me actually experience them.\n\n## What are keys, values, and KV caches?\n\nKeys, values, and KV caches only make sense if you first understand what's happening when you ask a chatbot a question. There are fundamentally two steps to inferencing: prefill and decode. And anyone who's used ChatGPT or Claude has seen them firsthand.\n\n### What is prefill?\n\nThis is [prefill](https://www.glennklockwood.com/garden/prefill):\n\nWhen you ask a chatbot a question, there will be a lag before output\nstarts being spit out. This lag the result of the *prefill* step\nrunning your query through the LLM. There are a bunch of terms that are\nrelated to this:\n\n- **Tokens** are what LLMs ingest and spit out. They are\napproximately equivalent to a word in English.\n- **Time to first token** is the lag time between you\nsubmitting your query and the first bits of output being generated. That\nis, it is the time it takes to perform prefill.\n- **Prefix** is your initial query (\"What are the key\ntakeaways...\"). It is usually combined with bunch of additional text\n(which you don't typically see) that gets sent to the LLM when you first\nask your question.\n- **Prefill** is the process of running the prefix\nthrough the model to get it to the point where it can begin predicting\nthe next token and generating output.\n- **Keys** and**values** are what the\nprefill step is calculating in silence. \"KV\" is shorthand for \"key\" and\n\"value,\" and keys/values are parts of the model's internal machinery.\nLLMs generate multiple keys and values for each token they ingest, but\nkeys and values are unique to each token. Different tokens never share\nthe same keys or values.\n\nThe *time to first token* is directly related to how much work\na GPU has to do to *prefill* the *prefix*. The longer your\nquestion, or the more context that has to be included in it (e.g., if it\nis pulling in context from the Internet or uploaded documents, or if it\nhas a lengthy *system prompt*), the longer prefill will take.\n\n### What is decode?\n\nThis is [decode](https://www.glennklockwood.com/garden/decode):\n\nDecode is what is happening as the chatbot is spitting out its response to your question. If you look closely at decode as it happens, you'll notice that it generates output word-by-word (or token-by-token); this is because decode is a loop where:\n\n1. The model starts with the prefilled prefix and all the calculated keys and values.\n2. The model uses ALL the keys and values to predict a good next token. That next token is what you see as the first bit of output.\n3. The model then appends that first output token to the prefix.\n4. The model then calculates the keys and values for the output token from step #3.\n5. The model loops back to step #2.\n\nAnother way of visualizing this decode loop is this:\n\nEach iteration of the loop generates one additional output token using all previous tokens' keys and values. It repeats this until the model predicts that there is no next token.\n\nThe time to generate the next output token is called time between tokens (TBT).\n\n### What makes keys and values cacheable?\n\nEvery time an LLM spits out a next word or token, it is using every previous token in that conversation to predict the best next word; in the above examples,\n\n- The first output token (`Good` ) is generated based on the\nprefix's keys and values\n- The second output token (`question` ) is generated using\nthe keys and values belonging to the prefix*and*`Good`\n- The third output token (`to` ) is generated using the keys\nand values belonging to the prefix,`Good` , and`question`\n\nEqually important is the fact that each output token does NOT depend on any future tokens that haven't yet been generated. As a result, keys and values have two properties that make them ideal for caching after they've been calculated:\n\n1. Keys and values for each output token never change after they're generated\n2. Keys and values for each output token are used every time a subsequent output token is generated\n\nThis is why [KV caches](https://www.glennklockwood.com/garden/kv-cache)\nexist. They store the computed keys and values of every token in the\nprefix and every output token, and these keys and values are repeatedly\nre-loaded into GPU cores' registers as output tokens are generated\nduring the decode process. And as each new output token is generated,\nits keys and values get calculated as well, and they are appended to the\nKV cache so they can be used to generate future tokens.\n\nKV cache is the reason why prefill time (time to first token; TTFT) is so much longer than the lag between each iteration of the decode loop (time between tokens; TBT):\n\n- Prefill has to compute keys and values for every token in the entire prefix\n- Each iteration of decode only needs to compute keys and values for a single token (the other keys and values can just be loaded from KV cache).\n\nKV cache is also the reason why prefill is compute-bound (benefitting from more GPU FLOPS) while decode is memory bandwidth-bound (benefitting from more HBM TB/s); prefill requires computing keys and values for a lot of tokens in the prefix, while decode requires computing keys and values for one token. The rest of decode's time is spent reading cached KVs from memory.\n\n## EVERYBODY uses KV caches\n\nKV caches provide such a huge speedup for decode that, in reality, everybody already uses KV caches. They are essential during decode.\n\nWithout a KV cache, you'd experience the prefill lag every time the LLM spits out a single next token. What's worse, that prefill time would get longer and longer, because the amount of tokens that would have to be prefilled would be getting longer as output tokens are appended to the prefix. The scaling is terrible; if calculating keys and values for one token takes a second...\n\n- after the second token, recomputing all keys and values would take 4 seconds\n- after the third token, recomputing would take 9 seconds\n- after the fourth token, recomputing would take 16 seconds\n\nCarrying out just a few back-and-forths with a chatbot would be excruciatingly slow without KV caches. A chatbot without a KV cache would just grind to a halt before it could finish outputting a single response.\n\n## Slow KV caches are only useful for offload\n\n**For a KV cache to be useful, it has to be faster than\nrecomputing keys and values.** The time it takes to recompute\nkeys and values must be longer than the time it takes to read keys and\nvalues from cache. Recalling that the cost of recomputing the entire\nprefix depends on the length of the prefix, this means that KV caches\nare more useful as prefixes get really long.\n\nThis has a few implications:\n\n1. **The KV cache used during decode is always stored in GPU\nHBM** . Going off-GPU every time a new token is being generated\nmakes using a GPU pointless for decode. This is why decode performance\ndepends on[memory\nbandwidth](https://www.glennklockwood.com/garden/memory-bandwidth) more than FLOPS.\n2. **Slow KV caches are only useful for prefill, not during\ndecode** , and only when the amount of prefix that must be\nprefilled is long. The longer the prefix, the slower the KV cache can be\nand still be faster than recomputing all keys and values.\n\nIn practice, this means a slow KV cache is only useful when a new query is sent to an existing back-and-forth conversation. But even then, there are limitations:\n\n**Prefixes must be identical in order for their keys and values\nto be useful.** Because each output token depends on all the\ntokens that came before it, any differences in a prefix makes everything\nthat follows the change useless. This is fine for a single chat session\nthat keeps extending itself, but it makes re-using KV caches across\ndifferent prompts impossible for many common scenarios. For example,\nconsider two people who ask the same chatbot the following:\n\n- User 1:\n`What are the key takeaways I want the audience to have...`\n- User 2:\n`Tell me the key takeaways I want the audience to have...`\n\nEven though most of the question is identical\n(`the key takeaways...`), the fact that the first letter of\nthe prompt is different means their KV caches will be completely useless\nto each other. You cannot cache just the common parts\n(`the key takeaways ...`) because their keys and values were\ngenerated based on the tokens that came before them in the prompt, and\nthose tokens were not the same (`What are` vs.\n`Tell me`).\n\nThere are [techniques that\nrelax this constraint](https://arxiv.org/abs/2405.16444). However, they pose a security risk; chunks of\ncached keys and values \"remember\" what came before them, and there's no\nway to scrub that memory without defeating the purpose of caching. As a\nresult, one person's query might be influenced by someone else's query\nif they share chunks of cached keys and values, and in the extreme case,\nit may be possible to infer what other people are querying.\n\n**Sharing KV caches across users can happen.** If those\ntwo people asked the following:\n\n- User 1:\n`What are the key takeaways I want the audience to have...`\n- User 2:\n`What are the key takeaways the audience should have...`\n\nThere is a *shared prefix* of\n`What are the key takeaways` for whom the keys and values are\nidentical. In principle, User 2 could benefit from User 1's cached keys\nand values for that shared prefix; this would make User 2's time to\nfirst token a little quicker than User 1's. However, this can also be\nthe basis for a security problem because it allows User 2 to infer that\nher query shares the same prefix as someone else's. For this reason, [production\nmulti-user inferencing environments do not allow KV caches to be shared\nacross users](https://www.microsoft.com/en-us/research/publication/splitwise-efficient-generative-llm-inference-using-phase-splitting/).\n\nIn practice, shared prefixes occur in the system prompt--that is, the\nsecret instructions that the chatbot sees *before* it gets to\nyour question. The system prompt is typically shared across all chat\nsessions because the chatbot is supposed to follow the same fundamental\ninstructions regardless of the user request. As a result, shared KV\ncaches are useful for chatbots with long system prompts; for example,\nthe Claude models have [very\nlong system prompts which Anthropic makes public](https://platform.claude.com/docs/en/release-notes/system-prompts).\n\n## Slow KV caches can be useful\n\nWe're narrowed down that slow KV caches (like those stored on SSDs or on remote storage) have a pretty narrow scope.\n\n**Sharing prefixes across conversations**: Long system\nprompts or common questions will have identical keys and values, and it\nmay be faster to read them from storage into GPU memory than it would\ntake to recompute them at the start of every conversation.\n\n**Offloading KV cache when there are long delays between\nsuccessive**: It may take me a minute or two to read a chatbot's\nresponse to my question, and during that time, the GPU is sitting idle.\nIn a busy environment, it is more efficient to dump all keys and values\nfrom GPU memory to a slow KV cache, then use that GPU to serve someone\nelse's query. This is called **KV cache offload**. When I\nam ready to ask another question, my keys and values can sometimes be\nreloaded into GPU memory from storage faster than it would take to\nrecompute the entire conversation history's keys and values. This is\nless about performance and more about efficiently juggling users to keep\nGPU utilization high.\n\nThere are more sophisticated efficiency techniques like [disaggregated\ninferencing](https://www.glennklockwood.com/garden/disaggregated-inferencing) which can also use slow KV caches to juggle more users\nand queries in a production environment. As with case #2 above (KV cache\noffloading when there are long delays between turns), this is a\nsophisticated efficiency optimization for inferencing and is primarily\nbeing applied in massive inferencing systems such as those deployed by\nhyperscalers.\n\nBroadly speaking, these ultra-high efficiency, multi-user scenarios\nare where slow KV caches deliver the most benefit. Any time there are\npauses within the back-and-forth of inferencing, KV cache offload can be\nused to free up a GPU to serve someone else's pending query *if the\ncost of offloading and reloading is lower than recomputing*. This\ncriteria tends to appear most often in two cases:\n\n1. **Interacting with codebases and rich media** . This\ncould be someone asking a lot of questions about a large codebase that\nhas been loaded into the model's context. Or it could be someone\ninteracting with rich media like videos, which themselves take up a lot\nof context space.\n2. **Tool calling** . Every time an agent or LLM has to\ncall an external tool (whether it be basic websearch or calling another\nagent), it has to wait for a response before it can continue\ninferencing. If this wait time is long, KV cache offloading can be used\nto allow another inferencing session to use the GPU while the tool is\ndoing its work. Agentic workflows can create huge working contexts as\nwell, making it easier to justify offloading KV cache instead of\nrecomputing.\n\n## Understanding vendor claims\n\nI write all this because AI infrastructure vendors have been running amok, making bold claims about how their KV caching software can give incredible speedups. Below are a few claims I found within a few minutes of Googling.\n\nClaim #1:\n\n**Traditional recompute approach** (112K token\ncontext): 57 seconds processing time\n**(redacted) with KV Cache**: 2.1 seconds loading\ntime\n**Result**: Over **27X\nfaster** performance\n\nClaim #2:\n\n... we observed dramatic improvements in both speed and scalability—delivering up to 75X faster prefill times for long-context prompts, ...\n\nClaim #3:\n\n... boost LLM inference efficiency by 90% and TTFT by 20x using KV Cache Offload ...\n\nClaim #4:\n\n... consistently accelerated inference—delivering up to twenty times and six times respectively for larger, large context multi-GPU models ...\n\nIn all of these cases, these claims are comparing the cost of recomputing all keys and values from scratch (prefill) to simply loading cached keys and values from each company's respective product. They all claim to improve TTFT explicitly (#2 and #3) or implicitly (#1 and #4), but none of them make any statements about decode or end-to-end inferencing speedups--because remember, the KV cache used during decode MUST fit into GPU memory.\n\nSlow KV caches, such as those supported by the above vendors'\nproducts, are only valuable when the cost to prefill is higher than the\ncost of reading from storage. And the cost to prefill is higher with\nlonger prefixes. If you look closely, these claims specifically say\ntheir speedups are for \"[long\ncontext](https://www.glennklockwood.com/garden/long-context)\" tests, because these slow KV caches are much less useful\nwhen the prefix is shorter.\n\nLet's take a closer look.\n\n### Claim #1\n\n**Traditional recompute approach** (112K token\ncontext): 57 seconds processing time\n**(redacted) with KV Cache**: 2.1 seconds loading\ntime\n**Result**: Over **27X\nfaster** performance\n\nIn this first example, prefilling from scratch is faster than using their KV cache if you are prefilling 4K tokens (112K tokens divided by 27x) or less. For reference, this blog post is about 2,600 tokens, and it would be faster to prefill it than load it from this vendor's KV cache.\n\nBy comparison, the 112K token context they tested is about the length of The Hobbit by Tolkien.\n\n### Claim #2\n\n... we observed dramatic improvements in both speed and scalability—delivering up to 75X faster prefill times for long-context prompts, ...\n\nThis 75x claim is based on the time to first token when prefilling from scratch versus loading all keys and values from storage. They used a prefix with 128,000 tokens to demonstrate this speedup. So again, this reflects the speed of recomputing all keys and values for the entirety of The Hobbit versus reading it from storage.\n\n### Claim #3\n\n... boost LLM inference efficiency by 90% and TTFT by 20x using KV Cache Offload ...\n\nAs with above, this is a measurement of prefill improvement, and this was also made using 128K tokens.\n\nIt is tempting to say that this vendor's claim of 20x must mean its storage solution is slower than the vendor claiming 75x. However, this is not the case! The 75x vendor was using a smaller model (70 billion parameters), whereas this 20x vendor was using a bigger model. Bigger models mean more keys and values, more parameters, and more aggregate GPU memory required to serve them.\n\nAs a result, the difference between this claim's 20x and the earlier claim's 75x here reflects some combination of actual KV cache performance, GPU server configuration, and the differences in how the model was distributed over the GPUs used in the test. It's apples and oranges.\n\n... consistently accelerated inference—delivering up to twenty times and six times respectively for larger, large context multi-GPU models ...\n\nAgain, this 20x claim is based on 128K tokens (The Hobbit) being prefilled. This vendor actually showed plots along with this claim, and to their credit, the plots reveal that the speedup drops dramatically with smaller prefix sizes.\n\n### Critically evaluating claims\n\nI don't say all this just to dunk on vendors selling software that is positioned to accelerate KV caching; every vendor is on the same bandwagon and trying to claim success in whatever dimensions of the AI industry they can. However, it is important to understand that these huge speedups are the result of setting up a test where the cost of prefill is very high.\n\nIt's not hard to come up with these scenarios where the cost of recomputing prefill is extreme compared to the cost of simply reading the precomputed keys and values from storage. If you want a KV cache to look really fast, the recipe is simple:\n\n1. Only look at prefill performance. It's the only place that slow KV caches are relevant. Decode performance on a slow KV cache will look atrocious no matter what, which is why nobody does it.\n2. Pick a model with a ton of parameters. More parameters means that prefill (and decode) require more computations.\n3. Test with a very large context window, and fill it to the brim with input tokens. Longer prefixes make prefill take longer.\n4. Run your GPUs at lower power. This reduces its FLOPS (increasing prefill time) but doesn't change its PCIe bandwidth (so I/O time stays the same).\n\nThe result would be a very slow prefill compared to the speed of reading cached keys and values from storage. This will result in a bombastic speedup.", "url": "https://wpnews.pro/news/what-are-kv-caches-really", "canonical_source": "https://blog.glennklockwood.com/2026/09/what-are-kv-caches-really.html", "published_at": "2026-09-22 04:24:07+00:00", "updated_at": "2026-09-22 04:54:21.670247+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "machine-learning"], "entities": ["ChatGPT", "Claude"], "alternates": {"html": "https://wpnews.pro/news/what-are-kv-caches-really", "markdown": "https://wpnews.pro/news/what-are-kv-caches-really.md", "text": "https://wpnews.pro/news/what-are-kv-caches-really.txt", "jsonld": "https://wpnews.pro/news/what-are-kv-caches-really.jsonld"}}