{"slug": "optimizing-prompt-cache", "title": "Optimizing prompt cache", "summary": "Prompt caching can reduce the cost per task for LLM API calls by reusing the expensive prefill computation for repeated input prefixes, with cache hits skipping the heavy computation and lowering time-to-first-token (TTFT). The article, based on real data from the author's coding tasks, explains that input pricing is much cheaper than output pricing (3x-10x difference) and that optimizing cache usage is key to deliberate cost reduction. The author notes that no AI was used in writing the article.", "body_md": "You probably know what prompt caching is (if not, there’s a quick intro) but what you may not know is the massive impact it has on cost per task (the token cost to do something).\n\nThis is a short visual article based on real data from my own coding tasks to show:\n\nWhat prompt cache is, how it works and why it’s important?\n\nHow to optimize your cache for deliberate cost reduction.\n\n**Declaration: no AI is used in this article.**\n\n## Understanding LLM APIs\n\nThe most common LLMs are [auto-regressive](https://en.wikipedia.org/wiki/Autoregressive_model) meaning they predict future tokens based on a given set of tokens:\n\n*Note: there are also diffusion models, for example Gemini Diffusion but for agentic workflows, auto-regressive dominates the majority of use cases.*\n\nIn the diagram above the green part contains:\n\n**System instructions:****System prompt:** high level instructions including persona and tone**Tool definitions:** to let the model know what tools are available**Skill definitions:** to let the model know of task-specific instructions\n\n**Session history:** what happened up until here (empty in a fresh session)**Extra context:**(optional) open files, attachments, RAG, etc.** User prompt:**what did the user ask\n\nBut to the model, these are all tokens. Upon receiving the request, the inference engine does 3 things:\n\n**Tokenization:** mapping string to numbers that the model can understand.**Prefill:** compute the KV Cache (key-value cache) which acts as the model’s short-term memory containing the attention state.**Decoding:** generating output, one token at a time.\n\nThe three phases impact two distinct metrics:\n\n**TTFT:** the delta between when the request is received till the first token of the response is ready**TPS:** tokens (decoded) per second\n\nI’ve skipped the irrelevant details to keep it focused on the topic (cache optimization) but what you need to know is this:\n\n**Prefill** is compute-bound and heavily**parallelizable**.** Decoding**is memory-bound and** serial**because for every token, the entire model weight and the growing KV cache must be processed.\n\nThe this shows in the API pricing model:\n\n**Input pricing:** much cheaper because it takes less time**Output pricing:** significantly more expensive (3x-10x) because the serial processing takes more time.\n\nHere’s an example [showing prices of two models](https://openrouter.ai/compare/google/gemma-4-26b-a4b-it/qwen/qwen3.6-35b-a3b) on the same provider:\n\nYou might be using local models, but the “cost” still holds in terms of time and performance. Here’s an example of a simple workload I ran locally:\n\nHere are the results using Gemma 4 26B on llama-cpp:\n\n**Prompt processing:**`140+`\n\nTPS and took`10s`\n\n**Token generation:**`~11 TPS`\n\nand took`36s`\n\nThat costs time and electricity so it’s not free.\n\nOK, you probably know where this is heading! Let’s switch gears and see how prompt caching comes into the picture.\n\n## What is prompt caching\n\nThe concept is very similar to how conventional API caching worked before AI.\n\nThe server caches the expensive computations for a period of time.\n\nUpon receiving a request, it checks the cache:\n\n**Cache hit:** a similar payload is already processed. Using it skips the heavy computation, which reduces the cost and increases TTFT**Cache miss:** no similar payload was processed before: pay the full price of computation (higher cost and TTFT).\n\nThis happens opaquely, meaning the user doesn’t know for sure if their request can be cached or not.\n\nNo rocket science there, but you need to remember that the result of decoding is likely to come back in the next request. That’s why it’s stored in the cache.\n\nAnother [obvious] point is that although cache is cheap, it’s not free. Therefore an [LRU](https://en.wikipedia.org/wiki/Cache_replacement_policies#LRU) (least recently used) algorithm regularly evicts old entries from the cache.\n\nNote that caching the output is not done for *idempotency*: the API isn’t designed to return the exact same response to the exact same request! If you have that kind of workload, you should maintain your cache outside the inference API. This cache is primarily for reducing the computation when generating output.\n\nHere’s the [cache pricing](https://api-docs.deepseek.com/quick_start/pricing/) from DeepSeek (they’re going to increase it soon):\n\nAnd here’s [Gemini 3.6 Flash pricing](https://ai.google.dev/gemini-api/docs/pricing):\n\n[Anthropic](https://platform.claude.com/docs/en/about-claude/pricing#prompt-caching) and [OpenAI](https://developers.openai.com/api/docs/pricing) both use a more complex cache pricing model where cache storage is also factored in to the calculations just like Google.\n\n## How to optimize cache for agentic workflows?\n\nNow the juicy stuff!\n\nTo understand how to optimize prompt caching, we first need to look into what is being cached.\n\nAs mentioned, the request payload (AKA prompt prefix) is composed of multiple parts:\n\nThe tool and skill descriptions let the LLM know what’s available while the “extra context” depends on the harness (e.g. VS Code Copilot may add the name of the file currently open in your IDE while Claude may convert a PDF and append it to the user prompt).\n\nAs your conversation grows, you keep sending all those parts to the LLM:\n\nWhether it is cached or not, the LLM needs the entire KV cache for its token generation.\n\nBut what is cachable?\n\nAnd this is the key to cost saving and speed in agentic workflows as each turn of the conversation can reuse a chunk of the growing cache.\n\nThere is a catch [no pun intended] though!\n\nIf you change a single character in the cache, it will be invalidated up until that point.\n\nLet’s say we change the list of tools available in the middle of the conversation. That means TD (tool definitions) and everything after it need to be computed again:\n\nIf you don’t want to pay cache miss prices, you should treat the session as immutable and avoid modifying it. If you mutate it, the earlier the mutation, the more of the cache you miss, leading to higher cost and delay.\n\n## System prompt\n\nIn the previous diagrams, we represented the tool and skill descriptions with two tiny squares. In reality, they can be the lion share of the system prompt and there’s huge opportunity for optimization.\n\nHere’s an example from my real workflows. I analyzed 3 agentic coding sessions and got these numbers:\n\nTool descriptions: 13.5k tokens\n\nSkill descriptions: 2.5k tokens\n\nThe raw system prompt (“You are a helpful assistant…”) were merely 1-8k tokens (depending on the task and harness).\n\nIn VS code some 50+ tools are included by default:\n\nFor comparison, Pi only has 4 tools: `read`\n\n, `write`\n\nand `edit`\n\nfiles + `bash`\n\n.\n\nThis makes Pi a much leaner option for local agentic coding because the prompt processing (prefill) phase on consumer hardware can be significantly compute-heavy.\n\n### Tips for tools & skills\n\nRegardless of the harness, here are a few pragmatic tips:\n\nRight after starting a session, choose the available tools before sending the first prompt to the LLM. As a side benefit, this also improves the quality of the agentic work because models tend to get confused when exposed to more than 30-50 tools (depending on model).\n\nThe situation for skills is more tricky because the harness usually assembles a list from the local and global skill folders and you have no control over it. One tip is to put your skills in a repo-local location (e.g. stored in .agent/skills) instead of relying on global location (~/.agent/skills). This allows you to limit the available skills to what’s relevant in a given repo.\n\nDo\n\n**NOT** change the available tools or skills**AFTER** a session has started especially if it’s a longer session. If you must, compact the session before mutating early tokens (system prompt).Session affinity: use the same model for the whole session. Changing the model or its config (e.g. thinking effort) mid-session, invalidates the cache.\n\nThat last item is why you see warnings like these in VS Code:\n\n## Session history\n\nWe showed the session history (earlier conversations) like this:\n\nH=session history\n\nU=user prompt (or the LLM, in an agentic\n\n*loop*setup)A=assistant response\n\nAs the session grows, there will be more and more tokens in the KV cache:\n\nThe longer the session, the longer it takes to process it. The computation time increases quadratically: `O(N²)`\n\n. Doubling the prompt length **quadruples** the prefill time.\n\nFortunately, this is very cache-friendly:\n\nA typical agentic workflow takes anywhere from 20-200 turns the majority of which is tool calls:\n\nLLM specifies the tool parameters for a call\n\nHarness adds the result of the tool to the session\n\nBoth of these can be quite big (analyzing a few of my sessions, a single tool invocation+result adds 1-3k tokens to the context window).\n\nNow we have another thing to think about: as the chat session grows, the size of KV cache grows, negatively impacting the token generation (TPS) speed.\n\nUnlike prompt processing (prefill) which increases quadratically with total input length `O(N²)`\n\n, token generation latency increases linearly with the KV cache length: `O(L)`\n\n. In other words, the longer the session gets, the lower the TPS goes.\n\nHowever, unlike prefill which can be cached, generating every token requires processing the entire KV cache.\n\nYour context window might be 1M tokens, but that doesn’t mean that token `#900,000`\n\nis generated at the same speed as token `#9000`\n\n.\n\nThere are primarily 2 ways to address this:\n\n**Start a new session:** this is not recommended mid-task.**Session compression:**(AKA compaction) retain important information while collapsing the old turns into a summary\n\nThere are two ways to compress a session:\n\n**Manual compaction:** the user types a`/compact [optional instructions]`\n\nprompt which will be paired with a predefined compaction prompt to get a summary.**Automatic compaction:** the harness keeps tabs on the session length and initiates the compression as needed.\n\nHere’s the automated compaction flow:\n\n### Tips for session history\n\nUse this heuristic:\n\nBefore typing a prompt ask yourself: does the model need the session history up to this point? If not, start a new session.\n\nIf the answer is yes, ask yourself: does it need the exact conversation or can it do with a summary? If yes, compress the session.\n\nCompression has a cost. You’re practically invalidating a cache that otherwise would be a hit and replacing it with a conversation summary which should go through the prefill phase on the next turn. If you trigger compression too frequently, you may actually spend more compute and money than proceeding with raw session.\n\nCompression is lossy, meaning some information may be lost. Typically the harness comes with a pre-tested compression prompt that puts more weight on the latest turns than the middle of the conversation. This means some critical pieces of information (e.g. error codes) may get lost, negatively impacting the LLM performance.\n\nUse the optional instruction to tell the model what to bring to the summary. You make that decision based on where you want the conversation to go in the rest of the session.\n\n[My monetization strategy](https://blog.alexewerlof.com/i/141786627/q-what-is-your-monetization-strategy) is to give away most content for free but these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to **like**, **subscribe** and **share** it. If you really want to support me lifting our community, you can consider a paid subscription. If you want to save, you can get 20% off via [this link](https://blog.alexewerlof.com/protipsdiscount). As a token of appreciation, subscribers get full access to the Pro-Tips sections and my online book [Reliability Engineering Mindset](https://blog.alexewerlof.com/p/rem). Your contribution also funds my open-source products like [Service Level Calculator](https://slc.alexewerlof.com/). You can also [invite your friends](https://blog.alexewerlof.com/leaderboard) to gain free access or save via a [group subscription](https://blog.alexewerlof.com/subscribe?group=true).\n\n*And to those of you who already support me, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.*", "url": "https://wpnews.pro/news/optimizing-prompt-cache", "canonical_source": "https://blog.alexewerlof.com/p/optimizing-prompt-cache", "published_at": "2026-08-08 14:07:13+00:00", "updated_at": "2026-08-09 12:13:10.231524+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure"], "entities": ["Gemma 4 26B", "llama-cpp", "OpenRouter"], "alternates": {"html": "https://wpnews.pro/news/optimizing-prompt-cache", "markdown": "https://wpnews.pro/news/optimizing-prompt-cache.md", "text": "https://wpnews.pro/news/optimizing-prompt-cache.txt", "jsonld": "https://wpnews.pro/news/optimizing-prompt-cache.jsonld"}}