Prompt caching strategies to cut LLM costs by 70%

A developer detailed how prompt caching can cut LLM API costs by 70-80% with minimal refactoring. The technique involves marking stable prompt prefixes as cacheable, allowing providers to reuse KV states and charge only a fraction of the input token price for cache reads. The post also covers common pitfalls like dynamic system prompts and offers a pattern for caching conversation history in multi-turn applications.

If you're running LLM-powered features in production, your token bill is probably higher than it should be. Most teams feed the same system prompt, tool definitions, or retrieval context with every request — paying full price to process tokens they've already processed. Prompt caching changes that equation significantly, and it requires almost no refactoring to implement. Prompt caching lets you mark a prefix of your prompt as cacheable — system instructions, tool schemas, static documents. The provider stores the attention KV state for those tokens on their side. When your next request starts with the exact same prefix, processing is skipped: you pay only for cache read tokens, which are priced at roughly 1/10th of regular input tokens. Both major providers support this at the API level. One uses a cache control block in the request body; the other exposes cache read input tokens in billing data. The mechanics differ slightly, but the principle is identical. The cost math is straightforward. If you're sending a 10,000-token system prompt with every request and you process 1,000 requests per day, that's 10M input tokens daily. With caching, the first write is slightly more expensive typically 1.25× input price , but each subsequent read is 10× cheaper. Over 1,000 requests, you pay for 1 write and 999 reads — a 70–80% cost reduction on that prefix. Caching only works on exact prefix matches. One changed character in the cached portion invalidates the cache for that prefix. This constraint shapes how you must structure your prompts: stable content at the top, dynamic content at the bottom . python import anthropic client = anthropic.Anthropic Build stable content once -- this gets cached across requests SYSTEM PROMPT = "You are a security analyst assistant." ... 5000 tokens of static instructions, rules, and context ... TOOL DEFINITIONS = your function/tool schemas -- also stable def query llm user message: str, session context: dict - str: """ Cache the stable prefix; dynamic data goes at the bottom in messages . """ response = client.messages.create model="claude-opus-5", max tokens=1024, system= { "type": "text", "text": SYSTEM PROMPT, "cache control": {"type": "ephemeral"}, mark for caching } , tools=TOOL DEFINITIONS, messages= { "role": "user", Dynamic per-request context goes here, NOT in system "content": f"Context: {session context}\n\nQuestion: {user message}" } , usage = response.usage print f"Input: {usage.input tokens} | Cache read: {usage.cache read input tokens}" return response.content 0 .text What kills cache hit rates in practice: If your system prompt is constructed dynamically, refactor it into a fixed static string. Pass per-user data as a user turn message instead. In multi-turn applications, conversation history grows with each exchange. You want to cache the stable parts system prompt, tools but also checkpoint conversation history so you're not re-processing past turns from scratch. The pattern: use two cache markers — one on the system prompt, one at a fixed depth into the conversation history. python def build messages with cache system prompt: str, conversation history: list dict , new user message: str, history cache depth: int = 10, - tuple list, list : """ Returns system, messages with cache markers at stable breakpoints. Caches the system prompt + last N turns of history. """ system = { "type": "text", "text": system prompt, "cache control": {"type": "ephemeral"}, } messages = history len = len conversation history for i, msg in enumerate conversation history : msg copy = dict msg Place a cache marker at the depth boundary if i == history len - history cache depth and history len = history cache depth: if isinstance msg copy "content" , str : msg copy "content" = { "type": "text", "text": msg copy "content" , "cache control": {"type": "ephemeral"}, } messages.append msg copy New message appended without cache marker -- it's the dynamic part messages.append {"role": "user", "content": new user message} return system, messages The cache marker at position history len - N tells the provider: compute and store KV state up to this point. The next request reuses that stored state if its prefix matches exactly. This works well for chatbots and agents with long sessions — the bulk of the conversation history stops being re-processed after the first time. Before optimizing, instrument. Most providers return cache usage in the API response — make it part of your standard logging from day one. python import json from dataclasses import dataclass, asdict from datetime import datetime @dataclass class LLMCallMetrics: timestamp: str model: str input tokens: int output tokens: int cache read tokens: int cache write tokens: int estimated cost usd: float @property def cache hit rate self - float: total = self.input tokens + self.cache read tokens return self.cache read tokens / total if total 0 else 0.0 def log llm call response, model: str, input price: float, cache read price: float, output price: float, - LLMCallMetrics: usage = response.usage cost = usage.input tokens / 1 000 000 input price + getattr usage, "cache read input tokens", 0 / 1 000 000 cache read price + usage.output tokens / 1 000 000 output price metrics = LLMCallMetrics timestamp=datetime.utcnow .isoformat , model=model, input tokens=usage.input tokens, output tokens=usage.output tokens, cache read tokens=getattr usage, "cache read input tokens", 0 , cache write tokens=getattr usage, "cache creation input tokens", 0 , estimated cost usd=cost, print json.dumps asdict metrics print f" - Cache hit rate: {metrics.cache hit rate:.1%}" return metrics A cache hit rate below 60% on a system-prompt-heavy workload signals that your prefix is changing between requests. Track this metric over a rolling window. If it drops after a deploy, something in your prompt construction changed. Good candidates: Do not try to cache: If you're building compliance or security tooling — say, an assistant that checks configurations against a fixed policy document — load your reference material once, cache it, then vary only the user's specific question. This maps well to the pattern of pre-loading security hardening checklists https://ayinedjimi-consultants.fr/checklists as static context: the document never changes between users, so it caches perfectly. For standard chat where context is mostly conversation history, realistic savings are 30–50% depending on turn length and session depth. The 70% figure applies when static prefixes dominate — RAG with fixed corpora, assistants with large tool schemas, or compliance tools with extensive rule sets. Prompt caching is one of the highest-ROI optimizations available for production LLM workloads: near-zero implementation cost, immediate cost impact, no model quality tradeoff. The main discipline is structural — stable content at the top, dynamic content at the bottom, no "just add a timestamp" shortcuts. Start with a single cache marker on your system prompt. Log cache read input tokens for every response. If your hit rate is above 80% after a few hundred requests, add multi-level caching for conversation history. If it's below 50%, audit your prefix — something is changing that shouldn't be. The pricing asymmetry makes this a no-brainer: cache reads cost roughly 1/10th of input tokens. Write once, read many, pay little. I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.