{"slug": "prompt-caching-strategies-to-cut-llm-costs-by-70", "title": "Prompt caching strategies to cut LLM costs by 70%", "summary": "A developer detailed how prompt caching can cut LLM API costs by 70-80% with minimal refactoring. The technique involves marking stable prompt prefixes as cacheable, allowing providers to reuse KV states and charge only a fraction of the input token price for cache reads. The post also covers common pitfalls like dynamic system prompts and offers a pattern for caching conversation history in multi-turn applications.", "body_md": "If you're running LLM-powered features in production, your token bill is probably higher than it should be. Most teams feed the same system prompt, tool definitions, or retrieval context with every request — paying full price to process tokens they've already processed. Prompt caching changes that equation significantly, and it requires almost no refactoring to implement.\n\nPrompt caching lets you mark a prefix of your prompt as cacheable — system instructions, tool schemas, static documents. The provider stores the attention KV state for those tokens on their side. When your next request starts with the exact same prefix, processing is skipped: you pay only for cache read tokens, which are priced at roughly 1/10th of regular input tokens.\n\nBoth major providers support this at the API level. One uses a `cache_control`\n\nblock in the request body; the other exposes `cache_read_input_tokens`\n\nin billing data. The mechanics differ slightly, but the principle is identical.\n\nThe cost math is straightforward. If you're sending a 10,000-token system prompt with every request and you process 1,000 requests per day, that's 10M input tokens daily. With caching, the first write is slightly more expensive (typically 1.25× input price), but each subsequent read is 10× cheaper. Over 1,000 requests, you pay for 1 write and 999 reads — a 70–80% cost reduction on that prefix.\n\nCaching only works on exact prefix matches. One changed character in the cached portion invalidates the cache for that prefix. This constraint shapes how you must structure your prompts: **stable content at the top, dynamic content at the bottom**.\n\n``` python\nimport anthropic\n\nclient = anthropic.Anthropic()\n\n# Build stable content once -- this gets cached across requests\nSYSTEM_PROMPT = (\n    \"You are a security analyst assistant.\"\n    # ... 5000 tokens of static instructions, rules, and context ...\n)\n\nTOOL_DEFINITIONS = []  # your function/tool schemas -- also stable\n\ndef query_llm(user_message: str, session_context: dict) -> str:\n    \"\"\"\n    Cache the stable prefix; dynamic data goes at the bottom in messages[].\n    \"\"\"\n    response = client.messages.create(\n        model=\"claude-opus-5\",\n        max_tokens=1024,\n        system=[\n            {\n                \"type\": \"text\",\n                \"text\": SYSTEM_PROMPT,\n                \"cache_control\": {\"type\": \"ephemeral\"},  # mark for caching\n            }\n        ],\n        tools=TOOL_DEFINITIONS,\n        messages=[\n            {\n                \"role\": \"user\",\n                # Dynamic per-request context goes here, NOT in system\n                \"content\": f\"Context: {session_context}\\n\\nQuestion: {user_message}\"\n            }\n        ],\n    )\n\n    usage = response.usage\n    print(f\"Input: {usage.input_tokens} | Cache read: {usage.cache_read_input_tokens}\")\n    return response.content[0].text\n```\n\nWhat kills cache hit rates in practice:\n\nIf your system prompt is constructed dynamically, refactor it into a fixed static string. Pass per-user data as a user turn message instead.\n\nIn multi-turn applications, conversation history grows with each exchange. You want to cache the stable parts (system prompt, tools) but also checkpoint conversation history so you're not re-processing past turns from scratch.\n\nThe pattern: use two cache markers — one on the system prompt, one at a fixed depth into the conversation history.\n\n``` python\ndef build_messages_with_cache(\n    system_prompt: str,\n    conversation_history: list[dict],\n    new_user_message: str,\n    history_cache_depth: int = 10,\n) -> tuple[list, list]:\n    \"\"\"\n    Returns (system, messages) with cache markers at stable breakpoints.\n    Caches the system prompt + last N turns of history.\n    \"\"\"\n    system = [\n        {\n            \"type\": \"text\",\n            \"text\": system_prompt,\n            \"cache_control\": {\"type\": \"ephemeral\"},\n        }\n    ]\n\n    messages = []\n    history_len = len(conversation_history)\n\n    for i, msg in enumerate(conversation_history):\n        msg_copy = dict(msg)\n        # Place a cache marker at the depth boundary\n        if i == history_len - history_cache_depth and history_len >= history_cache_depth:\n            if isinstance(msg_copy[\"content\"], str):\n                msg_copy[\"content\"] = [\n                    {\n                        \"type\": \"text\",\n                        \"text\": msg_copy[\"content\"],\n                        \"cache_control\": {\"type\": \"ephemeral\"},\n                    }\n                ]\n        messages.append(msg_copy)\n\n    # New message appended without cache marker -- it's the dynamic part\n    messages.append({\"role\": \"user\", \"content\": new_user_message})\n\n    return system, messages\n```\n\nThe cache marker at position `history_len - N`\n\ntells the provider: compute and store KV state up to this point. The next request reuses that stored state if its prefix matches exactly. This works well for chatbots and agents with long sessions — the bulk of the conversation history stops being re-processed after the first time.\n\nBefore optimizing, instrument. Most providers return cache usage in the API response — make it part of your standard logging from day one.\n\n``` python\nimport json\nfrom dataclasses import dataclass, asdict\nfrom datetime import datetime\n\n@dataclass\nclass LLMCallMetrics:\n    timestamp: str\n    model: str\n    input_tokens: int\n    output_tokens: int\n    cache_read_tokens: int\n    cache_write_tokens: int\n    estimated_cost_usd: float\n\n    @property\n    def cache_hit_rate(self) -> float:\n        total = self.input_tokens + self.cache_read_tokens\n        return self.cache_read_tokens / total if total > 0 else 0.0\n\ndef log_llm_call(\n    response,\n    model: str,\n    input_price: float,\n    cache_read_price: float,\n    output_price: float,\n) -> LLMCallMetrics:\n    usage = response.usage\n\n    cost = (\n        (usage.input_tokens / 1_000_000) * input_price\n        + (getattr(usage, \"cache_read_input_tokens\", 0) / 1_000_000) * cache_read_price\n        + (usage.output_tokens / 1_000_000) * output_price\n    )\n\n    metrics = LLMCallMetrics(\n        timestamp=datetime.utcnow().isoformat(),\n        model=model,\n        input_tokens=usage.input_tokens,\n        output_tokens=usage.output_tokens,\n        cache_read_tokens=getattr(usage, \"cache_read_input_tokens\", 0),\n        cache_write_tokens=getattr(usage, \"cache_creation_input_tokens\", 0),\n        estimated_cost_usd=cost,\n    )\n\n    print(json.dumps(asdict(metrics)))\n    print(f\"  -> Cache hit rate: {metrics.cache_hit_rate:.1%}\")\n    return metrics\n```\n\nA cache hit rate below 60% on a system-prompt-heavy workload signals that your prefix is changing between requests. Track this metric over a rolling window. If it drops after a deploy, something in your prompt construction changed.\n\n**Good candidates:**\n\n**Do not try to cache:**\n\nIf you're building compliance or security tooling — say, an assistant that checks configurations against a fixed policy document — load your reference material once, cache it, then vary only the user's specific question. This maps well to the pattern of pre-loading [security hardening checklists](https://ayinedjimi-consultants.fr/checklists) as static context: the document never changes between users, so it caches perfectly.\n\nFor standard chat where context is mostly conversation history, realistic savings are 30–50% depending on turn length and session depth. The 70% figure applies when static prefixes dominate — RAG with fixed corpora, assistants with large tool schemas, or compliance tools with extensive rule sets.\n\nPrompt caching is one of the highest-ROI optimizations available for production LLM workloads: near-zero implementation cost, immediate cost impact, no model quality tradeoff. The main discipline is structural — stable content at the top, dynamic content at the bottom, no \"just add a timestamp\" shortcuts.\n\nStart with a single cache marker on your system prompt. Log `cache_read_input_tokens`\n\nfor every response. If your hit rate is above 80% after a few hundred requests, add multi-level caching for conversation history. If it's below 50%, audit your prefix — something is changing that shouldn't be.\n\nThe pricing asymmetry makes this a no-brainer: cache reads cost roughly 1/10th of input tokens. Write once, read many, pay little.\n\n*I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.*", "url": "https://wpnews.pro/news/prompt-caching-strategies-to-cut-llm-costs-by-70", "canonical_source": "https://dev.to/ayinedjimi-consultants/prompt-caching-strategies-to-cut-llm-costs-by-70-2idi", "published_at": "2026-08-28 10:08:59+00:00", "updated_at": "2026-08-28 10:19:18.637385+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Anthropic", "Claude"], "alternates": {"html": "https://wpnews.pro/news/prompt-caching-strategies-to-cut-llm-costs-by-70", "markdown": "https://wpnews.pro/news/prompt-caching-strategies-to-cut-llm-costs-by-70.md", "text": "https://wpnews.pro/news/prompt-caching-strategies-to-cut-llm-costs-by-70.txt", "jsonld": "https://wpnews.pro/news/prompt-caching-strategies-to-cut-llm-costs-by-70.jsonld"}}