If you're running LLM-powered features in production, your token bill is probably higher than it should be. Most teams feed the same system prompt, tool definitions, or retrieval context with every request — paying full price to process tokens they've already processed. Prompt caching changes that equation significantly, and it requires almost no refactoring to implement.
Prompt caching lets you mark a prefix of your prompt as cacheable — system instructions, tool schemas, static documents. The provider stores the attention KV state for those tokens on their side. When your next request starts with the exact same prefix, processing is skipped: you pay only for cache read tokens, which are priced at roughly 1/10th of regular input tokens.
Both major providers support this at the API level. One uses a cache_control
block in the request body; the other exposes cache_read_input_tokens
in billing data. The mechanics differ slightly, but the principle is identical.
The cost math is straightforward. If you're sending a 10,000-token system prompt with every request and you process 1,000 requests per day, that's 10M input tokens daily. With caching, the first write is slightly more expensive (typically 1.25× input price), but each subsequent read is 10× cheaper. Over 1,000 requests, you pay for 1 write and 999 reads — a 70–80% cost reduction on that prefix.
Caching only works on exact prefix matches. One changed character in the cached portion invalidates the cache for that prefix. This constraint shapes how you must structure your prompts: stable content at the top, dynamic content at the bottom.
import anthropic
client = anthropic.Anthropic()
SYSTEM_PROMPT = (
"You are a security analyst assistant."
)
TOOL_DEFINITIONS = [] # your function/tool schemas -- also stable
def query_llm(user_message: str, session_context: dict) -> str:
"""
Cache the stable prefix; dynamic data goes at the bottom in messages[].
"""
response = client.messages.create(
model="claude-opus-5",
max_tokens=1024,
system=[
{
"type": "text",
"text": SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"}, # mark for caching
}
],
tools=TOOL_DEFINITIONS,
messages=[
{
"role": "user",
"content": f"Context: {session_context}\n\nQuestion: {user_message}"
}
],
)
usage = response.usage
print(f"Input: {usage.input_tokens} | Cache read: {usage.cache_read_input_tokens}")
return response.content[0].text
What kills cache hit rates in practice:
If your system prompt is constructed dynamically, refactor it into a fixed static string. Pass per-user data as a user turn message instead.
In multi-turn applications, conversation history grows with each exchange. You want to cache the stable parts (system prompt, tools) but also checkpoint conversation history so you're not re-processing past turns from scratch.
The pattern: use two cache markers — one on the system prompt, one at a fixed depth into the conversation history.
def build_messages_with_cache(
system_prompt: str,
conversation_history: list[dict],
new_user_message: str,
history_cache_depth: int = 10,
) -> tuple[list, list]:
"""
Returns (system, messages) with cache markers at stable breakpoints.
Caches the system prompt + last N turns of history.
"""
system = [
{
"type": "text",
"text": system_prompt,
"cache_control": {"type": "ephemeral"},
}
]
messages = []
history_len = len(conversation_history)
for i, msg in enumerate(conversation_history):
msg_copy = dict(msg)
if i == history_len - history_cache_depth and history_len >= history_cache_depth:
if isinstance(msg_copy["content"], str):
msg_copy["content"] = [
{
"type": "text",
"text": msg_copy["content"],
"cache_control": {"type": "ephemeral"},
}
]
messages.append(msg_copy)
messages.append({"role": "user", "content": new_user_message})
return system, messages
The cache marker at position history_len - N
tells the provider: compute and store KV state up to this point. The next request reuses that stored state if its prefix matches exactly. This works well for chatbots and agents with long sessions — the bulk of the conversation history stops being re-processed after the first time.
Before optimizing, instrument. Most providers return cache usage in the API response — make it part of your standard logging from day one.
import json
from dataclasses import dataclass, asdict
from datetime import datetime
@dataclass
class LLMCallMetrics:
timestamp: str
model: str
input_tokens: int
output_tokens: int
cache_read_tokens: int
cache_write_tokens: int
estimated_cost_usd: float
@property
def cache_hit_rate(self) -> float:
total = self.input_tokens + self.cache_read_tokens
return self.cache_read_tokens / total if total > 0 else 0.0
def log_llm_call(
response,
model: str,
input_price: float,
cache_read_price: float,
output_price: float,
) -> LLMCallMetrics:
usage = response.usage
cost = (
(usage.input_tokens / 1_000_000) * input_price
+ (getattr(usage, "cache_read_input_tokens", 0) / 1_000_000) * cache_read_price
+ (usage.output_tokens / 1_000_000) * output_price
)
metrics = LLMCallMetrics(
timestamp=datetime.utcnow().isoformat(),
model=model,
input_tokens=usage.input_tokens,
output_tokens=usage.output_tokens,
cache_read_tokens=getattr(usage, "cache_read_input_tokens", 0),
cache_write_tokens=getattr(usage, "cache_creation_input_tokens", 0),
estimated_cost_usd=cost,
)
print(json.dumps(asdict(metrics)))
print(f" -> Cache hit rate: {metrics.cache_hit_rate:.1%}")
return metrics
A cache hit rate below 60% on a system-prompt-heavy workload signals that your prefix is changing between requests. Track this metric over a rolling window. If it drops after a deploy, something in your prompt construction changed.
Good candidates:
Do not try to cache:
If you're building compliance or security tooling — say, an assistant that checks configurations against a fixed policy document — load your reference material once, cache it, then vary only the user's specific question. This maps well to the pattern of pre- security hardening checklists as static context: the document never changes between users, so it caches perfectly.
For standard chat where context is mostly conversation history, realistic savings are 30–50% depending on turn length and session depth. The 70% figure applies when static prefixes dominate — RAG with fixed corpora, assistants with large tool schemas, or compliance tools with extensive rule sets.
Prompt caching is one of the highest-ROI optimizations available for production LLM workloads: near-zero implementation cost, immediate cost impact, no model quality tradeoff. The main discipline is structural — stable content at the top, dynamic content at the bottom, no "just add a timestamp" shortcuts.
Start with a single cache marker on your system prompt. Log cache_read_input_tokens
for every response. If your hit rate is above 80% after a few hundred requests, add multi-level caching for conversation history. If it's below 50%, audit your prefix — something is changing that shouldn't be.
The pricing asymmetry makes this a no-brainer: cache reads cost roughly 1/10th of input tokens. Write once, read many, pay little.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.