Ten times. That's the price gap between cached and uncached input tokens, and your agent pays the wrong side of it if its prompt is ordered badly. Fixing the order is the cheapest optimization you'll ship this quarter.
- A cache hit only happens when the start of your prompt matches an earlier request token for token, so one changed character near the top voids everything after it.
- Timestamps, reordered tools, per-user text in the system prompt and unstable JSON key order are the usual silent culprits.
- Order every request from most stable to least stable: tools, system prompt, reference material, then the conversation, and only ever append to it.
- Anthropic's usage block reports cache_read_input_tokens and cache_creation_input_tokens, which is all you need to calculate hit rate on every call.
- On a 40-step agent run with a 30,000-token prefix, the arithmetic cuts input cost by roughly 87 percent, but only if the hit rate holds.
Ten times: that's the gap between what cached and uncached input tokens cost. In an engineering post on context design, the team behind the Manus agent wrote that on Claude Sonnet, cached input tokens cost $0.30 per million against $3 for uncached ones. They also said their agents read about 100 input tokens for every output token. Agents are input-heavy by nature: every step resends the tool definitions, the instructions and the whole history so far. So how does prompt caching reduce LLM costs? Mostly by making that repeated input cheap.
If those repeated tokens bill at the cached rate, your inference bill drops sharply. If they don't, you pay full price to say the same thing forty times in a row. Most teams are in the second group and have no idea, because nothing breaks. The agent works. The invoice is just larger than it needs to be. Manus went as far as calling the KV-cache hit rate the single most important metric for a production agent. Frankly, that's right, and it's the metric most startups never look at.
Providers don't cache your whole prompt as a document. They cache the computed state for a prefix, the opening stretch of the request, and they reuse it when a later request starts with exactly the same tokens. Same tokens, same order, from the first one onward. The moment your new request differs from the stored one, everything from that point forward is processed fresh and billed at the normal rate.
How Does Prompt Caching Work for LLMs, and Why It Cuts Bills in Half How does prompt caching work for LLMs? It's a token-storage system with its own write and read pricing, its own expiration clock, and a hidden minimum size that fails silently if you miss it. Get the prompt order wrong and you pay full price every time, even with caching turned on. - how prompt caching works for language models - why prompt caching reduces LLM API costs significantly
That's why a prefix matters more than a total. A 40,000-token prompt where the first 200 tokens change on every call caches almost nothing. The same prompt with those 200 tokens moved to the end caches nearly all of it.
Anthropic's documentation lays out the order it builds the prefix in: tools first, then the system prompt, then messages. A change to a tool definition invalidates the cache for all three layers. A change to the system prompt invalidates the system and message layers. A change in the middle of the conversation invalidates everything after it. You mark where you want a cache boundary with a cache_control breakpoint, and you get up to four of them per request.
The pricing follows the same logic. Anthropic's docs list a cache write at 1.25 times the base input price for the default five-minute lifetime, and a cache read at 0.1 times. A one-hour lifetime is available at 2 times the base price for the write. The five-minute clock resets every time the cache is read, so a busy agent keeps its own cache alive. A quiet one lets it expire and pays the write premium again. Check your provider's current price sheet before you plan around these numbers, since they differ by model and change.
OpenAI works differently on the surface. Caching is automatic for prompts of 1,024 tokens or more, with no breakpoints to place, and the API reports how many tokens were served from cache in the prompt_tokens_details.cached_tokens field. The principle is identical, though. Exact prefix, or no hit. OpenAI also lets you pass a prompt_cache_key, which helps route similar requests to the same cache and is worth setting if many users share one long prefix.
Four ways agents quietly break their own prompt cache #
The timestamp is the classic. A developer adds 'The current date and time is 2026-11-10 14:32:07' to the top of the system prompt so the agent knows when it is. Every request now has a different first line, and the cache never hits, on any of the 30,000 tokens that follow. Manus names this exact mistake in its post. If the model needs the date, put it in the latest user message, or give it a tool that returns the time. Better still, send the date at day granularity and place it after your cached block.
Tool order is sneakier. Many frameworks build the tool list from a registry, a dictionary or a database query, and the order isn't guaranteed. Two requests with the same five tools in different sequence are different prefixes. The same goes for agents that add and remove tools depending on the task, which is tempting because a shorter tool list feels cheaper. It isn't. Because tools sit at the very top of the prefix, every removal invalidates the whole cache. Manus's answer was to keep the tool set fixed and mask which tools the model is allowed to pick, rather than deleting definitions mid-run.
Then there's serialization. If you build tool outputs or memory snapshots with a JSON library that doesn't sort keys, the same data can come out in a different key order from one run to the next. Humans read it as identical. The tokenizer doesn't. Sort your keys, fix your whitespace, and keep the formatting deterministic.
How Context Window Length Actually Breaks Your AI Agent's Unit Economics How does context window length affect inference cost? Not linearly. Attention scales roughly with the square of context length, and the KV cache adds a growing memory tax on top, which is exactly where AI agent unit economics quietly fall apart. - how context window size affects AI agent costs - why large language model context breaks unit economics
The fourth is editing history. Agents that rewrite earlier turns, trim old tool results to save space, or insert a fresh summary in the middle of the conversation break the prefix at the point of the edit. Everything after it is billed as new. Context is append-only if you want it cached, and that's an uncomfortable rule for teams used to tidying up a long conversation.
A fifth culprit hides in plain sight: per-user text in the system prompt. 'You are helping Maria from Acme Corp' makes every customer's prefix unique, so you cache one request per customer instead of one prompt for all of them. Personalization belongs near the end.
How to restructure prompt caching for AI agents #
Think of your request as a stack ordered by how often each layer changes. At the bottom, the part that never changes: tool definitions in a fixed order, then the system prompt with your rules and tone. Next, material that changes rarely, such as a product manual, a policy document or a coding style guide. Then material that changes per session: the user's profile, their open project, the date. Last, the conversation itself, growing one turn at a time.
Put your cache breakpoint at the end of the stable layers. With Anthropic, that's a cache_control marker on the last block of your system prompt or reference material. A second breakpoint on the final message of the conversation lets each new step read everything before it and write only the new turn. This is the setup that gets multi-step agents from a trickle of hits to most of the input served from cache.
Two practical details trip people up. First, there's a minimum size. Anthropic's docs set a floor of 1,024 tokens for most models and a higher one for some, and a prefix below it simply isn't cached, with no error to tell you. Short prompts on a cheap model may not be worth the effort. Second, parallel calls. If you fire ten sub-agent requests at once with the same prefix, none of them can read a cache the others are still writing. Send one first, let it land, then fan out.
Sub-agents deserve their own thought. An orchestrator that spawns workers with a fresh, differently worded prompt each time pays full price on every worker. Give workers a shared prefix and put the task-specific instruction at the end.
There's one trade-off worth admitting. Putting dynamic text last sometimes makes the prompt read less naturally, and you may lose a little steering power by moving a key instruction away from the top. Test it. In most agents I'd expect the saving to dwarf the difference.
Measure your LLM prompt cache hit rate before you trust it #
You can't see a cache miss in your logs unless you log the right fields. Anthropic's response carries three: input_tokens, the part that wasn't cached and wasn't written; cache_creation_input_tokens, what you paid to write; and cache_read_input_tokens, what was served from cache. Hit rate is cache reads divided by the sum of all three. Log it per request, tagged with the agent name, the step number and the prompt version.
Look at it two ways. The first is the average across a run, which tells you whether the design works. The second is by step. A healthy agent shows a low rate on step one, since nothing has been written yet, and a rate that climbs toward 90 percent and stays there. If step two is as low as step one, something above your conversation is changing between calls, and you know where to look.
Set an alert on it. A deploy that adds a timestamp, a framework upgrade that reorders tools or a new feature flag in the system prompt will each show up as a cliff in the chart, and without the alert you'll find out at the end of the month. Treat a drop in hit rate the way you'd treat a jump in latency: as a regression with a commit behind it.
For OpenAI, the equivalent is cached_tokens divided by prompt_tokens in the usage object. For providers that don't expose this, compare your billed input tokens to what you sent. A gap of zero means you're getting no caching at all, however the marketing page describes it.
What the saving looks like on a real run #
Take the Sonnet rates Manus cited, $3 per million tokens uncached and $0.30 cached, and a write premium of 1.25 times. Say your agent has a 30,000-token prefix of tools, instructions and reference material, and a task that takes 40 steps.
With no caching, you send that prefix 40 times: 1.2 million tokens, or $3.60. With caching, step one writes it for about $0.11, and the other 39 steps read it for about $0.35. The prefix now costs roughly $0.46, a cut of about 87 percent. The conversation tokens added on each step are extra in both versions, so your total saving will be smaller, but it's the repeated prefix that dominates an agent's bill.
That's the case for doing this before anything else. Teams under cost pressure usually reach for a cheaper model first, which trades quality for savings and means re-running every evaluation. Fixing your prefix changes no output at all. The model sees the same tokens, the same answer comes back, and the invoice falls.
It doesn't work everywhere. One-shot calls with short prompts, traffic that arrives less often than every five minutes, and prompts that are mostly unique content won't gain much. Measure first. If your hit rate is already above 80 percent, you've done the work, and the next saving is somewhere else.
Also read: How to Calculate Gross Dollar Retention by Segment Before Diligence Does • How to Build an AI Agent Kill Switch That the Agent Can't Talk Its Way Around • Build your AI agent fallback routing policy before the outage hits
This article is posted in AI News, check it out for more related stories.
Join the discussion #
Open in the community → Almost there. Sign in and your reply posts straight away.