How a model upgrade took our cache hit rate from ~90% to almost nothing, and why the fix was about where our prompt text sits across messages, not what it says.
TL;DR
GPT-4o caches any identical prefix in 128-token steps. It doesn’t care where your messages begin or end.
GPT-5.6 (Sol, Terra, Luna) can only stop the cache at the end of an eligible message. A prefix that ends in the middle of a message does not count.
If you send one big system message with the variable text at the end, the only place the cache can stop is after the text that changes on every request. On GPT-5.6 you then pay a 1.25× cache write on every call and never read any of it back.
The fix: static text in system, everything variable in user. In a replay of 1,000 real production requests, the share of input tokens served from cache went back to 89%, on par with GPT-4o. The provisioned capacity we needed dropped by about 40% compared with a half-fix.
Next steps: explicit cache breakpoints and prompt_cache_key.
- The pattern that used to be fine
Many LLM applications build one long prompt and send it as a single system message:
Instructions (fixed)
Reference data: rules, catalogues, examples (fixed, or one of a few variants)
Conversation history or session context (varies per request)
The latest user input (varies per request)
prompt = instructions + reference_data + history + f"\nUser input: {user_input}"
response = client.chat.completions.create( model=deployment, messages=[{"role": "system", "content": prompt}],)
Our prompt looked exactly like this, at about 9,500 tokens. On GPT-4o it was cheap: around 91% of input tokens came from cache.
- Why it worked on GPT-4o
GPT-4o’s prompt caching is prefix-based at fixed intervals:
Caching only applies once a prompt has at least 1,024 tokens, and those first 1,024 must be identical.
After that, the cached prefix grows in 128-token blocks, counted from the start of the prompt.
Message boundaries don’t matter. The provider places the cut points for you.
So parts 1 and 2 (identical on every call) were cached no matter what followed. Only the variable tail missed the cache.
- What changed in GPT-5.6
GPT-5.6 shipped with a new caching model. The key sentence in the docs says where the cache is allowed to look for a match:
the implicit breakpoint, up to 20 earlier eligible message endings, and the endpoint of the initial consecutive block of developer messages
“Eligible message endings” are user messages, the last tool response in a group, and the last developer/system message in the initial block. Compared with earlier models:
Where the cache can stop. GPT-4o to 5.5: at fixed 128-token intervals. GPT-5.6: only at the end of an eligible message, or at an explicit breakpoint.
A prefix that ends mid-message. GPT-4o to 5.5: counts. GPT-5.6: doesn’t count.
Cache write. GPT-4o to 5.5: free. GPT-5.6: 1.25× the input price.
Cache read. GPT-4o to 5.5: discounted. GPT-5.6: 0.1× the input price.
Now apply those rules to a single-message prompt. There is exactly one eligible ending: the end of the system message, which comes after the user's input. Every request writes a cache entry that ends in text no other request will repeat. The next request looks for that prefix, doesn't find it, and writes its own.
Result: you pay a premium to write to the cache on every call and never get a hit.
The model didn’t get worse. The prompt was built on an assumption, “the cache follows the bytes”, that stopped being true.
- The fix: every variable byte goes in user
The rule is simple:
The system message must be identical for every request that should share a cache entry. Anything that varies per call, such as history, session context or the user's input, goes after it.
messages = [ {"role": "system", "content": instructions + reference_data}, {"role": "user", "content": history + f"\nUser input: {user_input}"},]
If your code already builds one string, you can split it at the first variable section without touching the prompt templates:
VARIABLE_SECTION_MARKERS = ("\n---\nConversation history:", "\n---\nSession context:")USER_INPUT_MARKER = "\nUser input: "
php
def split_prompt_messages(prompt: str) -> list[dict]: """Split a single prompt into a static system message and a variable user message.""" input_at = prompt.rfind(USER_INPUT_MARKER) if input_at == -1: return [{"role": "system", "content": prompt}] cut = input_at for marker in VARIABLE_SECTION_MARKERS: i = prompt.find(marker, 0, input_at) if i != -1: cut = min(cut, i) return [ {"role": "system", "content": prompt[:cut].rstrip()}, {"role": "user", "content": prompt[cut:].lstrip("\n")}, ]
Some design choices worth copying:
The model sees exactly the same text, in the same order. Only the boundary moves. That keeps the change easy to review and easy to A/B test.
Gate it by model. Requests to models outside the list stay byte-for-byte identical, so older deployments are untouched:
SPLIT_MODELS = set(filter(None, os.getenv( "PROMPT_SPLIT_MODELS", "gpt-5.6-terra,gpt-5.6-luna,gpt-5.6-sol").split(",")))messages = split_prompt_messages(prompt) if model in SPLIT_MODELS else [{"role": "system", "content": prompt}]
On Azure these are deployment names, which can differ between environments. That’s why the list lives in an environment variable.
Leave small prompts alone. Below 1,024 tokens no model caches anything, so splitting gains nothing.
A common half-fix
The obvious first move is to put only the user’s latest input in a user message and leave the conversation history in system. On the first request of a session, when there's no history yet, this looks great. On every later request, the history makes system unique again, and you're back to a prefix that never repeats.
In our replay, multi-turn requests with the history still in system got 21% of their input from cache. With the history moved to user, they got 84%.
- Impact
We replayed 1,000 real production requests against GPT-5.6 Terra with the prompt split this way:
First request of a session (~9.5k-token prompt): 98% of input tokens from cache.
Multi-turn requests (~5.4k-token prompt): 84%.
Weighted across all calls: 89%, compared with 91% for GPT-4o on the same traffic.
What surprised us:
Few distinct system messages. Across the replay there were only a few dozen distinct system messages, one per variant of the reference data. Low cardinality is what makes the cache work.
Capacity. On provisioned throughput, cached tokens count much less against the peak. Compared with the half-fix, the prompt tokens that counted at peak fell by more than half, and our capacity estimate by about 40%. That put GPT-5.6 on the same capacity as the GPT-4o deployment it replaces.
Latency didn’t change. Median latency was about 1.7s with or without cache. On our Standard deployment, caching saved cost and capacity, not time.
5.1 A rough cost illustration
At GPT-5.6 Terra list prices (input $2.00/M, cache read $0.20/M, cache write 1.25× = $2.50/M), for one 9.5k-token call:
No caching at all: about $0.019 (9.5k × $2.00/M).
Single message on GPT-5.6: about $0.024. Every call writes 9.5k tokens at 1.25×, and nothing is ever read back.
Split prompt on GPT-5.6, 98% cached: about $0.0023 (9.3k × $0.20/M plus a small written tail).
The single-message layout isn’t just “not cached”: it’s ~25% more expensive than no caching, and about 10× more expensive than the split prompt. Your exact numbers depend on your deployment type and contract.
- Don’t skip the quality check
Moving content from system to user changes how the model reads it. The model may treat history as something the user just said rather than as background context. In our replay the change in task accuracy was small and not statistically significant, but we're still validating it on a larger sample before rolling out.
If you apply this pattern, measure quality, not just cache. A cheaper wrong answer is not a win.
- Going further
We kept the first change minimal on purpose, to isolate its effect. There are more levers.
7.1 Explicit cache breakpoints
In implicit mode, the final breakpoint sits at the end of the last message, so the variable tail is also written to the cache at 1.25×, and it will never be read. Explicit mode lets you choose exactly where to write:
body = { "model": "gpt-5.6-terra", "prompt_cache_options": {"mode": "explicit", "ttl": "30m"}, "messages": [ {"role": "system", "content": instructions}, {"role": "user", "content": [ {"type": "text", "text": reference_data, "prompt_cache_breakpoint": {"mode": "explicit"}}, ]}, {"role": "user", "content": variable_part}, # after the breakpoint: never written to cache ],}
Gotchas:
The prefix up to the breakpoint must be ≥ 1,024 tokens.
Anything variable must come after the breakpoint.
Models older than 5.6 return HTTP 400 for these parameters. Only send them to 5.6 deployments.
Some provisioned deployment types (PTU-M on Azure, at the time of writing) don’t support breakpoints.
7.2 prompt_cache_key
Caches live on specific machines. Requests are routed by a hash of the start of the prompt, and when many requests share a long prefix they can spread across machines and miss each other’s caches. prompt_cache_key is a routing hint: requests with the same key are more likely to reach the same cache.
A natural key is “which static prompt is this”:
body = { "model": "gpt-5.6-terra", "prompt_cache_options": {"mode": "explicit", "ttl": "30m"}, "messages": [ {"role": "system", "content": instructions}, {"role": "user", "content": [ {"type": "text", "text": reference_data, "prompt_cache_breakpoint": {"mode": "explicit"}}, ]}, {"role": "user", "content": variable_part}, # after the breakpoint: never written to cache ],}
Keep the cardinality low (one key per distinct system message, not per call), and never put user data in it.
7.3 Measure the writes, not just the reads
Most dashboards track usage.prompt_tokens_details.cached_tokens. On GPT-5.6 there's also cache_write_tokens, and on a badly split prompt that's where the money goes. If your cost or capacity estimate subtracts cached tokens but doesn't add the 1.25× write premium, it's too optimistic.
- Checklist before you switch to GPT-5.6
Find single-message prompts where static and variable text share a message.
Move every variable byte after the last static message boundary: history, timestamps, IDs, session context, the user’s input.
Count distinct system messages in a sample. Dozens is fine. Thousands means something variable is still in there.
Gate the change by model so older deployments stay byte-for-byte identical.
Replay real traffic and compare cache %, latency and task quality on the same requests.
Track cache_write_tokens next to cached_tokens.
Then consider explicit breakpoints and prompt_cache_key.
The lesson we took away: with GPT-5.6, how your prompt is split into messages affects your bill. The model reads the same text either way, but the cache only sees message boundaries. Put the stable text at a boundary, and the cache will find it.