{"slug": "gpt-5-6-quietly-broke-our-prompt-cache-one-message-boundary-fixed-it", "title": "GPT-5.6 Quietly Broke Our Prompt Cache. One Message Boundary Fixed It.", "summary": "A model upgrade to GPT-5.6 (Sol, Terra, Luna) dropped one team's prompt cache hit rate from roughly 91% to near zero because the new caching model only stops at the end of an eligible message, not at fixed 128-token intervals as GPT-4o did. Moving all variable text (history, session context, user input) out of the system message and into a user message restored the share of input tokens served from cache to 89% across a replay of 1,000 real production requests, and cut the provisioned capacity needed by about 40% versus a half-fix. GPT-5.6 also charges a 1.25× cache write on every call while cache reads cost 0.1× the input price, so a single-message prompt that ends in changing text pays the write premium and never gets a hit.", "body_md": "How a model upgrade took our cache hit rate from ~90% to almost nothing, and why the fix was about where our prompt text sits across messages, not what it says.\n\nTL;DR\n\nGPT-4o caches any identical prefix in 128-token steps. It doesn’t care where your messages begin or end.\n\nGPT-5.6 (Sol, Terra, Luna) can only stop the cache at the end of an eligible message. A prefix that ends in the middle of a message does not count.\n\nIf you send one big system message with the variable text at the end, the only place the cache can stop is after the text that changes on every request. On GPT-5.6 you then pay a 1.25× cache write on every call and never read any of it back.\n\nThe fix: static text in system, everything variable in user. In a replay of 1,000 real production requests, the share of input tokens served from cache went back to 89%, on par with GPT-4o. The provisioned capacity we needed dropped by about 40% compared with a half-fix.\n\nNext steps: explicit cache breakpoints and prompt_cache_key.\n\n1. The pattern that used to be fine\n\nMany LLM applications build one long prompt and send it as a single system message:\n\nInstructions (fixed)\n\nReference data: rules, catalogues, examples (fixed, or one of a few variants)\n\nConversation history or session context (varies per request)\n\nThe latest user input (varies per request)\n\n```\nprompt = instructions + reference_data + history + f\"\\nUser input: {user_input}\"\nresponse = client.chat.completions.create(    model=deployment,    messages=[{\"role\": \"system\", \"content\": prompt}],)\n```\n\nOur prompt looked exactly like this, at about 9,500 tokens. On GPT-4o it was cheap: around 91% of input tokens came from cache.\n\n2. Why it worked on GPT-4o\n\nGPT-4o’s prompt caching is prefix-based at fixed intervals:\n\nCaching only applies once a prompt has at least 1,024 tokens, and those first 1,024 must be identical.\n\nAfter that, the cached prefix grows in 128-token blocks, counted from the start of the prompt.\n\nMessage boundaries don’t matter. The provider places the cut points for you.\n\nSo parts 1 and 2 (identical on every call) were cached no matter what followed. Only the variable tail missed the cache.\n\n3. What changed in GPT-5.6\n\nGPT-5.6 shipped with a new caching model. The key sentence in the docs says where the cache is allowed to look for a match:\n\nthe implicit breakpoint, up to 20 earlier eligible message endings, and the endpoint of the initial consecutive block of developer messages\n\n“Eligible message endings” are user messages, the last tool response in a group, and the last developer/system message in the initial block. Compared with earlier models:\n\nWhere the cache can stop. GPT-4o to 5.5: at fixed 128-token intervals. GPT-5.6: only at the end of an eligible message, or at an explicit breakpoint.\n\nA prefix that ends mid-message. GPT-4o to 5.5: counts. GPT-5.6: doesn’t count.\n\nCache write. GPT-4o to 5.5: free. GPT-5.6: 1.25× the input price.\n\nCache read. GPT-4o to 5.5: discounted. GPT-5.6: 0.1× the input price.\n\nNow apply those rules to a single-message prompt. There is exactly one eligible ending: the end of the system message, which comes after the user's input. Every request writes a cache entry that ends in text no other request will repeat. The next request looks for that prefix, doesn't find it, and writes its own.\n\nResult: you pay a premium to write to the cache on every call and never get a hit.\n\nThe model didn’t get worse. The prompt was built on an assumption, “the cache follows the bytes”, that stopped being true.\n\n4. The fix: every variable byte goes in user\n\nThe rule is simple:\n\nThe system message must be identical for every request that should share a cache entry. Anything that varies per call, such as history, session context or the user's input, goes after it.\n\n```\nmessages = [    {\"role\": \"system\", \"content\": instructions + reference_data},    {\"role\": \"user\", \"content\": history + f\"\\nUser input: {user_input}\"},]\n```\n\nIf your code already builds one string, you can split it at the first variable section without touching the prompt templates:\n\n```\nVARIABLE_SECTION_MARKERS = (\"\\n---\\nConversation history:\", \"\\n---\\nSession context:\")USER_INPUT_MARKER = \"\\nUser input: \"\nphp\ndef split_prompt_messages(prompt: str) -> list[dict]:    \"\"\"Split a single prompt into a static system message and a variable user message.\"\"\"    input_at = prompt.rfind(USER_INPUT_MARKER)    if input_at == -1:        return [{\"role\": \"system\", \"content\": prompt}]    cut = input_at    for marker in VARIABLE_SECTION_MARKERS:        i = prompt.find(marker, 0, input_at)        if i != -1:            cut = min(cut, i)    return [        {\"role\": \"system\", \"content\": prompt[:cut].rstrip()},        {\"role\": \"user\", \"content\": prompt[cut:].lstrip(\"\\n\")},    ]\n```\n\nSome design choices worth copying:\n\nThe model sees exactly the same text, in the same order. Only the boundary moves. That keeps the change easy to review and easy to A/B test.\n\nGate it by model. Requests to models outside the list stay byte-for-byte identical, so older deployments are untouched:\n\n```\nSPLIT_MODELS = set(filter(None, os.getenv(    \"PROMPT_SPLIT_MODELS\", \"gpt-5.6-terra,gpt-5.6-luna,gpt-5.6-sol\").split(\",\")))messages = split_prompt_messages(prompt) if model in SPLIT_MODELS else [{\"role\": \"system\", \"content\": prompt}]\n```\n\nOn Azure these are deployment names, which can differ between environments. That’s why the list lives in an environment variable.\n\nLeave small prompts alone. Below 1,024 tokens no model caches anything, so splitting gains nothing.\n\nA common half-fix\n\nThe obvious first move is to put only the user’s latest input in a user message and leave the conversation history in system. On the first request of a session, when there's no history yet, this looks great. On every later request, the history makes system unique again, and you're back to a prefix that never repeats.\n\nIn our replay, multi-turn requests with the history still in system got 21% of their input from cache. With the history moved to user, they got 84%.\n\n5. Impact\n\nWe replayed 1,000 real production requests against GPT-5.6 Terra with the prompt split this way:\n\nFirst request of a session (~9.5k-token prompt): 98% of input tokens from cache.\n\nMulti-turn requests (~5.4k-token prompt): 84%.\n\nWeighted across all calls: 89%, compared with 91% for GPT-4o on the same traffic.\n\nWhat surprised us:\n\nFew distinct system messages. Across the replay there were only a few dozen distinct system messages, one per variant of the reference data. Low cardinality is what makes the cache work.\n\nCapacity. On provisioned throughput, cached tokens count much less against the peak. Compared with the half-fix, the prompt tokens that counted at peak fell by more than half, and our capacity estimate by about 40%. That put GPT-5.6 on the same capacity as the GPT-4o deployment it replaces.\n\nLatency didn’t change. Median latency was about 1.7s with or without cache. On our Standard deployment, caching saved cost and capacity, not time.\n\n5.1 A rough cost illustration\n\nAt GPT-5.6 Terra list prices (input $2.00/M, cache read $0.20/M, cache write 1.25× = $2.50/M), for one 9.5k-token call:\n\nNo caching at all: about $0.019 (9.5k × $2.00/M).\n\nSingle message on GPT-5.6: about $0.024. Every call writes 9.5k tokens at 1.25×, and nothing is ever read back.\n\nSplit prompt on GPT-5.6, 98% cached: about $0.0023 (9.3k × $0.20/M plus a small written tail).\n\nThe single-message layout isn’t just “not cached”: it’s ~25% more expensive than no caching, and about 10× more expensive than the split prompt. Your exact numbers depend on your deployment type and contract.\n\n6. Don’t skip the quality check\n\nMoving content from system to user changes how the model reads it. The model may treat history as something the user just said rather than as background context. In our replay the change in task accuracy was small and not statistically significant, but we're still validating it on a larger sample before rolling out.\n\nIf you apply this pattern, measure quality, not just cache. A cheaper wrong answer is not a win.\n\n7. Going further\n\nWe kept the first change minimal on purpose, to isolate its effect. There are more levers.\n\n7.1 Explicit cache breakpoints\n\nIn implicit mode, the final breakpoint sits at the end of the last message, so the variable tail is also written to the cache at 1.25×, and it will never be read. Explicit mode lets you choose exactly where to write:\n\n```\nbody = {    \"model\": \"gpt-5.6-terra\",    \"prompt_cache_options\": {\"mode\": \"explicit\", \"ttl\": \"30m\"},    \"messages\": [        {\"role\": \"system\", \"content\": instructions},        {\"role\": \"user\", \"content\": [            {\"type\": \"text\", \"text\": reference_data, \"prompt_cache_breakpoint\": {\"mode\": \"explicit\"}},        ]},        {\"role\": \"user\", \"content\": variable_part},  # after the breakpoint: never written to cache    ],}\n```\n\nGotchas:\n\nThe prefix up to the breakpoint must be ≥ 1,024 tokens.\n\nAnything variable must come after the breakpoint.\n\nModels older than 5.6 return HTTP 400 for these parameters. Only send them to 5.6 deployments.\n\nSome provisioned deployment types (PTU-M on Azure, at the time of writing) don’t support breakpoints.\n\n7.2 prompt_cache_key\n\nCaches live on specific machines. Requests are routed by a hash of the start of the prompt, and when many requests share a long prefix they can spread across machines and miss each other’s caches. prompt_cache_key is a routing hint: requests with the same key are more likely to reach the same cache.\n\nA natural key is “which static prompt is this”:\n\n```\nbody = {    \"model\": \"gpt-5.6-terra\",    \"prompt_cache_options\": {\"mode\": \"explicit\", \"ttl\": \"30m\"},    \"messages\": [        {\"role\": \"system\", \"content\": instructions},        {\"role\": \"user\", \"content\": [            {\"type\": \"text\", \"text\": reference_data, \"prompt_cache_breakpoint\": {\"mode\": \"explicit\"}},        ]},        {\"role\": \"user\", \"content\": variable_part},  # after the breakpoint: never written to cache    ],}\n```\n\nKeep the cardinality low (one key per distinct system message, not per call), and never put user data in it.\n\n7.3 Measure the writes, not just the reads\n\nMost dashboards track usage.prompt_tokens_details.cached_tokens. On GPT-5.6 there's also cache_write_tokens, and on a badly split prompt that's where the money goes. If your cost or capacity estimate subtracts cached tokens but doesn't add the 1.25× write premium, it's too optimistic.\n\n8. Checklist before you switch to GPT-5.6\n\nFind single-message prompts where static and variable text share a message.\n\nMove every variable byte after the last static message boundary: history, timestamps, IDs, session context, the user’s input.\n\nCount distinct system messages in a sample. Dozens is fine. Thousands means something variable is still in there.\n\nGate the change by model so older deployments stay byte-for-byte identical.\n\nReplay real traffic and compare cache %, latency and task quality on the same requests.\n\nTrack cache_write_tokens next to cached_tokens.\n\nThen consider explicit breakpoints and prompt_cache_key.\n\nThe lesson we took away: with GPT-5.6, how your prompt is split into messages affects your bill. The model reads the same text either way, but the cache only sees message boundaries. Put the stable text at a boundary, and the cache will find it.", "url": "https://wpnews.pro/news/gpt-5-6-quietly-broke-our-prompt-cache-one-message-boundary-fixed-it", "canonical_source": "https://pub.towardsai.net/gpt-5-6-quietly-broke-our-prompt-cache-one-message-boundary-fixed-it-f71e08aa4c18?source=rss----98111c9905da---4", "published_at": "2026-09-24 12:31:01+00:00", "updated_at": "2026-09-24 13:06:47.400189+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "mlops", "ai-tools"], "entities": ["GPT-5.6", "GPT-4o", "Sol", "Terra", "Luna", "prompt_cache_key"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/gpt-5-6-quietly-broke-our-prompt-cache-one-message-boundary-fixed-it", "markdown": "https://wpnews.pro/news/gpt-5-6-quietly-broke-our-prompt-cache-one-message-boundary-fixed-it.md", "text": "https://wpnews.pro/news/gpt-5-6-quietly-broke-our-prompt-cache-one-message-boundary-fixed-it.txt", "jsonld": "https://wpnews.pro/news/gpt-5-6-quietly-broke-our-prompt-cache-one-message-boundary-fixed-it.jsonld"}}