A long-running AI workflow can look healthy while a tiny change quietly makes every turn reprocess thousands of tokens. A reordered tool, a request ID tucked into instructions, a changed schema, or a fallback model can turn a reusable prefix into fresh work. The response still arrives. Your logs still show success. The bill and first-token latency are the only clues.
OpenAI Prompt Cache Diagnostics gives Responses API teams a way to stop guessing. Instead of treating a low cached-token count as a vague cost problem, you can compare one response with a recent baseline and get a classified explanation for the first detected mismatch. This guide shows how to turn that signal into a safe engineering loop: isolate the difference, make one reversible fix, and verify the result on representative traffic.
The timing matters. OpenAI’s current API changelog lists the new GPT-6 Sol and GPT-6 Luna models, while Prompt Cache Diagnostics is available for GPT-5.6 and later supported models. A model rollout is a sensible time to confirm what your own application is actually reusing, rather than assume an upgraded model or a maintained session preserves every cost-saving property you expected.
Prompt caching reuses the work already done on an unchanged beginning of a request. OpenAI describes that reusable beginning as a prefix. When later calls share it, the platform can reuse cached computation rather than processing that material again. That can reduce input cost and shorten the time before a response begins. It is especially important for agent loops that carry forward instructions, a repository map, tool definitions, policy text, and previous messages.
The trap is that cache reuse is not the same thing as “we kept the same conversation.” A conversation can be continuous while the request sent to the model has changed near its start. A framework may regenerate tools in a new order. An experiment flag may select another model. A logging helper may inject a timestamp into the instructions. A structured-output schema may gain an optional field. All are reasonable product changes. All can alter a prefix that you expected to reuse.
That makes a cache miss a reliability signal as well as a finance signal. It tells you an implementation assumption changed between comparable calls. Sometimes that change is intentional and worthwhile. A safety fallback, context compaction, or a new tool may be exactly right. The goal is not a perfect hit rate. The goal is to know whether a costly change is deliberate, bounded, and paid for with eyes open.
Prompt Cache Diagnostics is available in the Responses API for GPT-5.6 and later supported models. You choose a recent completed response from the same organization as a baseline, pass its ID through prompt_cache_options.comparison_response_id, then read prompt_cache_diagnostics on the current response. This comparison requests analysis; it does not load the earlier conversation or change the cache behavior of the new request.
The result is deliberately narrower than a request-body diff. It may be a cache_hit, a cache_miss with a reason and affected-token estimate, comparison_response_not_found, or unavailable. A hit means the comparison found no cache miss. It does not mean every input token was reused, because new input still needs processing. For actual reuse and billing analysis, inspect usage.input_tokens_details.cached_tokens on the response.
This distinction matters. Treat diagnostics as a high-quality hypothesis generator, not as a magic accounting system. OpenAI notes that diagnostics are best effort and report the first classified reason. Fix that one difference, repeat the same comparison, and you may expose the next difference behind it.
Do not enable a comparison on every request forever without deciding how you will use it. Start with a small, representative route: an agent that does several tool calls, a support flow with a large policy prefix, or a coding task that repeatedly carries repository context. Save a response ID only after a successful, completed baseline. On its next comparable turn, compare against that baseline.
from openai import OpenAI
client = OpenAI()
python
def run_agent_turn(instructions, user_input, tools, baseline_id=None): cache_options = {"mode": "implicit"} if baseline_id: cache_options["comparison_response_id"] = baseline_id
response = client.responses.create( model="gpt-6-sol", instructions=instructions, input=user_input, tools=tools, prompt_cache_options=cache_options, )
cached = response.usage.input_tokens_details.cached_tokens diagnostic = response.prompt_cache_diagnostics record = { "response_id": response.id, "cached_tokens": cached, "diagnostic_type": diagnostic.type if diagnostic else None, "miss_reason": getattr(diagnostic, "reason", None), "missed_tokens": getattr(diagnostic, "cache_missed_tokens", None), } return response, record
Keep that record small and non-sensitive. You want the response ID, route name, model, service tier, cache-read and cache-write token counts, latency, result status, and diagnostic classification. Do not copy raw prompts or customer content into a cost dashboard just because you are investigating cache reuse. OpenAI says the diagnostic records themselves contain configuration metadata, token estimates, and hashes rather than raw prompts or outputs, which makes them useful in stricter data environments too.
Use a stable business key when grouping results: route plus customer plan, workflow version, or agent configuration version. That makes a spike actionable. “Cached tokens dropped” is hard to own. “The document-review route dropped after tool manifest version 14” gives an engineer a place to look.
The fastest way to waste a week is to rewrite prompts after every miss. Instead, take the reason literally, form one narrow theory, and compare a controlled request again. The common cases are more mechanical than mysterious.
Tool definitions are part of the rendered context. Adding, removing, reordering, or editing their names, descriptions, schemas, or configuration can prevent reuse. First, serialize the supplied tools in your own telemetry and compare a content hash, count, and order. Then decide whether you truly need to mutate the tool list. If the set should remain available, keep it stable and use selection controls such as tool_choice: "none" or allowed_tools to restrict use without changing the supplied list.
For teams that add capabilities frequently, use append-only evolution where possible. Retire a tool through permissions or routing before you delete or reshuffle the definition. This is not about preserving an obsolete contract forever; it is about making the cache cost of a contract change intentional and measurable.
This is the classic prefix-drift bug. Look for timestamps, request IDs, randomly ordered metadata, personalized inserts, edited prior messages, or regenerated tool results near the beginning of the request. Put stable instructions, fixed policy text, and unchanged definitions first. Put the new user message, current clock, and request-specific metadata after the reusable prefix and its cache breakpoint.
Do not “fix” this by lying to the model about current information. Separate stable context from fresh context. A current time is useful when the task needs it; it just belongs in the fresh tail, not in the static system material.
These reasons often expose an invisible platform policy. A cost router may choose another model under load. An A/B test may take a different path. The returned service tier can differ from the requested tier. Add the actual model and returned tier to the same event as the diagnostic result. Then set an explicit expectation for each route: should comparable calls stay on one model and tier, or is the lower reuse an accepted cost of resilience?
A structured-output schema is an API contract, not decoration. Reasoning effort and verbosity are operational choices, not presentation-only settings. Hash these settings into a configuration version and deploy them like code. If a change is experimental, route a small cohort to the new configuration rather than mixing it into a cache-sensitive control group.
Compaction replaces earlier conversation material. It can reduce cache reuse while still lowering total input work. In this case, a falling cache-hit ratio is not automatically bad. Compare total input tokens, cached tokens, cache-write tokens, latency, task success, and answer quality before reverting. The engineering question is whether the whole turn became cheaper and more dependable, not whether one metric looks prettier.
A prefix contract is a short record of what must remain stable for a route to reuse cache. It gives product, platform, and agent teams a shared language before an expensive regression lands. Keep it beside the route configuration, not in a forgotten wiki page.
That contract also prevents a common testing mistake: comparing unrelated calls. A baseline should have an equivalent route and a prefix you genuinely expect to share. Comparing a retrieval-heavy legal question with a short product-search query may produce a technically valid diagnostic but an operationally useless conclusion.
Suppose the diagnostic reports tools_changed and 8,000 estimated missed tokens. Do not simultaneously reorder tools, rewrite instructions, switch models, alter the schema, and change your cache breakpoint. You will not know what worked, and a later regression will be impossible to explain.
Instead, create a fixture with a safe representative prefix. Run a baseline. Run a deliberately changed version that reproduces the miss. Revert only the tool order. Compare again. Then run the repaired configuration across a modest sample of realistic traffic. Watch cache reads per write, total cost, first-token latency, tool-success rate, and task-quality checks. Only after the fix holds should it become the new baseline.
A cache fix is complete when it improves a representative workload without hiding a product change the workflow actually needs.
Keep a small test suite for predictable differences. One fixture can add a timestamp to the front of instructions. Another can reorder tools. Another can update the output schema. A fourth can compact conversation history. The point is not to test OpenAI’s platform. It is to prove that your harness emits the request shape you think it emits.
One cache miss is normal. A deployment, a new conversation, an intentional fallback, or an expired comparison record can cause one. Alerts should look for a sustained shift in a comparable slice: cached tokens falling, cache-write tokens increasing, or a particular reason suddenly dominating after a configuration rollout.
Make the alert useful by attaching context. Include route, configuration version, model, actual service tier, dominant reason, approximate affected tokens, recent baseline response age, and whether task success changed. A person on call should be able to decide whether to roll back a tool manifest, a configuration experiment, accept an intentional change, or investigate a routing path.
Do not page solely on a comparison_response_not_found or unavailable result. Those mean the diagnostic comparison did not yield a conclusion. They are prompts to check baseline freshness and model support, not evidence of a cache miss. The normal response can continue working in either case.
Start with one route whose prefix is long enough to benefit from caching. For GPT-5.6 and later, OpenAI documents a 1,024-token minimum cacheable prompt length, excluding hidden system content. Pick a flow with repeatable work, not a one-shot question-answer endpoint.
This process scales well because it respects the boundary between platform behavior and application behavior. The API can identify a first mismatch. Only your team can decide whether the mismatch was an accident, an acceptable tradeoff, or a bug in its own request builder.
When a cache regression reaches production, a Slack screenshot of token use is rarely enough. The person investigating needs enough information to replay the request shape without seeing customer content. A good incident record starts with a route identifier, deployment or configuration version, selected model, returned service tier, response ID, and the age of the comparison baseline. Add total input tokens, cached tokens, cache-write tokens, first-token latency, end-to-end latency, and the final task outcome.
For a miss, capture the diagnostic type, reason, estimated comparison-reusable tokens, and estimated missed tokens. Then attach safe fingerprints of the cache-sensitive components: an instruction-template version, tool-manifest hash and order hash, output-schema version, reasoning-effort setting, verbosity setting, and conversation-compaction state. These fields are enough to tell whether a request builder or a routing policy changed, without exporting raw user messages or proprietary instructions into an observability product.
Finally, record the decision. Was the miss expected because a tool was deliberately added? Was a timestamp accidentally placed in the prefix? Did a resilience rule move traffic to another model? Did compaction lower total work even though reuse fell? A brief decision note turns the record into institutional memory. The next engineer does not have to rediscover that a particular route accepts lower reuse during a peak-load fallback or that a tool catalog must stay ordered for normal operation.
This is also where teams can keep cache work honest. A lower cached-token number without a quality regression may be acceptable. A very high cached-token number with obsolete policy context may be dangerous. The durable goal is a request contract whose performance, safety, and behavior all have an owner.
It is a Responses API feature that compares a current request with a recent completed response and reports whether it detected a cache miss, plus a classified reason when available.
No. A hit means no miss was detected against the selected comparison. New material can still be processed. Use the response usage fields, especially cached-token details, to measure actual reuse.
No. The comparison request does not load the earlier conversation or change caching behavior. It asks the platform to compare cache-sensitive request characteristics.
Changes to the model, cache key, service tier, tool definitions or order, output format, reasoning effort, verbosity, compacted context, and early input can all lead to a reported miss.
No. Some misses are intentional, including necessary context compaction, safety fallbacks, or product changes. Evaluate total cost, latency, and outcome quality before deciding whether to preserve reuse.
Use another recent completed baseline, confirm the model supports diagnostics, and continue measuring normal usage fields. Neither status proves that the current request missed cache.
OpenAI Prompt Cache Diagnostics: Find the Prefix Drift Making Your AI App Expensive was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.