The local coding agent had the ideal workload for prompt caching.
Every request began with the same system instructions, the same tool definitions, and the same repository map. Only the final user message changed.
The cache was enabled. Memory was allocated. Metrics showed almost no reuse.
Near the top of the prompt sat one harmless line:
Current time: 2026-08-30T21:14:07Z
To a person, the prompts were 99% identical. To a prefix cache, they diverged before most of the expensive tokens began.
Large prompts have a costly prefill phase. The model must process the input and construct KV-cache state before generating the answer.
When several requests share an identical prefix, a serving engine can reuse cached blocks rather than recompute them. This is particularly attractive for agents because system prompts, policies, tool schemas, coding conventions, and repository summaries often repeat.
The word “identical” does most of the work.
Prefix caching is not semantic similarity. Two prompts that mean the same thing but differ in ordering, whitespace, serialization, or an early dynamic value may stop sharing cacheable blocks from the first changed region onward.
The server can implement caching correctly while the application defeats it on every request.
Timestamps are only one source of accidental variation.
Agent systems commonly inject:
If dynamic content appears before a long stable section, the stable section may no longer share the same prefix position.
A cache-aware prompt structure looks more like:
[stable policy][stable tool schemas in deterministic order][stable project instructions][stable or versioned repository map][session-specific context][current user request]
Put unavoidable variation after the largest reusable prefix. Serialize deterministically. Version stable blocks intentionally rather than allowing incidental changes.
Reusing a huge prefix is not automatically efficient.
Cached KV blocks consume memory. Keeping many project prefixes can reduce the room available for active sequences. Low-reuse entries may be evicted before their second use. A stale repository map can also make the agent faster at using yesterday’s truth.
Measure the complete effect:
The useful cache is not the one with the highest hit percentage. It is the one that reduces important latency without causing memory pressure or stale context failures.
Prompt builders often live as ordinary string concatenation scattered across application code. That makes cache behavior difficult to reason about.
Give the prompt structure a version. Log a hash of stable sections without logging secrets. Track how many prefix tokens each request reused. Alert when a deployment collapses the hit rate.
A simple regression test can render the prefix twice with different request metadata and assert that the stable bytes remain identical.
Also test ordering. JSON object properties, tool arrays, file lists, and retrieved documents should be deterministic when their meaning has not changed.
The operational lesson is larger than prompt caching. Features fail at boundaries between components: the server promises reuse, but the application determines whether reuse is possible.
The Production Engineering Library: The Complete Collection for Production Systems, System Design & Backend Engineering is built around finding those boundaries — the invisible choices that turn healthy-looking infrastructure into latency, memory pressure, wrong results, and difficult incidents.
Start with one request trace and divide its input into stable, versioned, session-specific, and request-specific sections.
Then make three decisions:
Do not put “current time” into every request unless the task requires it. When time matters, place it near the end. Do the same with identifiers and volatile environment details.
For repository context, consider a stable map keyed by a commit or index version. A real project change should invalidate that block. A new trace ID should not.
Finally, benchmark both cold and warm paths. Cold performance determines recovery after restart or eviction. Warm performance determines the steady state. A system that looks fast only after a hand-prepared cache is not ready for production.
The timestamp did not break caching because timestamps are dangerous.
It broke caching because a dynamic byte was allowed to sit in front of expensive stable work.
Once prompt construction is treated as part of the serving system, that class of failure becomes visible — and preventable.
There is a security boundary here too. Prefixes shared across users or tenants must never allow one caller’s private context to become another caller’s cached input. Cache keys need to include the authorization and isolation dimensions that affect visibility, not merely a text hash. A fast cross-tenant cache hit is a data leak.
Invalidation must be explicit. Model revision, tokenizer, chat template, tool schema, policy version, and repository revision can all change the meaning of cached state. Include relevant versions in the key or purge the cache when they change. Reusing incompatible state is worse than missing the cache because it can create confident, difficult-to-reproduce errors.
During rollout, graph hit rate beside time to first token and memory pressure. If hits rise while latency does not improve, the reused block may be too small or another phase dominates. If latency improves while evictions destabilize long requests, the cache needs a tighter budget. Success is the service objective, not the feature metric.
Froquiz lets you practice designing systems where caching, invalidation, observability, and changing data must coexist under real constraints.
→ Build a System Design interview architecture on Froquiz
Prompt Caching Worked Perfectly. One Dynamic Timestamp Destroyed Every Cache Hit. was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.