{"slug": "prompt-caching-worked-perfectly-one-dynamic-timestamp-destroyed-every-cache-hit", "title": "Prompt Caching Worked Perfectly. One Dynamic Timestamp Destroyed Every Cache Hit.", "summary": "A production engineering analysis found that a single dynamic timestamp in a prompt destroyed every cache hit for a local coding agent, despite 99% identical prompts. The article, from The Production Engineering Library, explains that prefix caching requires byte-identical prefixes, so dynamic content like timestamps placed before stable sections prevents reuse, and recommends structuring prompts with stable sections first and deterministic serialization.", "body_md": "The local coding agent had the ideal workload for prompt caching.\n\nEvery request began with the same system instructions, the same tool definitions, and the same repository map. Only the final user message changed.\n\nThe cache was enabled. Memory was allocated. Metrics showed almost no reuse.\n\nNear the top of the prompt sat one harmless line:\n\n```\nCurrent time: 2026-08-30T21:14:07Z\n```\n\nTo a person, the prompts were 99% identical. To a prefix cache, they diverged before most of the expensive tokens began.\n\nLarge prompts have a costly prefill phase. The model must process the input and construct KV-cache state before generating the answer.\n\nWhen several requests share an identical prefix, a serving engine can reuse cached blocks rather than recompute them. This is particularly attractive for agents because system prompts, policies, tool schemas, coding conventions, and repository summaries often repeat.\n\nThe word “identical” does most of the work.\n\nPrefix caching is not semantic similarity. Two prompts that mean the same thing but differ in ordering, whitespace, serialization, or an early dynamic value may stop sharing cacheable blocks from the first changed region onward.\n\nThe server can implement caching correctly while the application defeats it on every request.\n\nTimestamps are only one source of accidental variation.\n\nAgent systems commonly inject:\n\nIf dynamic content appears before a long stable section, the stable section may no longer share the same prefix position.\n\nA cache-aware prompt structure looks more like:\n\n```\n[stable policy][stable tool schemas in deterministic order][stable project instructions][stable or versioned repository map][session-specific context][current user request]\n```\n\nPut unavoidable variation after the largest reusable prefix. Serialize deterministically. Version stable blocks intentionally rather than allowing incidental changes.\n\nReusing a huge prefix is not automatically efficient.\n\nCached KV blocks consume memory. Keeping many project prefixes can reduce the room available for active sequences. Low-reuse entries may be evicted before their second use. A stale repository map can also make the agent faster at using yesterday’s truth.\n\nMeasure the complete effect:\n\nThe useful cache is not the one with the highest hit percentage. It is the one that reduces important latency without causing memory pressure or stale context failures.\n\nPrompt builders often live as ordinary string concatenation scattered across application code. That makes cache behavior difficult to reason about.\n\nGive the prompt structure a version. Log a hash of stable sections without logging secrets. Track how many prefix tokens each request reused. Alert when a deployment collapses the hit rate.\n\nA simple regression test can render the prefix twice with different request metadata and assert that the stable bytes remain identical.\n\nAlso test ordering. JSON object properties, tool arrays, file lists, and retrieved documents should be deterministic when their meaning has not changed.\n\nThe operational lesson is larger than prompt caching. Features fail at boundaries between components: the server promises reuse, but the application determines whether reuse is possible.\n\n[ The Production Engineering Library: The Complete Collection for Production Systems, System Design & Backend Engineering](https://yusufseyitoglu.gumroad.com/l/production-engineering-library) is built around finding those boundaries — the invisible choices that turn healthy-looking infrastructure into latency, memory pressure, wrong results, and difficult incidents.\n\nStart with one request trace and divide its input into stable, versioned, session-specific, and request-specific sections.\n\nThen make three decisions:\n\nDo not put “current time” into every request unless the task requires it. When time matters, place it near the end. Do the same with identifiers and volatile environment details.\n\nFor repository context, consider a stable map keyed by a commit or index version. A real project change should invalidate that block. A new trace ID should not.\n\nFinally, benchmark both cold and warm paths. Cold performance determines recovery after restart or eviction. Warm performance determines the steady state. A system that looks fast only after a hand-prepared cache is not ready for production.\n\nThe timestamp did not break caching because timestamps are dangerous.\n\nIt broke caching because a dynamic byte was allowed to sit in front of expensive stable work.\n\nOnce prompt construction is treated as part of the serving system, that class of failure becomes visible — and preventable.\n\nThere is a security boundary here too. Prefixes shared across users or tenants must never allow one caller’s private context to become another caller’s cached input. Cache keys need to include the authorization and isolation dimensions that affect visibility, not merely a text hash. A fast cross-tenant cache hit is a data leak.\n\nInvalidation must be explicit. Model revision, tokenizer, chat template, tool schema, policy version, and repository revision can all change the meaning of cached state. Include relevant versions in the key or purge the cache when they change. Reusing incompatible state is worse than missing the cache because it can create confident, difficult-to-reproduce errors.\n\nDuring rollout, graph hit rate beside time to first token and memory pressure. If hits rise while latency does not improve, the reused block may be too small or another phase dominates. If latency improves while evictions destabilize long requests, the cache needs a tighter budget. Success is the service objective, not the feature metric.\n\nFroquiz lets you practice designing systems where caching, invalidation, observability, and changing data must coexist under real constraints.\n\n→ **Build a System Design interview architecture on Froquiz**\n\n[Prompt Caching Worked Perfectly. One Dynamic Timestamp Destroyed Every Cache Hit.](https://blog.stackademic.com/prompt-caching-worked-perfectly-one-dynamic-timestamp-destroyed-every-cache-hit-0dec7dca49bd) was originally published in [Stackademic](https://blog.stackademic.com) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/prompt-caching-worked-perfectly-one-dynamic-timestamp-destroyed-every-cache-hit", "canonical_source": "https://blog.stackademic.com/prompt-caching-worked-perfectly-one-dynamic-timestamp-destroyed-every-cache-hit-0dec7dca49bd?source=rss----d1baaa8417a4---4", "published_at": "2026-09-03 11:25:18+00:00", "updated_at": "2026-09-03 11:51:53.447234+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "large-language-models"], "entities": ["The Production Engineering Library"], "alternates": {"html": "https://wpnews.pro/news/prompt-caching-worked-perfectly-one-dynamic-timestamp-destroyed-every-cache-hit", "markdown": "https://wpnews.pro/news/prompt-caching-worked-perfectly-one-dynamic-timestamp-destroyed-every-cache-hit.md", "text": "https://wpnews.pro/news/prompt-caching-worked-perfectly-one-dynamic-timestamp-destroyed-every-cache-hit.txt", "jsonld": "https://wpnews.pro/news/prompt-caching-worked-perfectly-one-dynamic-timestamp-destroyed-every-cache-hit.jsonld"}}