{"slug": "your-multi-agent-system-isn-t-hitting-prompt-cache-your-system-prompt-is-the", "title": "Your multi-agent system isn't hitting prompt cache. Your system prompt is the reason.", "summary": "A developer discovered that their multi-agent system was failing to hit prompt caches because the system prompt, which varied per agent, was placed before the shared context, causing the token prefix to diverge immediately. By reordering the prompt to put the shared context first and the persona last, they achieved significant cache hits and cost savings. The fix is presented as a general design principle: order prompts from most shared to most specific.", "body_md": "I run a multi-agent setup where ten agents analyse the same input. Same document, same market data, same everything. The only difference between them is persona: each one is instructed to look at the material through a different lens.\n\nTen agents, one shared context. That should be the ideal case for prompt caching. Send the expensive context once, pay full price for it once, and let the other nine reads come back at a fraction of the cost.\n\nMy cache hit rate was zero percent on three of the five models I was using, and under seven percent on the other two.\n\nI had been reading the bill for a while and optimising the wrong thing. Here is what was actually happening, because the mistake is structural and I doubt I am the only one making it.\n\nHosted inference providers cache on a **prefix**. The provider hashes your request from the first token forward and looks for the longest run it has already computed. If your request starts with the same 3,000 tokens as a recent one, those 3,000 tokens are a cache read, typically around five times cheaper than a fresh read. The moment the token stream diverges, caching stops for the rest of the request. There is no re-syncing later.\n\nThat word — prefix — is doing all the work, and I had not thought about it carefully.\n\nMy call looked like every example in every SDK doc:\n\n```\nmessages = [\n    {\"role\": \"system\", \"content\": agent_persona},   # differs per agent\n    {\"role\": \"user\",   \"content\": shared_context},  # identical for all 10\n]\n```\n\nThe persona is short. A couple of hundred tokens describing how this particular agent should reason. The shared context is large: several thousand tokens of source material.\n\nRead that message array as a flat token stream, which is what the provider does. The first thing in the stream is the persona. The persona is **different for every agent**. So the prefix diverges at roughly token one, and the several thousand tokens of identical context sitting behind it can never match anything.\n\nTen agents. Ten identical copies of the same context. Ten full-price reads.\n\nI did not want to guess, so I hashed both halves of every call for a single work item and counted the distinct values.\n\n```\nSELECT\n  COUNT(DISTINCT prompt_sha256) AS distinct_user,\n  COUNT(DISTINCT system_sha256) AS distinct_system\nFROM agent_calls\nWHERE item_id = ?\n```\n\nThe answer:\n\n```\ndistinct_user   = 1\ndistinct_system = 10\n```\n\nOne user prompt. Ten system prompts. The expensive half was **byte-identical across all ten calls**, and the cheap half in front of it was unique every time.\n\nThis lines up exactly with the provider's own usage report, which broke my spend into cached and uncached input tokens:\n\n| model | cached share of input |\n|---|---|\n| A | 0.0% |\n| B | 0.0% |\n| C | 3.4% |\n| D | 6.2% |\n| E | 11.0% |\n\nThose low non-zero numbers are incidental collisions between unrelated calls, not the structural reuse I should have been getting. If the design were right, nine out of every ten context reads would be cache hits.\n\nPut the shared, expensive, identical part first. Put the small, varying part last.\n\n```\nmessages = [\n    {\"role\": \"system\", \"content\": shared_context},          # identical -> caches\n    {\"role\": \"user\",   \"content\": f\"{agent_persona}\\n\\n{question}\"},  # varies, small, last\n]\n```\n\nNow the first several thousand tokens are the same for all ten agents. The first agent pays full price and warms the cache. The other nine read it back at cache rates. The only uncached part is the couple of hundred persona tokens at the tail, which is what you actually want to be paying for.\n\nThe general rule, which I now think should be a design constraint rather than an optimisation:\n\nOrder your prompt from\n\nmost sharedtomost specific. Caching rewards a stable prefix, and every byte that varies early poisons everything after it.\n\nThis also composes with how you batch. If you run the same model across many items back to back, you keep hitting a warm prefix. If you round-robin across models for each item, you cold-start the cache on every single call. Grouping by model, not by work item, keeps the cache warm.\n\nMoving the persona out of `system`\n\nand into `user`\n\nis not free.\n\nSome models weight system instructions more strongly than user content. That is often the point of a system prompt. If one of my agents is specifically instructed to argue an unpopular position, and I demote that instruction from system to user, it may hedge more. I would be trading spend for behaviour, and I would not necessarily notice, because the output would still be well-formed and plausible.\n\nSo this is not a change I would ship straight to production off the back of a cost argument. It needs an A/B on a sample of items, comparing the actual decisions each layout produces, not just checking that the responses parse.\n\nThere is a middle path worth trying first: keep a short stable instruction in `system`\n\nthat is **identical across all agents**, and move only the per-agent differentiation into the user message. You get a shared prefix and keep a system-role framing. Whether that is enough depends on how much of your agents' behaviour hangs off the system role, which is an empirical question about your prompts and your models.\n\nI had spent real effort choosing cheaper models before I checked whether I was paying for the same tokens ten times over. The model swap was worth doing. It was also the second-biggest lever, and I found it first because it was the one I was looking for.", "url": "https://wpnews.pro/news/your-multi-agent-system-isn-t-hitting-prompt-cache-your-system-prompt-is-the", "canonical_source": "https://dev.to/rickeshtn/your-multi-agent-system-isnt-hitting-prompt-cache-your-system-prompt-is-the-reason-4gb2", "published_at": "2026-08-11 21:49:39+00:00", "updated_at": "2026-08-11 22:17:38.704431+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/your-multi-agent-system-isn-t-hitting-prompt-cache-your-system-prompt-is-the", "markdown": "https://wpnews.pro/news/your-multi-agent-system-isn-t-hitting-prompt-cache-your-system-prompt-is-the.md", "text": "https://wpnews.pro/news/your-multi-agent-system-isn-t-hitting-prompt-cache-your-system-prompt-is-the.txt", "jsonld": "https://wpnews.pro/news/your-multi-agent-system-isn-t-hitting-prompt-cache-your-system-prompt-is-the.jsonld"}}