{"slug": "measuring-llm-prefix-caching-the-cache-hit-rate-metric", "title": "Measuring LLM Prefix Caching: The Cache Hit Rate Metric", "summary": "An engineer's benchmarking guide introduces a cache hit rate metric for measuring prefix caching effectiveness in LLM serving, implemented in the open-source tool llmperf-rs. The metric calculates the ratio of cached tokens to total tokens from non-final turns, highlighting its importance for agentic workloads where multi-turn requests re-send history. The guide notes that explicit prompt caching mechanisms like Anthropic's cache_control differ from automatic prefix caching, and that the metric only measures the automatic kind.", "body_md": "Prefix caching is one of the biggest cost levers in LLM serving. vLLM, SGLang, TGI, and most hosted providers all do some version of it: during prefill they compute a key-value (KV) cache, and if a later request shows up with the same prompt prefix, they reuse that cache instead of recomputing it. Done well, a lot of expensive prefill compute turns into a cheap cache lookup.\n\nWhether it helps depends on how much of your traffic re-sends the same prefix, and most benchmarking runs don't tell you. This is part of my [LLM benchmarking guide](https://wheynelau.dev/posts/2025-12-15-benchmarking-performance/). Here I want to focus on how to actually measure cache effectiveness: the metric, why it matters for agentic workloads, and the cost angle.\n\nDuring prefill, the server computes the KV cache for the input prompt. If a later request sends a prefix the server has already seen, it can skip recomputing that part and just serve it from cache. The classic example is a multi-turn conversation: each turn re-sends the entire prior history, and ideally everything except the newest user message comes back from cache.\n\nThis is also why cache reuse only shows up in multi-turn or shared-prefix workloads. A single isolated request has nothing to reuse. Turn 0 is always a cold start. So if you want to measure caching, you have to re-send history, which means multi-turn requests.\n\nA chat conversation re-sends some history every turn, but an agentic coding loop does this on fast-forward, and at a scale where caching stops being optional and starts dominating both latency and cost.\n\nHere's how a coding agent actually runs (Claude Code, Cursor, Cline, that sort of thing). It loops: read the task, decide on an action, call a tool to read a file or run a command, get the result back, decide the next action, call another tool. Each one of those iterations is a new API request, and every request re-sends the *entire accumulated context*. The system prompt, the original task, all the prior reasoning, every previous tool call and its result. The only genuinely new content is the latest tool result and the model's next decision. Everything before that is a prefix the server has already computed.\n\nSo an agentic session is really just a long multi-turn conversation where history gets re-sent every turn, which is exactly the case `cache_hit_rate`\n\nwas built for. If you're picking a serving setup for agentic workloads, cache hit rate under a realistic multi-turn load is one of the most telling numbers you can collect.\n\nOne nuance worth knowing: some providers also offer *explicit* prompt caching, where the client marks cache breakpoints (Anthropic's `cache_control`\n\nis the example). That's a different mechanism from the automatic prefix caching most OpenAI-compatible endpoints do, and llmperf-rs only measures the automatic kind. For a standard tool-call loop against a vLLM-style endpoint, automatic prefix caching is what applies.\n\nThe metric I use measures cache reuse against the content that was *previously sent*, not the whole request. Caching only reuses what the server has already seen: the assistant's prior outputs and earlier user prompts that get echoed back in the next request. New tokens in the current turn can never be cached, because the server hasn't seen them before.\n\nImplemented in [llmperf-rs](https://github.com/wheynelau/llmperf-rs):\n\n```\ncache_hit_rate = sum(cached_tokens) / sum(total_tokens_of_non_final_turns)\n```\n\n`cached_tokens`\n\nreported by the endpoint on each turn, read from `prompt_tokens_details.cached_tokens`\n\nin the streamed `usage`\n\nobject. `None`\n\nmeans the endpoint didn't report the field.100% means every previously-sent token came back from cache. In a perfect cache, `cached_tokens`\n\nequals the prior-turn total on every warm turn.\n\nA few that bite in practice:\n\n`None`\n\n.`None`\n\nrun reports `None`\n\n.`cached_tokens`\n\n, you get `None`\n\n, not zero. That's deliberate: `None`\n\nmeans \"not measurable\", which is different from `0.0`\n\n(a cache that's just never hit).`cached_tokens`\n\nand others don't, the unobserved turns are left out of the numerator but their re-sent history still counts in the denominator. So a noisy endpoint just pulls the ratio down rather than wiping it out.`cached_tokens = 0`\n\nor just omits it, so it doesn't move the numerator either way.You need multi-turn requests, which in llmperf-rs is `--multi-turn N`\n\n:\n\n```\nexport OPENAI_API_BASE=http://localhost:8000/v1   # vLLM with prefix caching enabled\nllmperf --model Qwen/Qwen3-4B-Instruct-2507 \\\n        --multi-turn 5 \\\n        --max-num-completed-requests 10\n```\n\nThe summary then includes a `cache_hit_rate`\n\nfield (alongside the TTFT/ITL/throughput metrics covered in the [main guide](https://wheynelau.dev/posts/2025-12-15-benchmarking-performance/)):\n\n*Example value only, illustrative and not from a real run.*\n\n```\n\"cache_hit_rate\": 0.91\n```\n\nThat single summary number aggregates across the whole run. Per-turn `cached_tokens`\n\nand `turn_index`\n\nare also written to the individual-responses file if you want to see how the cache builds up over turns after the cold start.\n\nThere's a subtlety if you're benchmarking reasoning models. The common guidance is to discard a model's `reasoning_content`\n\nfrom the message history you send back, to save tokens. llmperf-rs does the opposite for multi-turn runs: it echoes the previous turn's `reasoning_content`\n\non the assistant message.\n\nThe reason is exactly this topic. Providers that support prefix caching over reasoning (Z.ai's \"Preserved thinking\" with `clear_thinking: false`\n\n, for example) can reuse the KV cache across turns only if the reasoning is re-sent. Dropping it to save on echoed-input tokens throws away the cache reuse, which usually costs more than it saves. Providers that don't understand `reasoning_content`\n\njust ignore the field, so it's safe to send.\n\nSo if you're measuring cache hit rate on a reasoning model, make sure you're re-sending the reasoning. Otherwise you're measuring a workload that disables its own cache.\n\nIf you've enabled prefix caching, cache hit rate is how you confirm it's earning its keep. The key thing to get right is the denominator: measure cache reuse against the history you re-sent, not against the whole request, or you'll understate a cache that's working fine. And remember it's strictly a multi-turn metric. A single-turn benchmark tells you nothing about caching.\n\nThe full version with the exact math and more detail is on [my blog](https://wheynelau.dev/posts/2026-08-01-measuring-llm-prefix-caching/).", "url": "https://wpnews.pro/news/measuring-llm-prefix-caching-the-cache-hit-rate-metric", "canonical_source": "https://dev.to/wheynelau/measuring-llm-prefix-caching-the-cache-hit-rate-metric-2n9m", "published_at": "2026-08-05 02:49:08+00:00", "updated_at": "2026-08-05 03:09:48.804616+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["vLLM", "SGLang", "TGI", "Anthropic", "llmperf-rs", "Claude Code", "Cursor", "Cline"], "alternates": {"html": "https://wpnews.pro/news/measuring-llm-prefix-caching-the-cache-hit-rate-metric", "markdown": "https://wpnews.pro/news/measuring-llm-prefix-caching-the-cache-hit-rate-metric.md", "text": "https://wpnews.pro/news/measuring-llm-prefix-caching-the-cache-hit-rate-metric.txt", "jsonld": "https://wpnews.pro/news/measuring-llm-prefix-caching-the-cache-hit-rate-metric.jsonld"}}