Measuring LLM Prefix Caching: The Cache Hit Rate Metric

An engineer's benchmarking guide introduces a cache hit rate metric for measuring prefix caching effectiveness in LLM serving, implemented in the open-source tool llmperf-rs. The metric calculates the ratio of cached tokens to total tokens from non-final turns, highlighting its importance for agentic workloads where multi-turn requests re-send history. The guide notes that explicit prompt caching mechanisms like Anthropic's cache_control differ from automatic prefix caching, and that the metric only measures the automatic kind.

Prefix caching is one of the biggest cost levers in LLM serving. vLLM, SGLang, TGI, and most hosted providers all do some version of it: during prefill they compute a key-value KV cache, and if a later request shows up with the same prompt prefix, they reuse that cache instead of recomputing it. Done well, a lot of expensive prefill compute turns into a cheap cache lookup. Whether it helps depends on how much of your traffic re-sends the same prefix, and most benchmarking runs don't tell you. This is part of my LLM benchmarking guide https://wheynelau.dev/posts/2025-12-15-benchmarking-performance/ . Here I want to focus on how to actually measure cache effectiveness: the metric, why it matters for agentic workloads, and the cost angle. During prefill, the server computes the KV cache for the input prompt. If a later request sends a prefix the server has already seen, it can skip recomputing that part and just serve it from cache. The classic example is a multi-turn conversation: each turn re-sends the entire prior history, and ideally everything except the newest user message comes back from cache. This is also why cache reuse only shows up in multi-turn or shared-prefix workloads. A single isolated request has nothing to reuse. Turn 0 is always a cold start. So if you want to measure caching, you have to re-send history, which means multi-turn requests. A chat conversation re-sends some history every turn, but an agentic coding loop does this on fast-forward, and at a scale where caching stops being optional and starts dominating both latency and cost. Here's how a coding agent actually runs Claude Code, Cursor, Cline, that sort of thing . It loops: read the task, decide on an action, call a tool to read a file or run a command, get the result back, decide the next action, call another tool. Each one of those iterations is a new API request, and every request re-sends the entire accumulated context . The system prompt, the original task, all the prior reasoning, every previous tool call and its result. The only genuinely new content is the latest tool result and the model's next decision. Everything before that is a prefix the server has already computed. So an agentic session is really just a long multi-turn conversation where history gets re-sent every turn, which is exactly the case cache hit rate was built for. If you're picking a serving setup for agentic workloads, cache hit rate under a realistic multi-turn load is one of the most telling numbers you can collect. One nuance worth knowing: some providers also offer explicit prompt caching, where the client marks cache breakpoints Anthropic's cache control is the example . That's a different mechanism from the automatic prefix caching most OpenAI-compatible endpoints do, and llmperf-rs only measures the automatic kind. For a standard tool-call loop against a vLLM-style endpoint, automatic prefix caching is what applies. The metric I use measures cache reuse against the content that was previously sent , not the whole request. Caching only reuses what the server has already seen: the assistant's prior outputs and earlier user prompts that get echoed back in the next request. New tokens in the current turn can never be cached, because the server hasn't seen them before. Implemented in llmperf-rs https://github.com/wheynelau/llmperf-rs : cache hit rate = sum cached tokens / sum total tokens of non final turns cached tokens reported by the endpoint on each turn, read from prompt tokens details.cached tokens in the streamed usage object. None means the endpoint didn't report the field.100% means every previously-sent token came back from cache. In a perfect cache, cached tokens equals the prior-turn total on every warm turn. A few that bite in practice: None . None run reports None . cached tokens , you get None , not zero. That's deliberate: None means "not measurable", which is different from 0.0 a cache that's just never hit . cached tokens and others don't, the unobserved turns are left out of the numerator but their re-sent history still counts in the denominator. So a noisy endpoint just pulls the ratio down rather than wiping it out. cached tokens = 0 or just omits it, so it doesn't move the numerator either way.You need multi-turn requests, which in llmperf-rs is --multi-turn N : export OPENAI API BASE=http://localhost:8000/v1 vLLM with prefix caching enabled llmperf --model Qwen/Qwen3-4B-Instruct-2507 \ --multi-turn 5 \ --max-num-completed-requests 10 The summary then includes a cache hit rate field alongside the TTFT/ITL/throughput metrics covered in the main guide https://wheynelau.dev/posts/2025-12-15-benchmarking-performance/ : Example value only, illustrative and not from a real run. "cache hit rate": 0.91 That single summary number aggregates across the whole run. Per-turn cached tokens and turn index are also written to the individual-responses file if you want to see how the cache builds up over turns after the cold start. There's a subtlety if you're benchmarking reasoning models. The common guidance is to discard a model's reasoning content from the message history you send back, to save tokens. llmperf-rs does the opposite for multi-turn runs: it echoes the previous turn's reasoning content on the assistant message. The reason is exactly this topic. Providers that support prefix caching over reasoning Z.ai's "Preserved thinking" with clear thinking: false , for example can reuse the KV cache across turns only if the reasoning is re-sent. Dropping it to save on echoed-input tokens throws away the cache reuse, which usually costs more than it saves. Providers that don't understand reasoning content just ignore the field, so it's safe to send. So if you're measuring cache hit rate on a reasoning model, make sure you're re-sending the reasoning. Otherwise you're measuring a workload that disables its own cache. If you've enabled prefix caching, cache hit rate is how you confirm it's earning its keep. The key thing to get right is the denominator: measure cache reuse against the history you re-sent, not against the whole request, or you'll understate a cache that's working fine. And remember it's strictly a multi-turn metric. A single-turn benchmark tells you nothing about caching. The full version with the exact math and more detail is on my blog https://wheynelau.dev/posts/2026-08-01-measuring-llm-prefix-caching/ .