Prefix caching is one of the biggest cost levers in LLM serving. vLLM, SGLang, TGI, and most hosted providers all do some version of it: during prefill they compute a key-value (KV) cache, and if a later request shows up with the same prompt prefix, they reuse that cache instead of recomputing it. Done well, a lot of expensive prefill compute turns into a cheap cache lookup.
Whether it helps depends on how much of your traffic re-sends the same prefix, and most benchmarking runs don't tell you. This is part of my LLM benchmarking guide. Here I want to focus on how to actually measure cache effectiveness: the metric, why it matters for agentic workloads, and the cost angle.
During prefill, the server computes the KV cache for the input prompt. If a later request sends a prefix the server has already seen, it can skip recomputing that part and just serve it from cache. The classic example is a multi-turn conversation: each turn re-sends the entire prior history, and ideally everything except the newest user message comes back from cache.
This is also why cache reuse only shows up in multi-turn or shared-prefix workloads. A single isolated request has nothing to reuse. Turn 0 is always a cold start. So if you want to measure caching, you have to re-send history, which means multi-turn requests.
A chat conversation re-sends some history every turn, but an agentic coding loop does this on fast-forward, and at a scale where caching stops being optional and starts dominating both latency and cost.
Here's how a coding agent actually runs (Claude Code, Cursor, Cline, that sort of thing). It loops: read the task, decide on an action, call a tool to read a file or run a command, get the result back, decide the next action, call another tool. Each one of those iterations is a new API request, and every request re-sends the entire accumulated context. The system prompt, the original task, all the prior reasoning, every previous tool call and its result. The only genuinely new content is the latest tool result and the model's next decision. Everything before that is a prefix the server has already computed.
So an agentic session is really just a long multi-turn conversation where history gets re-sent every turn, which is exactly the case cache_hit_rate
was built for. If you're picking a serving setup for agentic workloads, cache hit rate under a realistic multi-turn load is one of the most telling numbers you can collect.
One nuance worth knowing: some providers also offer explicit prompt caching, where the client marks cache breakpoints (Anthropic's cache_control
is the example). That's a different mechanism from the automatic prefix caching most OpenAI-compatible endpoints do, and llmperf-rs only measures the automatic kind. For a standard tool-call loop against a vLLM-style endpoint, automatic prefix caching is what applies.
The metric I use measures cache reuse against the content that was previously sent, not the whole request. Caching only reuses what the server has already seen: the assistant's prior outputs and earlier user prompts that get echoed back in the next request. New tokens in the current turn can never be cached, because the server hasn't seen them before.
Implemented in llmperf-rs:
cache_hit_rate = sum(cached_tokens) / sum(total_tokens_of_non_final_turns)
cached_tokens
reported by the endpoint on each turn, read from prompt_tokens_details.cached_tokens
in the streamed usage
object. None
means the endpoint didn't report the field.100% means every previously-sent token came back from cache. In a perfect cache, cached_tokens
equals the prior-turn total on every warm turn.
A few that bite in practice:
None
.None
run reports None
.cached_tokens
, you get None
, not zero. That's deliberate: None
means "not measurable", which is different from 0.0
(a cache that's just never hit).cached_tokens
and others don't, the unobserved turns are left out of the numerator but their re-sent history still counts in the denominator. So a noisy endpoint just pulls the ratio down rather than wiping it out.cached_tokens = 0
or just omits it, so it doesn't move the numerator either way.You need multi-turn requests, which in llmperf-rs is --multi-turn N
:
export OPENAI_API_BASE=http://localhost:8000/v1 # vLLM with prefix caching enabled
llmperf --model Qwen/Qwen3-4B-Instruct-2507 \
--multi-turn 5 \
--max-num-completed-requests 10
The summary then includes a cache_hit_rate
field (alongside the TTFT/ITL/throughput metrics covered in the main guide):
Example value only, illustrative and not from a real run.
"cache_hit_rate": 0.91
That single summary number aggregates across the whole run. Per-turn cached_tokens
and turn_index
are also written to the individual-responses file if you want to see how the cache builds up over turns after the cold start.
There's a subtlety if you're benchmarking reasoning models. The common guidance is to discard a model's reasoning_content
from the message history you send back, to save tokens. llmperf-rs does the opposite for multi-turn runs: it echoes the previous turn's reasoning_content
on the assistant message.
The reason is exactly this topic. Providers that support prefix caching over reasoning (Z.ai's "Preserved thinking" with clear_thinking: false
, for example) can reuse the KV cache across turns only if the reasoning is re-sent. Dropping it to save on echoed-input tokens throws away the cache reuse, which usually costs more than it saves. Providers that don't understand reasoning_content
just ignore the field, so it's safe to send.
So if you're measuring cache hit rate on a reasoning model, make sure you're re-sending the reasoning. Otherwise you're measuring a workload that disables its own cache.
If you've enabled prefix caching, cache hit rate is how you confirm it's earning its keep. The key thing to get right is the denominator: measure cache reuse against the history you re-sent, not against the whole request, or you'll understate a cache that's working fine. And remember it's strictly a multi-turn metric. A single-turn benchmark tells you nothing about caching.
The full version with the exact math and more detail is on my blog.