Prefix caching at scale: when it saves you 80% of prefill cost, and the eviction policies that quietly turn it into 5%

A developer found that deploying a 70B Llama model with RAG features caused time-to-first-token (TTFT) to jump from 180 ms to 1.4 seconds, as the model recomputed identical attention states for repeated 6,000-token contexts on every request. Prefix caching solves this by reusing KV cache blocks for matching token prefixes, with vLLM's Automatic Prefix Caching (APC) using block-based, content-addressed hashing to skip prefill computation. However, the technique's effectiveness varies dramatically by workload—achieving 80% prefill savings in RAG with high prefix overlap, but dropping to just 5% when cache eviction or low prefix reuse undermines the benefit.