{"slug": "why-my-prefix-cache-missed-identical-requests", "title": "Why My Prefix Cache Missed Identical Requests", "summary": "A developer built mini-vllm-rs, a Rust LLM inference engine for macOS, and found that prefix caching failed to reuse tokens when identical requests arrived concurrently to a cold cache because pages were only published at request completion. Adding incremental publication after each batched model forward let followers restore 480 of 493 prompt tokens and cut mean time to first token by 48–93% across groups of 2, 4, and 8 concurrent requests, while exact-input single-flight coordination introduced a trade-off where mean TTFT worsened for two simultaneous requests.", "body_md": "I enabled prefix caching in [mini-vllm-rs](https://github.com/lun0522/mini-vllm-rs), my Rust LLM inference engine for macOS, and submitted two identical requests concurrently to a cold cache.\n\nNeither reused a single token. Both computed the same prompt.\n\nWhy? My cache published reusable pages only when a request finished. Both requests arrived before either had published anything, so neither found a prefix to reuse. Later requests reused completed prefixes correctly.\n\nSupporting concurrent reuse required more decisions: when should cached pages become visible, should another request wait for them, and when should that wait end?\n\nBenchmarks led me to add incremental publication and single-flight coordination. Deciding when to release waiting requests then exposed another problem: an increasingly complicated AI-generated state machine.\n\nI tested a 4-bit quantized Qwen2.5 0.5B model on my Mac mini's Metal GPU, with 16-token cache pages and 3 repeated runs. My [first reuse benchmark](https://github.com/lun0522/mini-vllm-rs/blob/main/benchmarks/prefix_cache_reuse.md) compared warm reuse with simultaneous cold arrivals:\n\n| Workload | Result | \n|---|---|\n| Reuse a warm document prefix | Restored 880 of 904 input tokens, reducing time to first token (TTFT) by **96.2%** . | \n| 4 identical requests arriving to a cold cache | All restored zero tokens and performed full prefill. | \n\nThat cold-cache result was the problem I encountered at the start of this article.\n\nI added **incremental publication**. After each successful batched model forward, the engine immediately publishes any newly completed pages containing committed tokens, so they become available for reuse before this request finishes.\n\nTo test this change, I started one request and waited until it returned its first output token. While it continued generating the rest of its 128-token output, I submitted more requests with the same prompt to see whether they could reuse the cached prefix.\n\nIncremental publication let each follower restore 480 of its 493 prompt tokens. Across groups of 2, 4, and 8 concurrent requests, mean TTFT fell by **48–93%** compared with completion-only publication. That average includes every request's wait from submission to first streamed output. For detailed measurements, see the [follow-up benchmark report](https://github.com/lun0522/mini-vllm-rs/blob/main/benchmarks/prefix_cache_concurrent_population.md).\n\nHowever, incremental publication did not eliminate duplicate work for simultaneous cold arrivals. A request can reuse complete prefix pages already published when it starts execution, even if the other request is still prefilling. But it may find only part of the prompt, or nothing yet, and compute the rest itself. Reuse therefore depends on execution order and publication timing.\n\nAvoiding that race required deciding who should compute the prefix.\n\nI added **exact-input single flight**. One request becomes the leader, while other requests with identical input tokens wait as followers. This coordinates prompt computation, not answers. Every follower still generates its own output, as expected: identical prompts can use different generation settings, such as temperature.\n\nMy initial implementation made followers wait until the leader's entire request finished. The cache reuse became predictable, but the policy also made them wait through decode.\n\nWith 8 simultaneous requests each generating a 128-token output, this policy reduced both mean TTFT and the slowest request's completion latency compared with independent execution. But with only 2 such requests, **mean TTFT became worse** than independent execution, even though completion latency improved. Avoiding duplicate computation had introduced a new waiting cost.\n\nWhy wait through decode when the prompt cache was already ready? I added a second release policy. Once the leader finishes prefill, the engine publishes its completed prefix pages, then makes followers eligible for admission:\n\n```\nsequenceDiagram\n    participant S as Scheduler\n    participant C as Prefix cache\n    participant L as Leader\n    participant F as Followers\n    S->>L: Admit leader\n    S->>S: Block identical followers\n    L->>L: Finish prompt prefill\n    L->>C: Publish completed prefix pages\n    L->>S: Report successful publication\n    S->>S: Unblock followers\n    S->>F: Admit as capacity allows\n    F->>C: Look up matching prefix\n    C-->>F: Return shared KV page references\n    par Independent generation\n        L->>L: Continue decode\n    and Follower generation\n        F->>F: Generate their own outputs\n    end\n```\n\nFor simultaneous identical requests generating 128 output tokens each, both policies restored the same number of cached tokens, while releasing followers after prefill reduced mean TTFT:\n\n| Concurrent requests | Release at request completion | Release after prefill | TTFT reduction | \n|---|---|---|---|\n| 2 | 1,357.5 ms | 584.7 ms | 56.9% | \n| 4 | 1,757.9 ms | 617.1 ms | 64.9% | \n| 8 | 1,984.4 ms | 646.7 ms | 67.4% | \n\nTo compare both policies with independent execution, the table below focuses on the 8-request burst. Each request had a 532-token prompt and a 128-token output:\n\n| Strategy | Prompt tokens restored per follower | Mean TTFT change | Slowest-request completion-time change | \n|---|---|---|---|\n| Independent execution, publish at completion | 0 | Baseline | Baseline | \n| Single flight, release at request completion | 528 | **47.5% lower** | **25.8% lower** | \n| Single flight, release after prompt publication | 528 | **82.9% lower** | **32.4% lower** | \n\n*Regular incremental publication is disabled in all rows. The early-release policy explicitly publishes the prompt prefix when prefill finishes.*\n\nHowever, earlier release did not improve every metric. In a separate test, a second request arrived right after the original request entered decode phase. Compared with release-at-completion single flight, early release reduced mean TTFT by **72%**, but increased the slowest request's completion latency by **14%**.\n\nWhy? The original flight had already closed, so the second request started a new flight and reused the warm prefix. The two requests then decoded concurrently rather than sequentially, changing the active batch. That may have contributed to the longer completion time.\n\nThe earlier-release policy publishes the required checkpoint itself. Extra incremental publications added little for these coordinated exact-input workloads. They remain useful without coordination, including requests that share only part of a prefix.\n\n**Publication decides when data becomes reusable. Coordination decides who waits, and for how long.**\n\nThese are synthetic workloads on one backend with a small model. Exact-input coordination showed no meaningful completion-time regression for distinct inputs in this control, but the benchmark does not establish behavior under production traffic. Some future directions are:\n\nThe release change also exposed a design problem in the coordinator itself.\n\nAI helped implement these features. As in [my first article](https://pujun-ml-sys-blogs.hashnode.dev/building-an-llm-inference-engine-with-ai-the-code-was-the-easy-part), reviewing its designs was part of building a system I could understand and maintain.\n\n**Initially, AI deleted a flight when it released the followers,** which seemed reasonable because the waiting had ended.\n\nBut the first single-flight benchmark exposed a problem: followers later completed and notified the coordinator that no longer recognized their IDs. AI proposed removing the unknown-request check.\n\nI rejected that fix because it would also hide termination notifications carrying genuinely invalid request IDs. The original design had no `FlightState`. To preserve validation, we added `Active` and `Draining` states and tracked released followers until they terminated. Here is that first revision, with methods omitted:\n\n```\nenum FlightState {\n    Active { leader_request_id: u64 },\n    Draining,\n}\n\nstruct Flight {\n    input_token_ids: Arc<[u32]>,\n    state: FlightState,\n    follower_request_ids: VecDeque<u64>,\n}\n```\n\nThe leader had already finished, so `Draining` tracked only the released followers.\n\n**Earlier release made that representation more complicated.** The leader could now still be decoding after followers were released. AI retained the same `Flight` fields but changed the enum:\n\n```\nenum FlightState {\n    Active { leader_request_id: u64 },\n    Draining { leader_request_id: Option<u64> },\n}\n```\n\nThe optional leader made me pause. Different release policies required different combinations. If a policy should not retain a leader but the state contained one, would we need another check?\n\nI took a step back: did we really need a `Draining` state? We had added it to recognize released requests when they eventually terminated. That required remembering their IDs, but not preserving their former flight membership. Why not collect those IDs separately across flights?\n\nI asked AI to separate those responsibilities. We removed `FlightState` and tracked released request IDs separately:\n\n```\npub(super) struct PrefixCacheFlightCoordinator {\n    flights: HashMap<FlightId, Flight>,\n    released_request_ids: HashSet<u64>,\n    // Other fields omitted.\n}\n```\n\nOn release, we remove the flight and its indexes, keeping only the IDs of requests that have not yet terminated. This preserves termination validation without retaining leader/follower relationships that no longer affect execution. Unknown request IDs still produce errors, and a new request with the same prompt starts a new flight.\n\nAI preserved validation once I challenged the deletion. Later, it kept extending that solution until I questioned the abstraction itself.\n\n**Human review meant both preserving an invariant and reconsidering the structure used to enforce it.** The benchmark showed when requests could stop waiting. Reviewing the code showed when the flight could stop existing.", "url": "https://wpnews.pro/news/why-my-prefix-cache-missed-identical-requests", "canonical_source": "https://dev.to/lun0522/why-my-prefix-cache-missed-identical-requests-4imn", "published_at": "2026-10-16 18:44:13+00:00", "updated_at": "2026-10-16 18:50:38.424675+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["mini-vllm-rs", "Rust", "macOS", "Qwen2.5 0.5B", "Metal", "Mac mini"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-my-prefix-cache-missed-identical-requests", "markdown": "https://wpnews.pro/news/why-my-prefix-cache-missed-identical-requests.md", "text": "https://wpnews.pro/news/why-my-prefix-cache-missed-identical-requests.txt", "jsonld": "https://wpnews.pro/news/why-my-prefix-cache-missed-identical-requests.jsonld"}}