LLM Serving: A Smarter Approach to Prefill and Decode

A new scheduler for large language model serving allows decode nodes to assist prefill phases, cutting P95 time-to-first-token by up to 81% and improving service-level objective attainment by up to 79% in tests with DeepSeek-V2-Lite on production-style traces.

LLM Serving: A Smarter Approach to Prefill and Decode A new scheduler disrupts traditional LLM serving by allowing decode nodes to assist prefill phases. This innovation drastically cuts latency and boosts performance. Advancements in large language model /glossary/large-language-model LLM serving often hinge on optimizing how tasks are split between prefill and decode phases. Traditionally, these phases are run on separate GPU /glossary/gpu pools to prevent interference. Yet, this separation can introduce inefficiencies, particularly under bursty, heavy-tailed workloads. Here's where the new scheduler comes in. The Current Bottleneck In a typical setup, particularly in a production-style A100 cluster with 2 prefill and 2 decode nodes known as 2P2D , the prefill execution is surprisingly minor. It only accounts for 2-23% of the P95 Time-to-First- Token /glossary/token TTFT . What's dragging on performance? Queuing and inter-node GPU-GPU KV-cache transfers are the real culprits. It's clear that simply dividing tasks between prefill and decode nodes isn't cutting it. We need a smarter way to allocate resources and reduce latency. A Smarter Scheduler Enter the proactive prefill-deflecting scheduler. This innovation allows decode nodes to take on prefill tasks in a chunked-prefill manner, interleaving them with ongoing decode operations. By estimating the TTFT for each queued request, the scheduler finds the largest chunk schedule that maintains Time-Between-Tokens TBT service level objectives SLO on decode nodes. When it helps tail latency, it deflects from the prefill path. What does this mean in practice? By running the prefill phase of deflected requests directly on the decode node, the dreaded inter-node KV transfer is eliminated. This is a big deal for reducing latency. Real-World Impact Implemented on vLLM and tested with DeepSeek /compare/llama-4-vs-deepseek-r1 -V2-Lite on production-style traces, this approach reduces P95 TTFT by up to 81% and improves SLO attainment by up to 79% compared to the best disaggregated schedulers. All this at a sub-millisecond per-request routing cost. Why should we care? Because this isn't just about marginal gains. It's about rethinking how we manage compute /glossary/compute resources in LLMs. The architecture matters more than the parameter /glossary/parameter count. In a world where AI efficiency is key, this leap in performance could set a new standard. The question is, will the industry adopt this proactive scheduling approach, or will it cling to outdated methods? The numbers tell a different story. Get AI news in your inbox Daily digest of what matters in AI.