A new scheduler disrupts traditional LLM serving by allowing decode nodes to assist prefill phases. This innovation drastically cuts latency and boosts performance.
Advancements in large language model (LLM) serving often hinge on optimizing how tasks are split between prefill and decode phases. Traditionally, these phases are run on separate GPU pools to prevent interference. Yet, this separation can introduce inefficiencies, particularly under bursty, heavy-tailed workloads. Here's where the new scheduler comes in.
The Current Bottleneck #
In a typical setup, particularly in a production-style A100 cluster with 2 prefill and 2 decode nodes (known as 2P2D), the prefill execution is surprisingly minor. It only accounts for 2-23% of the P95 Time-to-First-Token (TTFT). What's dragging on performance? Queuing and inter-node GPU-GPU KV-cache transfers are the real culprits.
It's clear that simply dividing tasks between prefill and decode nodes isn't cutting it. We need a smarter way to allocate resources and reduce latency.
A Smarter Scheduler #
Enter the proactive prefill-deflecting scheduler. This innovation allows decode nodes to take on prefill tasks in a chunked-prefill manner, interleaving them with ongoing decode operations. By estimating the TTFT for each queued request, the scheduler finds the largest chunk schedule that maintains Time-Between-Tokens (TBT) service level objectives (SLO) on decode nodes. When it helps tail latency, it deflects from the prefill path.
What does this mean in practice? By running the prefill phase of deflected requests directly on the decode node, the dreaded inter-node KV transfer is eliminated. This is a big deal for reducing latency.
Real-World Impact #
Implemented on vLLM and tested with DeepSeek-V2-Lite on production-style traces, this approach reduces P95 TTFT by up to 81% and improves SLO attainment by up to 79% compared to the best disaggregated schedulers. All this at a sub-millisecond per-request routing cost.
Why should we care? Because this isn't just about marginal gains. It's about rethinking how we manage compute resources in LLMs. The architecture matters more than the parameter count. In a world where AI efficiency is key, this leap in performance could set a new standard.
The question is, will the industry adopt this proactive scheduling approach, or will it cling to outdated methods? The numbers tell a different story.
Get AI news in your inbox
Daily digest of what matters in AI.