The Problem: Prefill and Decode Fighting Over the Same GPUs #
Standard LLM inference collocates two fundamentally different workloads on the same GPU resources. Prefill is compute-bound—it processes the entire input prompt in parallel using large matrix multiplications, with cost scaling directly by input length. Decode is memory-bandwidth-bound—it generates tokens one at a time, repeatedly model weights from high-bandwidth memory with relatively low compute per byte.
When both phases share the same instance, they interfere. A single prefill can block dozens of ongoing decode streams, causing visible stuttering in inter-token latency (ITL). Decode workloads delay scheduling of new prefills. The result is a system where neither phase runs efficiently or predictably.
vLLM’s April 2026 blog post demonstrates a practical solution: prefill-decode disaggregation on a single 8-GPU node using AMD’s MORI-IO connector, achieving 2.5x higher goodput compared to standard collocated serving.
MORI-IO: RDMA-Based KV Cache Transfer #
The challenge in disaggregation is the handoff. The KV cache generated during prefill must be transferred to the decode instance—and this can involve gigabytes of data. If the transfer itself becomes a bottleneck, the benefits of separation evaporate.
AMD addresses this with MORI-IO, an RDMA-based KV cache connector contributed to vLLM and built on the open-source MORI (Modular RDMA Interface) framework. It supports two transfer modes:
Read mode: The proxy waits for prefill to complete, forwards KV block locations to decode, and decode pulls the cache via RDMA. Adds proxy serialization overhead to time-to-first-token (TTFT).Write mode (default): The proxy dispatches to prefill and decode concurrently. Prefill pushes KV data layer-by-layer directly into decode’s pre-allocated memory as it computes. Eliminates the proxy serialization overhead.
Both modes perform a one-time metadata exchange via ZMQ before the first RDMA transfer—sharing KV cache base addresses, block sizes, and per-layer tensor strides. The resulting RDMA session is cached for all subsequent requests.
Benchmark Results: 2.5x Goodput on Same Hardware #
The benchmark used Qwen3-235B-A22B-FP8 (a mixture-of-experts model) on an 8× AMD Instinct MI300X node with AMD EPYC 9654 processors, running ROCm 6.10.5 and vLLM 0.16.0rc1. The team tested four configurations with 100 requests at 8 req/s, 2,000-token prompts, and 1,000-token outputs.
SLO targets: TTFT < 1 second and ITL < 50 ms per token. Goodput measures the maximum request rate where requests satisfy both thresholds—capturing cost and service quality in a single metric, following the DistServe methodology.
| Configuration | Requests Meeting SLOs | Relative Goodput |
|---|---|---|
| Standard (1× TP8) | 26/100 | 0.9x |
| Standard (2× TP4) | 30/100 | 1x |
| MORI-IO Read (1P+1D) | 70/100 | 2.4x |
| MORI-IO Write (1P+1D) | 73/100 | 2.5x |
Standard serving fails because ITL concentrates in two clusters—the high-latency cluster at ~150ms far exceeds the 50ms threshold. Both disaggregated modes eliminate ITL violations entirely. Write mode edges out read mode (73 vs 70) because concurrent proxy dispatch lowers TTFT.
The Trade-off: TTFT vs. ITL #
Disaggregation gives you stable, predictable ITL at the cost of longer waits for the first token. In read mode, TTFT increases by at least one full prefill forward pass plus the RDMA transfer time. In write mode, proxy serialization is eliminated—TTFT increases only by the RDMA transfer time, which overlaps with prefill compute, so the net penalty is smaller.
The practical guidance is straightforward: disaggregate when ITL P99 exceeds your SLO under production load, or when high concurrency with long prompts makes prefill interference untenable. Standard serving may be preferable when TTFT is your binding constraint, such as chatbot UX, or when request rates are low with short prompts.
Important Caveats #
The 2.5x goodput result is specific to the benchmark workload, hardware configuration, and SLO targets—it should not be generalized to all models or hardware. The vLLM documentation labels disaggregated prefilling as experimental. The technique does not inherently increase raw throughput; the reported gain is workload- and SLO-dependent goodput, not a universal throughput multiplier.
The experiment used a mixture-of-experts model (Qwen3-235B-A22B-FP8), which tends to amplify the interference pattern since expert routing adds variability to per-step compute, making ITL jitter more pronounced. The underlying prefill/decode interference is fundamental to transformer inference and applies to dense models as well, though the magnitude of improvement may differ.