{"slug": "vllms-disaggregated-serving-cuts-gpu-interference-delivering-2-5x-higher-goodput", "title": "vLLM’s Disaggregated Serving Cuts GPU Interference, Delivering 2.5x Higher Goodput on the Same Hardware", "summary": "VLLM's disaggregated serving using AMD's MORI-IO connector achieved 2.5x higher goodput on the same 8-GPU node compared to standard collocated serving, according to an April 2026 blog post. The benchmark with Qwen3-235B-A22B-FP8 on 8x AMD Instinct MI300X showed 73/100 requests meeting SLOs in write mode versus 30/100 for standard TP4 serving.", "body_md": "## The Problem: Prefill and Decode Fighting Over the Same GPUs\n\nStandard LLM inference collocates two fundamentally different workloads on the same GPU resources. Prefill is compute-bound—it processes the entire input prompt in parallel using large matrix multiplications, with cost scaling directly by input length. Decode is memory-bandwidth-bound—it generates tokens one at a time, repeatedly loading model weights from high-bandwidth memory with relatively low compute per byte.\n\nWhen both phases share the same instance, they interfere. A single prefill can block dozens of ongoing decode streams, causing visible stuttering in inter-token latency (ITL). Decode workloads delay scheduling of new prefills. The result is a system where neither phase runs efficiently or predictably.\n\n[vLLM’s April 2026 blog post](https://vllm.ai/blog/2026-04-07-moriio-kv-connector) demonstrates a practical solution: prefill-decode disaggregation on a **single 8-GPU node** using AMD’s MORI-IO connector, achieving 2.5x higher goodput compared to standard collocated serving.\n\n## MORI-IO: RDMA-Based KV Cache Transfer\n\nThe challenge in disaggregation is the handoff. The KV cache generated during prefill must be transferred to the decode instance—and this can involve gigabytes of data. If the transfer itself becomes a bottleneck, the benefits of separation evaporate.\n\nAMD addresses this with MORI-IO, an RDMA-based KV cache connector contributed to vLLM and built on the open-source [MORI (Modular RDMA Interface)](https://github.com/ROCm/mori) framework. It supports two transfer modes:\n\n**Read mode**: The proxy waits for prefill to complete, forwards KV block locations to decode, and decode pulls the cache via RDMA. Adds proxy serialization overhead to time-to-first-token (TTFT).**Write mode (default)**: The proxy dispatches to prefill and decode concurrently. Prefill pushes KV data layer-by-layer directly into decode’s pre-allocated memory as it computes. Eliminates the proxy serialization overhead.\n\nBoth modes perform a one-time metadata exchange via ZMQ before the first RDMA transfer—sharing KV cache base addresses, block sizes, and per-layer tensor strides. The resulting RDMA session is cached for all subsequent requests.\n\n## Benchmark Results: 2.5x Goodput on Same Hardware\n\nThe benchmark used Qwen3-235B-A22B-FP8 (a mixture-of-experts model) on an 8× AMD Instinct MI300X node with AMD EPYC 9654 processors, running ROCm 6.10.5 and vLLM 0.16.0rc1. The team tested four configurations with 100 requests at 8 req/s, 2,000-token prompts, and 1,000-token outputs.\n\nSLO targets: TTFT < 1 second and ITL < 50 ms per token. Goodput measures the maximum request rate where requests satisfy both thresholds—capturing cost and service quality in a single metric, following the [DistServe methodology](https://haoailab.com/blogs/distserve/).\n\n| Configuration | Requests Meeting SLOs | Relative Goodput |\n|---|---|---|\n| Standard (1× TP8) | 26/100 | 0.9x |\n| Standard (2× TP4) | 30/100 | 1x |\n| MORI-IO Read (1P+1D) | 70/100 | 2.4x |\n| MORI-IO Write (1P+1D) | 73/100 | 2.5x |\n\nStandard serving fails because ITL concentrates in two clusters—the high-latency cluster at ~150ms far exceeds the 50ms threshold. Both disaggregated modes eliminate ITL violations entirely. Write mode edges out read mode (73 vs 70) because concurrent proxy dispatch lowers TTFT.\n\n## The Trade-off: TTFT vs. ITL\n\nDisaggregation gives you stable, predictable ITL at the cost of longer waits for the first token. In read mode, TTFT increases by at least one full prefill forward pass plus the RDMA transfer time. In write mode, proxy serialization is eliminated—TTFT increases only by the RDMA transfer time, which overlaps with prefill compute, so the net penalty is smaller.\n\nThe practical guidance is straightforward: disaggregate when ITL P99 exceeds your SLO under production load, or when high concurrency with long prompts makes prefill interference untenable. Standard serving may be preferable when TTFT is your binding constraint, such as chatbot UX, or when request rates are low with short prompts.\n\n## Important Caveats\n\nThe 2.5x goodput result is specific to the benchmark workload, hardware configuration, and SLO targets—it should not be generalized to all models or hardware. The vLLM documentation labels disaggregated prefilling as experimental. The technique does not inherently increase raw throughput; the reported gain is workload- and SLO-dependent goodput, not a universal throughput multiplier.\n\nThe experiment used a mixture-of-experts model (Qwen3-235B-A22B-FP8), which tends to amplify the interference pattern since expert routing adds variability to per-step compute, making ITL jitter more pronounced. The underlying prefill/decode interference is fundamental to transformer inference and applies to dense models as well, though the magnitude of improvement may differ.", "url": "https://wpnews.pro/news/vllms-disaggregated-serving-cuts-gpu-interference-delivering-2-5x-higher-goodput", "canonical_source": "https://forkast.news/vllms-disaggregated-serving-cuts-gpu-interference-delivering-2-5x-higher-goodput-on-the-same-hardware/", "published_at": "2026-08-22 03:14:16+00:00", "updated_at": "2026-08-22 03:42:39.214351+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["vLLM", "AMD", "MORI-IO", "Qwen3-235B-A22B-FP8", "AMD Instinct MI300X", "AMD EPYC 9654", "ROCm 6.10.5", "DistServe"], "alternates": {"html": "https://wpnews.pro/news/vllms-disaggregated-serving-cuts-gpu-interference-delivering-2-5x-higher-goodput", "markdown": "https://wpnews.pro/news/vllms-disaggregated-serving-cuts-gpu-interference-delivering-2-5x-higher-goodput.md", "text": "https://wpnews.pro/news/vllms-disaggregated-serving-cuts-gpu-interference-delivering-2-5x-higher-goodput.txt", "jsonld": "https://wpnews.pro/news/vllms-disaggregated-serving-cuts-gpu-interference-delivering-2-5x-higher-goodput.jsonld"}}