Efficient Decode Context Parallelism with vLLM for Long Context Workloads

VLLM's Decode Context Parallelism (DCP) enables higher concurrency and throughput for long-context agentic workloads by splitting the KV cache across GPUs, according to a new blog post from the vLLM team. In tests on a single 8×B200 node serving Kimi K2.6 in NVFP4, DCP sustained far higher concurrency (up to 512 requests) and delivered markedly higher throughput per GPU across the entire throughput–interactivity Pareto frontier compared to baseline tensor parallelism. The feature, supported in vLLM for almost a year, addresses memory duplication in GQA and MLA models, which previously capped concurrent requests and raised cost per token.

Efficient Decode Context Parallelism with vLLM for Long Context Workloads 1. Introduction Long-context inference is becoming essential for agentic AI, where assistants may need to reason over large code repositories and long chat histories. Agent-trace benchmarks now run from 64K all the way to 1M tokens and their KV caches are correspondingly large. Under a baseline tensor-parallel TP setup, this KV cache is partitioned by attention head, which puts a hard floor on how much it can shrink. Modern models use one of two attention schemes, and both hit this floor. Grouped-query attention GQA models store a small number of KV heads, and TP can only split the KV cache down to one head per GPU; once TP exceeds the number of KV heads, the cache starts duplicating across GPUs. Multi-head latent attention MLA models make this even worse: MLA compresses the Key/Value into a single low-rank latent vector shared across all query heads, so it effectively has only one KV head. Under normal TP there is nothing to split by head, meaning the latent KV cache is replicated in full across every TP rank. In both cases the duplicated KV cache eats into GPU memory, leaving very little room to serve additional requests. This caps the number of concurrent requests the system can handle, driving down throughput and pushing up cost per token. Decode Context Parallelism addresses this by splitting KV cache across the GPUs so each GPU stores and reads only part of the KV cache. This frees up GPU memory, allowing each GPU to take on more requests and thus run at a larger batch size. On systems with high-bandwidth GPU-to-GPU interconnects, this helps preserve interactive responsiveness while serving many long-context agents at once. vLLM has supported DCP for almost a year, but we are writing this blog now to highlight the feature, along with the recent improvements and advancements we have made to it, because the rise of long-context agentic use cases has made its benefits more relevant than ever. 2. Performance Results To quantify the benefit of Decode Context Parallelism, we compared a baseline tensor-parallel deployment against DCP on an identical set of GPUs, holding the model, hardware, and workload fixed and varying only how the KV cache is sharded during decode. 2.1 Dataset The dataset is a publicly available agentic long-context trace in Mooncake trace format, published here https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k2.6/perf/traces/64k 400 90kv agent new noschedule short 15perc.jsonl . See this section https://github.com/ai-dynamo/dynamo/blob/main/recipes/kimi-k2.6/perf/README.md dataset for more details on the dataset. It ships as JSONL where each line is a single request with input length , output length , and hash ids fields, so it can be replayed directly with any Mooncake-compatible harness e.g. aiperf --custom-dataset-type mooncake trace . The hash ids field encodes shared prefix blocks, making it well-suited for benchmarking KV-cache reuse and prefix-caching behavior. It's an agentic multi-turn workload of long inputs paired with short generations, chosen to reflect realistic long-horizon agent behavior. Inputs are centered around a median of ~67k tokens and paired with short ~400-token outputs, but the input distribution is bimodal rather than uniformly huge: roughly half the requests sit at 64k+ ≈53%, with a heavy tail reaching ~1M tokens and half are short-to-mid ≈47% under 64k, ~18% under 8k . About 8% of requests exceed 128k and ~3–4% exceed 256k. 2.2 Benefits of Decode Context Parallelism We ran an experiment on a single 8×B200 node serving Kimi K2.6 in NVFP4 with vLLM, sweeping request concurrency from 16 to 512 see table below . DCP sustains far higher concurrency and delivers markedly higher throughput per GPU across the entire throughput–interactivity Pareto frontier. The difference comes down to where the KV cache lives. Baseline TP replicates the KV cache on every GPU, so peak memory fills quickly. It reaches 100% at a concurrency of 64 and hits a wall, and throughput plateaus near 1,863 tok/s/GPU because no additional requests can fit. On the other hand, DCP shards the KV cache along the sequence dimension, so each GPU stores only 1/N of every request's KV. This allows space on the GPU to support more incoming requests. As a result, even at high concurrencies DCP keeps scaling where TP hits a wall. DCP reaches 6,091 tok/s/GPU at c512 while still sitting at just 82% KV usage. The core value of DCP is that it sustains far higher concurrency, even on long-context runs, precisely the regime where replicated-KV TP runs out of memory first. 2.3 Comparison by Sequence Length We also plotted performance against full sequence length input + output . The figure shows a single throughput–interactivity Pareto frontier with requests grouped into five length bands <32k, 32–64k, 64–128k, 128–200k, and 200k+ so we can see how performance shifts with context length. DCP keeps a high, stable frontier even in the 200k+ range , with the curves for short and long buckets nearly overlapping: throughput scales with concurrency while per-user speed stays usable at the long context lengths where the replicated-KV baseline runs out of memory and cannot scale. 3. Challenges of Serving Long Contexts Under tensor parallelism, the KV cache is partitioned by the attention head . Each KV head owns its own separate K and V tensors, and the head is the smallest unit you can hand to a GPU. A standard TP has no mechanism to slice a single head's KV cache. So if you have K KV heads, you can give each GPU a distinct subset of those heads, but only down to the point where every GPU holds one head. Once TP goes beyond K, there aren't enough distinct heads to go around, so two or more GPUs end up holding a copy of the same head's KV cache instead of a unique slice. 4. What is DCP? Unlike pure TP methods, DCP is able to split KV cache across GPUs by sequence context dimension. Each GPU is made responsible for the KV cache of a chunk of token positions from the same sequence. For a single 200K-token request, GPU 0 might hold the cache for tokens 0–50K, GPU 1 for tokens 50K–100K, GPU 2 for 100K–150K, and GPU 3 for 150K–200K. By sharding KV cache, the KV cache footprint per GPU keeps shrinking as you add GPUs, freeing the memory that lets you raise the batch size and serve higher concurrencies. 4.1 Decode Context Parallelism Process Standard Decode Context Parallelism keeps the communication pattern simple, following the rhythm AllGather Q → Compute → AllGather + ReduceScatter . - AllGather Q: Each GPU has computed only a fragment of the query, but attention requires the full query vector to score against any key. An all-gather across the DCP group assembles a complete copy of the query on every GPU. This is cheap during decode because the query is a single token. As an opt-in alternative for MLA, vLLM 45964 https://github.com/vllm-project/vllm/pull/45964 can replicate the small query projection within each DCP group at load time so decode skips this query all-gather entirely VLLM DCP Q REPLICATE=1 . - Compute: Each GPU runs attention between the gathered query and its local slice of the KV cache. In vLLM this is k up for MLA or tensor broadcast for GQA. - AllGather + ReduceScatter The partial results are combined into the true output. AllGather shares each GPU's partial output and LSE; the LSE values reweight and merge the partials the online-softmax trick , and ReduceScatter sums them while handing each GPU back only its own head-slice. cp lse ag out rs : 5. vLLM Usage DCP is enabled with a single extra argument, decode context parallel size , alongside your existing tensor-parallel setting. 5.1 Offline 5.2 Online 5.3 MLA Backend Models: DeepSeek-V2 / V3 / R1, Kimi K2.6 models using Multi-head Latent Attention. Why it's different. MLA compresses the Key/Value into a single low-rank latent vector that is shared across all query heads — effectively one KV "head." Under pure tensor parallelism there's nothing to split by head, so that latent KV cache is replicated in full on every TP rank. TP does nothing to shrink it, which makes MLA the ideal candidate for DCP: the whole cache is redundant, so the whole cache can be sequence-split. What they do. DCP splits the latent KV cache along the sequence dimension, so each rank stores only its chunk of the latent; at attention time each rank up-projects its latent slice the k up step to reconstruct the Keys/Values it needs. Because the effective KV-head count is 1, the sequence can be split up to the full TP degree — hence the constraints: tensor parallel size = decode context parallel size tensor parallel size % decode context parallel size == 0 5.4 GQA Backend Example models: Qwen3-235B, and other Grouped-Query-Attention models Llama-family, etc. . Why it's different. GQA stores num key value heads KV heads, and TP splits the KV cache by those heads first. That works cleanly only up to num key value heads ; once tensor parallel size exceeds it, the KV cache begins duplicating, with tp // num key value heads identical copies across ranks. What they do. DCP takes those would-be-duplicate copies and fills them with different sequence chunks instead, while the shared KV heads are broadcast across their query heads the "tensor broadcast for GQA" step . So the sequence-split degree is capped by the duplication factor tp // num key value heads : tensor parallel size // num key value heads = decode context parallel size tensor parallel size // num key value heads % decode context parallel size == 0 6. Future Work Looking ahead, we plan to extend DCP along several main directions. We will add support for finer-grained parallelism sizes for both TP and DCP, giving users more precise control over their parallelism layout and reclaiming efficiency lost to over-provisioned sharding. We are also developing better DCP all-to-all A2A communication kernels for both multinode and single-node settings, reducing exposed communication and improving overlap with compute as context length and device count grow. We are working on better support for MTP and speculative decoding, so that DCP can deliver its efficiency gains without sacrificing the latency benefits of speculative methods, as well as hardening prefill/decode P/D disaggregation support to make DCP robust in disaggregated serving deployments. Finally, we aim to broaden DCP's reach by extending support to a wider variety of backends and integrating it with hybrid models and Dynamic Chunked Pipeline Parallelism, so a much wider range of workloads can benefit from context-parallel efficiency gains. The community is also expanding DCP to additional models such as GLM-5.2 and Kimi K3, and there is a longer roadmap for Prefill Context Parallelism PCP . We are working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures. For deployment guidance and historical notes on DCP, see the vLLM Decode Context Parallel docs https://docs.vllm.ai/en/latest/serving/context parallel deployment/ decode-context-parallel . 7. Conclusion Decode Context Parallelism represents a fundamental rethinking of how GPUs are organized for long-context inference. Rather than forcing GPUs to duplicate KV cache or sit underutilized, DCP puts every GPU to work: sharding the sequence during attention, then immediately reconfiguring those same GPUs to amortize FFN weight loading across the full pool. The result is a system that scales gracefully with context length rather than degrading under it. With native support in vLLM, Decode Context Parallelism is ready to power the next generation of long-context agentic applications, from document reasoning to multi-session agentic pipelines, at the throughput and latency that production demands. It joins a broader industry move toward Decode Context Parallelism, a direction NVIDIA has also pursued with Helix Parallelism https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech blog/blog22 Helix Parallelism Scaling Multi Million Token Decoding with KV Cache Sharding.md in TensorRT-LLM. We are also working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures. About Us Special thanks to the NVIDIA team Anahita Bhiwandiwalla, Xin Li, Pavani Majety, Nidhi Bhatia, Roman Ageev, Pen Chung Li, and Chris Hoge for their reviews, benchmarking support, and engineering input throughout this study. We also thank Moonshot AI https://www.moonshot.cn/ for the initial Decode Context Parallel work upstreamed in vLLM 23734 https://github.com/vllm-project/vllm/pull/23734 , and Lucas Wilkinson https://github.com/LucasWilkinson for substantial follow-up contributions that helped harden and extend DCP. We also thank the broader vLLM community, whose open-source engine and continued collaboration made this benchmarking effort possible. For more on DCP deployment and related history, see the vLLM Decode Context Parallel docs https://docs.vllm.ai/en/latest/serving/context parallel deployment/ decode-context-parallel . The DCP results in this post were measured on NVIDIA B200 GPUs with Kimi K2.6 in NVFP4, and the recipes can be reproduced with current vLLM releases that support --decode-context-parallel-size . We are also working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures.