{"slug": "disaggregated-prefill-and-decode-for-llm-inference-on-sagemaker-hyperpod", "title": "Disaggregated prefill and decode for LLM inference on SageMaker HyperPod", "summary": "AWS introduced disaggregated prefill and decode (DPD) for LLM inference on SageMaker HyperPod, separating compute-bound prefill and memory-bound decode onto different GPU pools connected via EFA RDMA to eliminate interference and improve tail latency. The approach, built on vLLM and LMCache, targets long-context, high-concurrency workloads like chatbots and RAG, with a router that decides whether to use disaggregated or colocated paths based on prompt length.", "body_md": "[Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Disaggregated prefill and decode for LLM inference on SageMaker HyperPod\n\nWhen prefill and decode share a GPU, long prompts stall token generation for every concurrent request. Disaggregated Prefill and Decode (DPD) removes this interference by running each phase on separate GPU pools connected through Elastic Fabric Adapter (EFA) with Remote Direct Memory Access (RDMA). Large language model (LLM) inference has two fundamentally different phases. Prefill is compute-bound. It processes the entire input prompt in parallel to generate the initial key-value (KV) cache. Decode is memory-bound. It generates one token at a time and requires substantial memory bandwidth to access model weights and the growing KV cache. By disaggregating these into specialized engines, you can assign different parallel strategies to each phase. With this separation, you can tune time to first token (TTFT) and inter-token latency (ITL) independently, control tail latency more reliably than chunked prefill tuning, and keep long-context prefills from blocking ongoing decode requests. vLLM improves single-node efficiency through continuous batching and PagedAttention. However, organizations that deploy at scale still face challenges when they orchestrate multi-node deployments and optimize routing.\n\nIn this post, we show how to implement DPD with vLLM on [Amazon SageMaker HyperPod](https://aws.amazon.com/sagemaker/ai/hyperpod/) using the [HyperPod Inference Operator](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment.html).\n\n## When to use disaggregated inference\n\nDisaggregating prefill and decode delivers the strongest gains for long-context, high-concurrency streaming workloads: chat assistants, agentic pipelines, document-analysis endpoints, and Retrieval Augmented Generation (RAG) with large retrieved contexts. In these cases, a single long prompt on a colocated GPU stalls in-flight decode for every other request, causing per-token latency spikes that DPD removes by construction.\n\n**Consider DPD when your workload has:**\n\n- Input prompts that regularly exceed 4,096 tokens.\n- Multiple concurrent users or requests.\n- Streaming responses where consistent token delivery matters.\n- Mixed traffic with both long and short prompts.\n\nA colocated deployment is the simpler choice when GPU contention is not a real concern: batch or offline workloads optimizing for TTFT, low-concurrency deployments, or short-prompt-only traffic. Below the routing threshold, the fixed cost of transferring KV cache over EFA RDMA outweighs the benefit of isolating decode. The DPD router sends those requests straight to a decoder. A single endpoint therefore handles mixed long and short traffic automatically, without manual routing logic.\n\nDPD requires at minimum one prefill node and one decode node with RDMA-capable EFA networking. For supported instance types, see the [Deploy a DPD model endpoint to your HyperPod cluster](#deploy-a-dpd-model-endpoint-to-your-hyperpod-cluster) section.\n\n## Architecture\n\nThe HyperPod DPD implementation is built on the [vLLM](https://vllm.ai/) Production Stack router, with [LMCache](https://lmcache.ai/) providing the KV cache transfer layer over NIXL and EFA. The deployment has three components plus a transport stack.\n\n### Intelligent router\n\nThe router is the control plane. It tokenizes each prompt and applies a configurable token threshold to decide whether the request takes the disaggregated path or runs end-to-end on a decoder. Long-context prompts go through a prefiller then a decoder. Short prompts skip the prefiller, avoiding cross-GPU KV transfer that isn’t worthwhile. For disaggregated requests, it directs the prefiller to compute and push KV cache to a decoder, then forwards the request to that decoder for generation. It also supports per-prefiller routing strategies (`prefixaware`\n\n, `kvaware`\n\n, `session`\n\n, `roundrobin`\n\n) through `intelligentRoutingSpec.routingStrategy`\n\nto maximize cache locality across replicas.\n\n### Prefiller pod\n\nThe prefiller is a vLLM worker with LMCache as its KV connector through `LMCacheConnectorV1`\n\n. It computes KV cache for long prompts and pushes it to the chosen decoder via LMCache’s PD sender backend layer-by-layer, overlapping compute and transfer to keep its GPUs saturated. LMCache also gives each prefiller an L1 CPU cache. When a prefix recurs (system prompts, multi-turn history, retrieval contexts), it serves from CPU memory without GPU recomputation. This produces significant TTFT gains. Activating DPD on an `InferenceEndpointConfig`\n\nprovisions both the connector and the cache automatically.\n\n### Decoder pod\n\nThe decoder is a vLLM worker with LMCache as its receiver. It reserves GPU memory (the PD buffer, sized by `PD_BUFFER_SIZE`\n\n) for incoming KV transfers. It runs full CUDA graphs for the decode kernel and starts generation as soon as the transfer completes. Because it never executes prefill, decode latency stays stable under concurrency and adding a long-context request never disturbs tokens already streaming.\n\n### KV transfer\n\nKV cache transfer uses a four-layer stack (LMCache PD → NIXL → `libfabric`\n\n→ EFA) that HyperPod composes end-to-end. LMCache’s PD backend orchestrates the prefiller-side put and decoder-side retrieval. NIXL provides a unified memory abstraction across GPU, CPU, and remote peers and selects the right RDMA operation. The `libfabric`\n\nprovider exposes EFA as kernel-bypass, GPU-Direct RDMA, keeping the host CPU off the data path. This makes transfer cost negligible relative to prefill compute: on `ml.p5.48xlarge`\n\nwith 3,200 Gbps of EFA, an 8,000-token transfer for Llama 3.3 70B takes single-digit milliseconds. HyperPod ships the stack pre-integrated, so you select a DPD-supported worker image and the operator wires up the connector, NIXL, and EFA on every pod.\n\n## Deployment overview\n\nDisaggregated Prefill and Decode (DPD) is a functionality implemented by the HyperPod Inference Operator. This section covers prerequisites, installation of the inference operator and deployment of an inference endpoint that uses DPD for efficiently serving a Llama 70B model.\n\n### Prerequisites and HyperPod Inference Operator installation\n\nMake sure you have the [AWS Command Line Interface](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) (AWS CLI), [kubectl](https://kubernetes.io/docs/tasks/tools/) access to your HyperPod cluster, a [HuggingFace](https://huggingface.co/) token, and sufficient service quota. Set up your local `kubectl`\n\nconfiguration to connect to your HyperPod cluster. For more information, see [Disaggregated Prefill and Decode for HyperPod inference](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-dpd.html).\n\nDPD requires HyperPod Inference Operator version 3.2 or later. The operator is installed by default on new HyperPod EKS clusters. For installation, setup, and upgrade instructions, see [ Unlock efficient model deployment: simplified Inference Operator setup on Amazon SageMaker HyperPod](https://aws.amazon.com/blogs/architecture/unlock-efficient-model-deployment-simplified-inference-operator-setup-on-amazon-sagemaker-hyperpod/).\n\nVerify your operator version by running:\n\nThe output is the full container image reference. The tag at the end encodes the version, for example:\n\nIf your operator version is not up-to-date, upgrade before continuing by following the upgrade instructions in the [HyperPod Inference Operator Release Notes](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-inference-release-notes.html#sagemaker-hyperpod-inference-release-notes-20260506).\n\n### Deploy a DPD model endpoint to your HyperPod cluster\n\nIn this example, we deploy the Meta Llama 3.3 70B model on two `ml.p5.48xlarge`\n\ninstances. Verify that the instances are available in an instance group within the HyperPod cluster before proceeding. For DPD inference deployments, choose instance types that support both NVLink and EFA. EFA needs to support RDMA in read and write mode. This includes the [P5](https://aws.amazon.com/ec2/instance-types/p5/) and [P6](https://aws.amazon.com/ec2/instance-types/p6/) instance families on AWS. Note that instances are required to be located within the same Availability Zone (AZ) for EFA high-bandwidth communication. Although G6, G6e, and G7e instance families do support EFA with RDMA read/write, performance on multi-GPU instances is bottlenecked by GPU-to-GPU communication over PCIe.\n\nThe worker image for the inference deployment must include vLLM, LMCache, NVIDIA NIXL, and the EFA `libfabric`\n\nprovider. At the time of writing, we support two image options:\n\n- Open source LMCache:\n[lmcache/vllm-openai v0.4.3](https://hub.docker.com/r/lmcache/vllm-openai/tags). - SageMaker Deep Learning Container (DLC): vllm:server-hyperpod-cuda-v1.1.\n\n#### Model checkpoint location\n\nThe HyperPod Inference Operator supports a broad range of checkpoint loading sources, including [Amazon Simple Storage Service (Amazon S3) buckets, Amazon FSx file systems, and direct pulling from HuggingFace](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-deploy-ftm.html) and the [instance NVMe storage](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-deploy-nvme.html). For this post, we load the model checkpoint from an Amazon S3 bucket.\n\nVerify that you have downloaded your preferred model checkpoint to an S3 bucket in the same Region as your HyperPod cluster. If you haven’t done so yet, set up your bucket name and HuggingFace token, then download Meta Llama 3.3 70B Instruct from HuggingFace and sync it to Amazon S3. To achieve high-bandwidth networking to Amazon S3, we recommend that you run this from an Amazon Elastic Compute Cloud (Amazon EC2) instance.\n\nPrepare the model deployment manifest and change the environment variables as needed:\n\nFor the full deployment YAML, see [Deploy a DPD endpoint](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-dpd.html#sagemaker-hyperpod-model-deployment-dpd-deploy).\n\n#### DPD-relevant fields in the deployment manifest\n\nMost `InferenceEndpointConfig`\n\nfields are shared with non-DPD endpoints and documented in the [Inference Operator documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment.html). The fields below are required or have different semantics for DPD.\n\n**spec.pdSpec**: Declares the prefill/decode topology and specifies arguments. Presence of this field is what makes the endpoint disaggregated: the operator creates separate `Deployment`\n\nobjects for prefill and decode and wires them together through the router and LMCache PD backend.\n\n`replicas`\n\n: Scale prefill and decode independently.`resources`\n\n: Applied to the role’s pod spec. Top-level`worker.resources`\n\nis ignored for DPD pods. Per-role values override.`routingThreshold`\n\n: Token length threshold above which requests use the disaggregated path. Below the threshold, requests bypass the prefiller and go directly to the decoder.`args`\n\n: vLLM flags specific to that role, merged into`worker.args`\n\nat startup. Flags already in`worker.args`\n\nare replaced with the per-role value, and flags not present are appended.\n\n**spec.worker.environmentVariables**: These environment variables are applied identically to both the prefiller and decoder containers. There is no per-role environment-variable field today. For per-role behavior, use `pdSpec.{prefillSpec,decodingSpec}.args`\n\ninstead.\n\nMore details about environment variables are in [Deploy a DPD endpoint](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-dpd.html#sagemaker-hyperpod-model-deployment-dpd-deploy).\n\n## Apply the manifest and validate the deployment\n\nThe operator creates two `Deployment`\n\nobjects in your namespace and a router `Deployment`\n\nin `hyperpod-inference-system`\n\n. Image pull and model load take a few minutes. The pods first enter `ContainerCreating`\n\n, then become `Running`\n\nas containers come up. List the pods across both namespaces:\n\nEach model pod has 3 containers: the vLLM worker, an Nginx reverse proxy, and an OpenTelemetry collector. The router pod has 2 containers (router, otel). The IEC condition reports readiness:\n\n## Invoke the endpoint and verify KV transfer\n\nOnce the endpoint is ready, send a short and a long prompt to exercise both routing paths. Check the prefiller and decoder logs to confirm KV cache is being transferred over EFA. The following commands assume the IEC was deployed with `${NAMESPACE}`\n\n, `${ENDPOINT_NAME}`\n\n, and `${DEPLOYMENT_NAME}`\n\nset as in the preceding manifest.\n\nGet the required pod names and router url:\n\n### Short prompt (below threshold, direct-to-decoder path)\n\nRun a short prompt through a pod within the cluster that invokes the endpoint:\n\n### Long prompt (above threshold, DPD path)\n\nA prompt above the 4,096-token routing threshold is routed through the prefiller, then to the decoder for token generation. The following example builds a roughly 6,000-token prompt by repeating a sentence:\n\nTo confirm that the invocation used the disaggregated path, we can check the router pod logs:\n\nFor the long prompt, you can observe the following output:\n\n`disaggregate=True`\n\nconfirms the request took the prefiller path. The two `Routing request`\n\nlines show the prefill hop followed by the decode hop.\n\n## Scaling guidance\n\nDPD currently supports a single decoder replica with multiple prefiller replicas. This means you scale prefill capacity independently while the decoder remains fixed at one instance.\n\n**Start with a 1:1 prefill-to-decode ratio** for balanced workloads (chat, code generation) where input and output lengths are comparable. **Scale to 2:1 or 3:1** when your workload is prefill-heavy: summarization, classification, or RAG with long retrieved contexts. This ratio is appropriate when you observe TTFT climbing under load while per-token output latency (TPOT) remains stable.\n\nWith multiple prefillers, set `intelligentRoutingSpec.routingStrategy`\n\non your workload. Use `kvaware`\n\nfor workloads with repeated prefixes (this maximizes L1 cache hits across prefiller partitions). Use `session`\n\nfor multi-turn conversations that benefit from keeping a user’s context on one prefiller.\n\nIf instead TPOT is climbing and output throughput plateaus despite prefiller availability, the single decoder is saturated. In that case, increase `PD_BUFFER_SIZE`\n\n, reduce `max-model-len`\n\n, or reduce concurrency to the endpoint until multi-decoder support is available.\n\n## Benchmarking performance\n\nBenchmarks used `genai-bench`\n\nwith fixed-length synthetic prompts (4,096 input tokens, 256 output tokens) at concurrency levels 8, 16, and 32. Each concurrency level ran until results stabilized. DPD configuration: 1 prefiller and 1 decoder across 2 nodes (16 GPUs), KV-aware routing, `enforce-eager`\n\non the prefiller, and CUDA graphs on the decoder. Baseline: single node (8 GPUs), same model and GPU settings. Hardware: `ml.p5.48xlarge`\n\n(8x H100 80GB, EFA enabled). Model: Llama-3.3-70B-Instruct with `tensor-parallel-size=8`\n\nand `max-model-len=16,384`\n\n. The following charts show the percentage improvement DPD delivers over the colocated baseline on two instance families. Higher bars indicate larger DPD advantage.\n\nThe following chart shows the same benchmarks on `ml.p5en.48xlarge`\n\ninstances with H200 GPUs, where DPD gains are equally pronounced.\n\nAcross both hardware configurations, DPD delivers consistent gains on per-token output latency (TPOT), end-to-end latency, and throughput as concurrency grows:\n\n**Per-token latency stays flat under load**. DPD isolates decode from prefill interference, keeping TPOT constant regardless of concurrent long-context requests. For D(4096,256) workloads at concurrency 8 to 32, improvement ranges from 22% at low concurrency to 66% at high concurrency on H100, and 28% to 48% on H200.**Throughput scales with concurrency**. The dedicated decoder runs at full CUDA graph efficiency without prefill interruption. Output throughput improves up to 35% on H100 and up to 64% on H200 at higher concurrency.**End-to-end latency improves at P50**. The cumulative TPOT savings across output tokens outweigh the KV transfer cost. E2E P50 improves 14-32% on H100 and 29-41% on H200.\n\nDPD does introduce a modest increase in time to first token because of the KV cache transfer over EFA RDMA. For streaming workloads where consistent per-token delivery matters more than initial response, this tradeoff is favorable. The conditional routing threshold (default 4,096 tokens) is designed to make sure that short requests bypass disaggregation entirely, avoiding transfer overhead where it is not needed.\n\n## Observability\n\nYou can monitor DPD metrics through the SageMaker HyperPod Observability features. For more information, see [Accelerate foundation model development with one-click observability in Amazon SageMaker HyperPod](https://aws.amazon.com/blogs/machine-learning/accelerate-foundation-model-development-with-one-click-observability-in-amazon-sagemaker-hyperpod/).\n\nDPD metrics are available in the Inference dashboard.\n\nAdditional metrics that may be useful are CPU/GPU usage, which is available in the **Tasks** dashboard, as well as the metrics available in the **Cluster Overview** dashboard.\n\n## Clean up\n\nTo avoid ongoing charges, delete the resources created during this walkthrough when you are done experimenting.\n\n- Delete the\n`InferenceEndpointConfig`\n\nto remove the prefill pod, decode pod, router, and all associated services: - (Optional) Remove the model from S3 if you uploaded it specifically for this walkthrough:\n- (Optional) Scale down or remove the HyperPod instance group if the GPU instances were provisioned solely for this deployment. Refer to the\n[Managing HyperPod clusters](https://docs.aws.amazon.com/sagemaker/latest/dg/smcluster-scale-down.html)documentation for instructions.\n\n## Conclusion\n\nDisaggregated Prefill and Decode (DPD) on Amazon SageMaker HyperPod runs prefill and decode on separate GPU pools. KV cache transfers between them over EFA using GPU-Direct RDMA. Prefill is compute-bound. Decode is memory-bandwidth-bound. When colocated, the two phases compete for the same GPU resources. A single long prompt can stall in-flight decoding and inflate tail per-token latency. Separating them removes that interference, produces more predictable latency under mixed traffic, and lets you scale each phase independently.\n\nThe HyperPod Inference Operator handles the underlying orchestration: provisioning the router, wiring the prefill and decode pods together, and integrating with HyperPod observability. You activate DPD by adding a few fields to the same `InferenceEndpointConfig`\n\nresource you already use for non-disaggregated endpoints.\n\nYou can get started today by deploying a DPD endpoint on your HyperPod EKS cluster following the steps in this post. To learn more, visit the [Amazon SageMaker HyperPod documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod.html), the [HyperPod Inference Operator model deployment guide](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment.html) or directly try out the example manifest in this post.", "url": "https://wpnews.pro/news/disaggregated-prefill-and-decode-for-llm-inference-on-sagemaker-hyperpod", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/disaggregated-prefill-and-decode-for-llm-inference-on-sagemaker-hyperpod/", "published_at": "2026-07-10 15:20:16+00:00", "updated_at": "2026-07-10 15:39:35.623211+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-products"], "entities": ["AWS", "SageMaker HyperPod", "vLLM", "LMCache", "Elastic Fabric Adapter"], "alternates": {"html": "https://wpnews.pro/news/disaggregated-prefill-and-decode-for-llm-inference-on-sagemaker-hyperpod", "markdown": "https://wpnews.pro/news/disaggregated-prefill-and-decode-for-llm-inference-on-sagemaker-hyperpod.md", "text": "https://wpnews.pro/news/disaggregated-prefill-and-decode-for-llm-inference-on-sagemaker-hyperpod.txt", "jsonld": "https://wpnews.pro/news/disaggregated-prefill-and-decode-for-llm-inference-on-sagemaker-hyperpod.jsonld"}}