Run multi-node inference workloads using Ray LLM Deep Learning Containers AWS launched multi-node Ray LLM Deep Learning Containers, pre-built Docker images that let teams deploy a single large language model across two GPU nodes on Amazon EKS behind one OpenAI-compatible endpoint, splitting prefill and decode phases across separate nodes. The container bundles Ray Serve LLM, NVIDIA Inference Transfer Library (NIXL), and the Elastic Fabric Adapter (EFA) stack, requiring no application code, with complete code in an accompanying AWS samples repository. AWS says separating the compute-bound prefill phase from the memory-bandwidth-bound decode phase prevents long prompts from stalling token generation for in-progress requests, targeting workloads such as RAG, document summarization, and agentic applications. Artificial Intelligence https://aws.amazon.com/blogs/machine-learning/ Run multi-node inference workloads using Ray LLM Deep Learning Containers This post shows how to deploy one large language model LLM across two GPU nodes on Amazon Elastic Kubernetes Service Amazon EKS with the multi-node Ray LLM Deep Learning Containers https://aws.github.io/deep-learning-containers/ray-llm/ . It uses separate prefill and decode nodes behind one OpenAI-compatible endpoint. This kind of multi-node setup is common for tensor, pipeline, and data parallelism, as well as prefill/decode disaggregation inference workloads. In a previous post https://aws.amazon.com/blogs/machine-learning/simplify-and-support-your-torchserve-workloads-using-ray-serve-deep-learning-containers/ , we introduced the Ray Serve DLC https://aws.github.io/deep-learning-containers/ray/ and used it to stand up a small model behind an HTTP endpoint on a single GPU node. You had a working inference endpoint, and the container keeps the GPU stack compatible, including CUDA libraries, so you did not need to write boilerplate code. Ray https://docs.ray.io/en/latest/ray-overview/index.html is a distributed compute framework, so Ray Serve deployments can span multiple nodes. Customers use multi-node Ray Serve for several reasons. Some want to run foundation or open-weight models that are too large for a single GPU. Others need more throughput than one replica can deliver. Still others want to separate the two phases of inference to improve performance. In each case, customers must build the dependency chain themselves: choosing compatible libraries for the GPU stack and debugging issues when a component drifts out of alignment. This is undifferentiated work that takes time and effort away from delivering models. AWS Deep Learning Containers https://aws.amazon.com/ai/machine-learning/containers/ DLCs have long solved this kind of problem for training and inferencing workloads. DLCs are pre-built, performance-optimized Docker images that bundle a framework, its dependencies, and the GPU stack into a tested, patched combination you can pull and use immediately. With the launch of the Ray LLM DLC https://aws.github.io/deep-learning-containers/ray-llm/ for multi-node deployments, that same approach now extends to multi-node inference. You get a container purpose-built for serving models behind an HTTP endpoint, maintained and tested by AWS, with the full inference stack already assembled. This post walks through implementing multi-node disaggregated inference using multi-node Ray LLM DLC: deploying a single large language model across two GPU-powered nodes on Amazon EKS https://aws.amazon.com/eks/ cooperating behind a single endpoint. While one of the GPU nodes handles the prefill phase, the other GPU node handles the decode phase. Separating the phases prevents compute-heavy long prompts from stalling token generation for requests already in progress. This approach is well suited for workloads such as Retrieval Augmented Generation RAG , document summarization, and agentic applications. We use the multi-node Ray LLM DLC, which ships Ray Serve LLM https://docs.ray.io/en/latest/serve/llm/index.html , NVIDIA Inference Transfer Library NIXL https://github.com/ai-dynamo/nixl , and the Elastic Fabric Adapter EFA https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html stack, and exposes an OpenAI-compatible API with no application code. The complete code is available in the accompanying repository https://github.com/aws-samples/sample-aws-deep-learning-containers/tree/main/inference/ray-serve/ray-serve-multi-node . Why prefill/decode PD disaggregation LLM inference has two distinct phases. Prefill processes the entire input prompt in parallel and builds the key-value KV cache. Decode then generates the response one token at a time, reading and extending that cache. Prefill is compute bound, and Decode is bound by memory bandwidth. When both phases share one GPU, they contend for the same resources, and the bursty, compute-heavy prefill work interferes with the steady, latency-sensitive decode work. Separating the prefill phase from the decode phase means you can select the hardware and serving properties for each phase, improving overall performance at scale. PD disaggregation places the two phases on separate nodes, so each phase runs without interfering with the other and can be scaled independently. The trade-off is that the KV cache built during prefill must be handed to the decode node, and that transfer should be fast enough not to erase the benefit. That is exactly why the interconnect between the nodes matters. What the Ray LLM DLC provides AWS DLCs are pre-built, performance-optimized Docker images that bundle a framework, its dependencies, and the GPU stack into a tested, patched combination you can pull and use immediately. All components are validated together before each release, so there is no version drift between the CUDA runtime, the framework, and the serving layer. Security patches are applied at build time. The Ray LLM DLC https://aws.github.io/deep-learning-containers/ray-llm/ extends this to distributed LLM serving. It ships vLLM https://github.com/vllm-project/vllm as the inference engine, Ray Serve’s HTTP as the serving layer, NIXL for KV-cache transfer between nodes, and the EFA stack libfabric https://aws.amazon.com/hpc/efa/ and aws-ofi-nccl https://github.com/aws/aws-ofi-nccl for high-bandwidth node-to-node communication. Because the Ray Serve LLM library https://docs.ray.io/en/latest/serve/llm/index.html includes a built-in application builder, the entire serving application is a block of YAML. There’s no handler class, no model archiver, and no custom image to build. A single helper, build pd openai app , takes a prefill configuration and a decode configuration and wires the KV-cache transfer between them behind one OpenAI-compatible endpoint. Architecture The following diagram shows how the deployment is laid out on Amazon EKS. A user request reaches the Ray head on the CPU system node, and the two GPU worker nodes run the prefill and decode phases of the model. The deployment is an EKS cluster inside a virtual private cloud VPC . Nodes run in private subnets with outbound access through a NAT Gateway. A CPU system node m7i.xlarge hosts system workloads, the KubeRay https://docs.ray.io/en/latest/cluster/kubernetes/index.html operator, and the Ray head pod, which is CPU-only. Two GPU worker nodes g6.8xlarge run the Ray workers: one serving the prefill phase, and the other serving the decode phase. Both nodes pull the Ray LLM DLC from Amazon Elastic Container Registry Amazon ECR https://aws.amazon.com/ecr/ . KubeRay watches the RayService you apply and wires the head-to-worker join automatically. Ray Serve places the prefill and decode deployments on the two GPU workers. The KV cache flows from the prefill worker to the decode worker over NIXL, using EFA through NIXL’s libfabric backend. The GPU node group is pinned to a single Availability Zone AZ because EFA cannot cross AZs. Prerequisites To follow along with this post, you need: - An AWS account with billing enabled. - Sufficient service quota for running two g6.8xlarge instances at least 64 vCPU in your target Region. - AWS Command Line Interface AWS CLI https://aws.amazon.com/cli/ with credentials configured. - kubectl CLI for communicating with a Kubernetes cluster’s control plane. - eksctl CLI for creating and managing clusters on EKS. - helm package manager for Kubernetes to install KubeRay operator. - envsubst utility part of the GNU gettext to render the manifest. Verify that your AWS credentials are active with the following command: Use the pre-built application to serve inference The application is the serveConfigV2 block in manifest/rayservice.yaml . It uses build pd openai app , which takes a prefill config and a decode config and connects the KV-cache transfer between them: A few settings carry most of the weight. max model len: 4096 sizes the context to fit the measured KV cache on the GPU. dtype: bfloat16 and gpu memory utilization: 0.90 set precision and how much VRAM vLLM can claim. enforce eager: true skips CUDA graph capture to reduce peak GPU memory model- and input-dependent . The kv transfer config setting is what makes disaggregation work: both phases use NixlConnector with kv role: kv both and a distinct engine id prefill on one, decode on the other . The critical setting is kv connector extra config: { backends: "LIBFABRIC" } . Without that line, NIXL defaults to UCX, which runs at TCP speed even on EFA. The rest of the manifest, in rayClusterConfig , describes the pods. The head pod is CPU-only num-gpus: "0" and pinned to the system node group. Each worker pod requests one GPU and one EFA interface nvidia.com/gpu: "1", vpc.amazonaws.com/efa: "1" and targets the GPU nodes through nodeSelector: { role: gpu-worker } . It runs privileged: true because NIXL registers CUDA VRAM with the EFA NIC. The IPC LOCK capability alone isn’t enough. The workers set FI PROVIDER=efa , read VLLM HOST IP from the pod IP, and mount a shared-memory volume. Memory limits sit well above the roughly 18 GiB checkpoint so the OS page cache can hold it. For Qwen3.5-9B model loading on g6.8xlarge with the preceding cache configuration, the model weights loading drops from around 145 seconds to about 2 seconds on restart. Deploy on Amazon EKS All scripts share a single configuration file, scripts/env.sh . The script contains defaults for CLUSTER NAME , REGION , K8S VERSION , and NAMESPACE . Other defaults cover the system and GPU node groups, the KubeRay version, and the model. Override any variable by exporting it before running a script. From the project root, set a base directory, then load the variables: Step 1: Create the EKS cluster This provisions the EKS cluster VPC, OIDC provider, core add-ons and a CPU system node group that runs system workloads, the KubeRay operator, and the Ray head. Nodes run in private subnets with egress through a NAT Gateway, and cluster subnets are placed in the AZs that offer the GPU instance type. The script is idempotent and takes 15–20 minutes on a fresh run. Step 2: Add GPU worker nodes This creates the GPU node group: two g6.8xlarge instances with EFA enabled, pinned to a single AZ and labeled role=gpu-worker so the Ray workers target them. Enabling EFA also installs the EFA device plugin, which advertises vpc.amazonaws.com/efa on each node. This step takes about 5-20 minutes, usually longer on a first run. Step 3: Install the KubeRay operator This installs the KubeRay operator using Helm, pinned to the system node group to keep the GPU nodes available for inference. KubeRay watches the RayService you apply next and wires the head-to-worker join automatically. This step takes about 1–2 minutes. Step 4: Deploy the RayService This renders manifest/rayservice.yaml with your image, model, and naming variables using envsubst and applies it. KubeRay creates the head and two worker pods, and Ray Serve LLM brings up the prefill and decode deployments. The script waits for the head to be Ready, both workers to be Running, and finally for the Serve app to report RUNNING. Because each node loads the model, expect this step to take several minutes. Check status at any time, including which node each pod landed on and the GPU capacity available: Invoke the model After the Serve app is healthy, KubeRay creates a stable