cd /news/ai-infrastructure/managed-nvidia-dynamo-on-amazon-sage… · home › topics › ai-infrastructure › article
[ARTICLE · art-171417] src=aws.amazon.com ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, letting customers run multi-node and disaggregated prefill/decode LLM serving through the HyperPod Inference add-on, AWS announced. The add-on installs Dynamo custom resource definitions plus the HyperPod Inference Operator, Dynamo operator, Grove and the Kubernetes AI (KAI) Scheduler, with AWS Deep Learning Containers supplying the Dynamo runtime and Elastic Fabric Adapter networking carrying KV cache transfers between prefill and decode pools. AWS cited GLM-5.3's FP8 weights, which exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance before KV cache and runtime overhead, as the kind of workload the multi-node setup addresses.

key takeaway

HyperPod Inference's add-on now deploys NVIDIA Dynamo via an InferenceEndpointConfig dynamoSpec, enabling multi-node and disaggregated prefill/decode serving with KV-aware routing and EFA-based KV cache transfer for models too large for one instance.

by read8 min views4 publishedNov 17, 2026
Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
Image: AWS ML Blog

Artificial Intelligence #

Serving large language models (LLMs) efficiently requires sufficient GPU memory and careful coordination of prompt processing and token generation. GLM-5.3’s FP8 weights, for example, exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance, before accounting for KV cache and runtime overhead. Long prompts can also slow token generation when both share the same GPUs.

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges. Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments with high-performance Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, and node health monitoring and recovery. In this post, we illustrate multi-node and disaggregated serving through a deployment example, share benchmark results, and outline best practices for optimizing inference on HyperPod.

How the integration works #

The HyperPod Inference add-on installs the Dynamo custom resource definitions and these serving and scheduling components:

  • HyperPod Inference Operator manages endpoint deployment through an InferenceEndpointConfig (IEC) custom resource. ItsdynamoSpec section supplies the Dynamo configuration.
  • Dynamo operator manages resources defined in aDynamoGraphDeployment (DGD) , including the frontend, model workers, replica counts, and resource requirements.
- *[Grove](https://github.com/ai-dynamo/grove)* coordinates related groups of pods, including multi-node workers, and their scaling relationships.
- *[Kubernetes AI (KAI) Scheduler](https://github.com/kai-scheduler/KAI-Scheduler)* places pods on GPU resources, using gang scheduling to schedule the pods required for a multi-node worker together.

AWS Deep Learning Containers (DLCs) provide the Dynamo runtime, while the add-on manages integrated component versions. To adopt a supported Dynamo release, update the add-on.

Deployment architecture

The following diagram shows the deployment paths, serving components, and infrastructure management for disaggregated inference on SageMaker HyperPod. The Dynamo frontend routes requests to prefill workers, which process prompts, and decode workers, which generate tokens. These separate pools allow independent scaling and tuning. Both pools load the full model weights, and KV cache transfers between them over EFA.

Dynamo also supports aggregated serving, where prefill and decode run in the same worker. In either topology, workers can run on one instance or span multiple instances using tensor and pipeline parallelism. Dynamo’s KV-aware router considers reusable prefix KV cache and active worker load in both topologies. Prefix reuse can reduce prompt recomputation and improve time to first token, while load awareness helps avoid over workers. For more details on the Dynamo architecture, see the Dynamo Knowledge Base overview.

HyperPod provides node, GPU, and EFA health monitoring and manages node replacement, while the orchestration components recreate affected workers. Deployments configured through InferenceEndpointConfig with dynamoSpec support invocation through SageMaker endpoints.

Two ways to configure your deployment #

Whether you are new to HyperPod, new to Dynamo, or both, the add-on offers two deployment paths to fit your workflow:

Bring-your-own DynamoGraphDeployment (DGD) lets existing Dynamo users continue using upstream documentation, recipes, and manifests, with the Dynamo operator managing the deployment. To bring an existing DGD to HyperPod, use a supported container image and configure storage, networking, and pod placement for your cluster.

InferenceEndpointConfig with dynamoSpec lets existing HyperPod Inference users deploy a new Dynamo endpoint using the same configuration format and OpenAI-compatible interface they already know. For the inference engine, you can choose vLLM, SGLang, or TensorRT-LLM, with vLLM used by default. The operator also supplies defaults for KV-aware routing and replica counts.

Both paths support single-instance and multi-node deployments. In either case, HyperPod manages node health and replacement. The table summarizes the responsibilities for each path.

| | Bring-your-own DGD | InferenceEndpointConfig with dynamoSpec | | You write | Complete DGD manifest | IEC containing dynamoSpec | | Best for | Teams wanting direct control of the serving graph or reuse of existing DGDs | Teams wanting managed endpoint setup through IEC | | Model weights | You configure down or mounting the weights | The operator stages weights from Amazon Simple Storage Service (Amazon S3) or Amazon FSx | | Endpoint, TLS, registration | You configure and manage these resources | Managed through the HyperPod Inference Operator | | Autoscaling | You configure and manage scaling | Kubernetes Event-driven Autoscaling (KEDA) through autoScalingSpec for aggregated and disaggregated deployments |

Deployment example: GLM-5.3 #

The following excerpts use the InferenceEndpointConfig (IEC) deployment path with vLLM on ml.p5.48xlarge instances. Dynamo-specific settings belong under spec.dynamoSpec. Model source and endpoint settings are included in the complete manifest.

Multi-node aggregated serving

This configuration runs one model-worker replica across two instances (16 GPUs), with two frontend replicas for request handling. Setting dynamoSpec.frontend.routingMode to kv-aware turns on KV-aware routing.

Here, spec.replicas controls the number of model-worker replicas, while frontend.replicas controls the number of frontend replicas.

Multi-node disaggregated serving

Replace aggregation under spec.dynamoSpec with the following disaggregation block, keeping the preceding frontend configuration. Replica counts are set separately for prefill and decode. This example runs one prefill replica and two decode replicas, each spanning two instances, for a total of six instances.

Apply the deployment

Apply the complete manifest with kubectl apply. The HyperPod Inference Operator makes the configured model weights available to the workers and generates the DGD. Grove and KAI Scheduler coordinate placement so that the pods required for each multi-node worker are scheduled together.

After the SageMaker endpoint is ready and configured for chat completions, clients can send an OpenAI-compatible request payload through the SageMaker Runtime API:

Benchmarks #

We benchmarked Dynamo on SageMaker HyperPod across two models to compare serving configurations.

Disaggregated prefill and decode for Qwen3-235B

We used AIPerf to compare aggregated and disaggregated serving for Qwen3-235B-A22B-FP8 using Dynamo with TensorRT-LLM on two p5en.48xlarge instances (16 H200 GPUs). Requests contained 4,000 input tokens and 200 output tokens. We retained measured points that met two targets: P90 time to first token below one second, and median generation speed of at least 50 tokens per second per user. Among these points, disaggregated serving achieved 2.05× the highest aggregated throughput: 207 versus 101 output tokens per second per GPU, at concurrency 80 and 32, respectively.

Multi-node aggregated serving for Qwen3.5-397B

Dynamo supports multi-node serving in both aggregated and disaggregated modes for models that exceed a single instance’s GPU memory. To evaluate a worker spanning nodes, we used AIPerf to benchmark multi-node aggregated serving for Qwen3.5-397B-A17B-FP8 with Dynamo and vLLM, using TP4 × PP2 and eight active H200 GPUs. This model fits on one instance, so in practice you would not split it. It serves here as a controlled baseline. We compared eight GPUs on one p5en.48xlarge against four GPUs on each of two instances, using the same model, parallelism, and workload configuration. With an agentic workload featuring high prefix reuse at concurrency 32, the two-node configuration delivered 126.4 output tokens per second per GPU, retaining 99.1 percent of single-node throughput. This shows that Dynamo on SageMaker HyperPod can serve a model that needs more than one instance across nodes. The node boundary itself does not become a significant throughput penalty.

Best practices for optimized inference #

NVIDIA Dynamo provides recipes and tools to help you select a serving configuration, benchmark performance, and tune your deployment on SageMaker HyperPod.

  • Start with a matching recipe – Choose aDynamo recipe for your model, inference engine, and GPU type. Account for model weights, KV cache, and runtime memory when sizing workers. For supported configurations,AISimulate can help compare aggregated and disaggregated serving, choose parallelism, and size prefill and decode worker pools before testing on GPUs.
  • Verify communication before benchmarking – For deployments spanning instances, confirm that communication uses the intended EFA transport, including KV-cache transfers between prefill and decode workers. See theDynamo EFA guide .
  • Benchmark representative traffic – Replay your traffic withAIPerf , or use agenerated agentic trace with representative prompt lengths, output lengths, and prefix reuse. Sweep concurrency for each configuration, retain points meeting your SLAs, and choose based on cost and latency priorities.
  • Tune from measured bottlenecks – Adjust topology and worker allocation before engine settings, changing one variable at a time. Follow theperformance tuning guide , or useDynamo optimization agent skills to assist with benchmarking and tuning. Validate each candidate on your target HyperPod deployment.

Conclusion #

NVIDIA Dynamo on SageMaker HyperPod Inference simplifies deploying and operating large models with multi-node serving, disaggregated prefill and decode, and KV-aware routing. The managed add-on installs the serving components, and configuration defaults simplify endpoint setup. HyperPod manages model weights, auto-scales clusters based on GPU and inference metrics, monitors node health, and replaces failed nodes. Together, these capabilities reduce undifferentiated infrastructure work so teams can focus on their models and applications. Get started with the setup instructions and complete deployment examples in the linked repository.

Acknowledgements

The authors would like to thank Eliuth Triana, Felipe Garcia, Sanjana Checker, Junyuan Yang, Jiahong Liu, Anish Maddipoti, Jacob Kern and Abdullahi Olaoye from NVIDIA for their contributions to this post. We also would like to thank Can Sun, Ophelia Yang, Aryaman Kothari, David Oh, and Ayush Kumar from AWS for their contributions.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @amazon sagemaker hyperpod 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/managed-nvidia-dynam…] indexed:0 read:8min 2026-11-17 · —