Accelerate agentic AI with NVIDIA Nemotron models on Amazon EKS and SageMaker HyperPod EKS

NVIDIA's Nemotron 3.5 Super VL, a 120B-parameter mixture-of-experts model that activates 12B parameters per token, is now available on AWS alongside the existing Nemotron 3 Nano, Nano Omni, Super, Ultra and Nemotron 3.5 Lightning models, AWS said. The open multimodal model, distilled from Nemotron 3 Ultra, handles text, image, document and video input with a context length up to 1M tokens and ships in BF16 and NVFP4 checkpoints; NVIDIA reports 63.44 percent on Hermes Bench, 87.05 percent on LangChain agent evaluations and 80.8 percent on the CharXiv chart-reasoning benchmark. AWS documents deploying the model on Amazon EKS, Amazon EKS Auto Mode and Amazon SageMaker HyperPod EKS for agent workflows spanning coding, web research, document intelligence, cyber security and video understanding.

Artificial Intelligence https://aws.amazon.com/blogs/machine-learning/ Accelerate agentic AI with NVIDIA Nemotron models on Amazon EKS and SageMaker HyperPod EKS NVIDIA Nemotron 3.5 Super VL https://aws.amazon.com/about-aws/whats-new/2026/01/nvidia-nemotron-3.5-super-vl-on-sagemaker-jumpstart/ joins the NVIDIA Nemotron 3 Nano https://aws.amazon.com/blogs/machine-learning/run-nvidia-nemotron-3-nano-as-a-fully-managed-serverless-model-on-amazon-bedrock/ , Nano Omni https://aws.amazon.com/blogs/machine-learning/nvidia-nemotron-3-nano-omni-model-now-available-on-amazon-sagemaker-jumpstart/ , Super https://aws.amazon.com/blogs/machine-learning/run-nvidia-nemotron-3-super-on-amazon-bedrock/ , Ultra https://aws.amazon.com/blogs/machine-learning/nvidia-nemotron-3-ultra-now-available-on-amazon-sagemaker-jumpstart/ , and Nemotron 3.5 Lightning https://aws.amazon.com/blogs/machine-learning/nvidia-nemotron-3-5-lightning-now-available-in-amazon-sagemaker-jumpstart/ models already available on AWS. It’s a 120B-parameter mixture-of-experts MoE model that activates 12B parameters per token. It combines reasoning and tool use with text, image, document, and video understanding, so one model serves both text and visual tasks. This post explores its architecture and use cases, then shows how to deploy it on Amazon Elastic Kubernetes Service Amazon EKS https://aws.amazon.com/eks/ , Amazon EKS Auto Mode https://aws.amazon.com/eks/auto-mode/ , and Amazon SageMaker HyperPod EKS https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-eks.html . About Nemotron 3.5 Super VL NVIDIA Nemotron 3.5 Super VL is an open multimodal model distilled from Nemotron 3 Ultra. NVIDIA publishes the weights, training datasets, and post-training recipes, so you can customize the model for domain-specific workloads and run it on infrastructure you control. Model details: - 120B total parameters, 12B active per token. - Hybrid Mamba-2, MoE https://www.nvidia.com/en-us/glossary/mixture-of-experts/ , and attention layers with a RADIO vision encoder. - Context length up to 1M tokens. - Input: text, image, document, and video. - Output: text. - BF16 and NVFP4 checkpoints. NVIDIA reports 63.44 percent on Hermes Bench https://portal.nousresearch.com/bench and 87.05 percent on LangChain agent evaluations, and 80.8 percent on the CharXiv chart-reasoning benchmark. technical report https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-Lightning-Technical-Report.pdf Model architecture Nemotron 3.5 Super VL uses a hybrid Mamba-Transformer MoE architecture, interleaving Mamba-2, attention, and expert layers. Sparse routing activates 12B of the 120B parameters per token, so compute per token matches a much smaller model while GPU memory still holds the full weights. Mamba-2 layers carry a constant-size state during generation, so their memory does not grow with sequence length the way KV cache does, which keeps long contexts affordable. A recursively shared Multi-Token Prediction MTP head supports native speculative decoding. Use cases Nemotron 3.5 Super VL supports agent workflows that combine reasoning with visual understanding: - Coding: test and validate code, analyze errors, recommend fixes. - Web research: cross-reference and verify sources returned by search tools. - Document intelligence: read tables, charts, and page layouts to produce summaries. - Cyber security: identify vulnerabilities and triage threats. - Video understanding: search, content, review, and incident analysis. Nemotron 3.5 Super VL integrates with major open agent frameworks and orchestration systems through NVIDIA reference implementations, cookbooks, and the NVIDIA Agent Toolkit https://blogs.nvidia.com/blog/nvidia-agent-toolkit-open-models-tools-skills-secure-runtime-ai-agents/ . For the full list of supported frameworks, see the NVIDIA developer blog https://developer.nvidia.com/blog/nvidia-nemotron-3-5-super-vl-powers-specialized-agents-with-multimodal-reasoning/ . Minimum GPU configurations | vLLM GPU Model / min count | BF16 | NVFP4 | | B200/GB200 | 8 | 2 | | B300/GB300 | 4 | 1 | | H200 | 8 | 4 | | H100 | 16 | 8 | Deployment on AWS This section gives an overview of architecture, validation, and performance evaluation approaches. A detailed walkthrough https://github.com/aws-samples/aws-do-eks/tree/main/Container-Root/eks/deployment/inference/agentic-ai/nemotron of deploying Nemotron 3.5 Super VL and Nemotron 3 Ultra on AWS is available in the aws-do-eks https://bit.ly/do-eks project, including prerequisites, cluster creation, model weights download, deployment, testing, benchmarking, and cleanup. The same pattern can also be applied to other models in the NVIDIA Nemotron family. Deployment architecture The reference implementation https://bit.ly/aws-do-nemotron runs on standard Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS. With EKS Auto Mode, AWS manages scaling, GPU, and EFA support in the cluster on your behalf. The Kubernetes manifests for model deployment are the same for each of the orchestrator options. Each cluster has a control plane hosting the Kubernetes API and a data plane of CPU and GPU nodes. CPU nodes run system and test pods, while the GPU nodes run the model. Amazon Elastic Container Registry Amazon ECR stores the container images, a shared Amazon FSx for Lustre volume holds the model weights, and Elastic Fabric Adapter EFA connects the GPU nodes. The following figure shows a standard Amazon EKS cluster. NVIDIA Dynamo serves the model aggregated on a single node or disaggregated with prefill and decode on different nodes. For vision-language models, the image/video encoding stage can in addition run disaggregated on its own node. Validation and performance This architecture has been validated on both p6-b200.48xlarge 8 x NVIDIA B200 GPUs and p5en.48xlarge 8 x NVIDIA H200 GPUs instances. You can validate the cluster and its network with an NVIDIA Collective Communications Library NCCL all-reduce micro-benchmark. Seeing utilization on all GPUs and traffic on all EFA adapters while the NCCL test is running, verifies that the cluster is working as expected. You can measure the performance of a deployed model on the cluster with the NVIDIA aiperf https://github.com/ai-dynamo/aiperf tool. The aiperf output in the upper left pane shows key performance indicators KPIs like time to first token TTFT , inter-token latency ITL , and throughput in tokens per second TPS . By taking these measurements at various levels of concurrency, we can study how latency and throughput depend on the number of concurrent requests. This information is essential to thoughtfully plan capacity and model deployment topology. The aws-do-eks container https://gallery.ecr.aws/hpc-cloud/aws-do-eks provides automation for visualizing GPU and EFA utilization as well as running aiperf and concurrency sweeps with single commands. The preceding concurrency sweep diagram reveals that the inference of Nemotron 3.5 Super VL has lower TTFT and about the same ITL between aggregated and disaggregated deployments on two nodes. A chart like this is useful when making deployment decisions. Draw your latency budget as horizontal lines, for example TTFT under 600 ms about 1 second and ITL under 100 ms per token. The rightmost point under both lines is the throughput one replica of this topology can serve within the service level objective SLO , here c32, or about 320 tokens per second. To plan capacity, divide your target tokens per second by this per-replica throughput and find the number of prefill-decode pairs to deploy. Re-running the same sweep after a change precision, TP/PP shape, aggregated versus disaggregated, a new Dynamo or vLLM release makes the trade-off visible immediately. A curve that moves right and down is a win. Because TTFT and ITL are plotted separately, it also tells you which side to scale: rising ITL says add decode capacity, and rising TTFT says add prefill capacity. Conclusion This post covers the architecture and performance evaluation patterns for serving NVIDIA Nemotron models on Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS with NVIDIA Dynamo and vLLM. To get started, deploy Nemotron 3.5 Super VL in your own account, using the reference implementation https://github.com/aws-samples/aws-do-eks/tree/main/Container-Root/eks/deployment/inference/agentic-ai/nemotron . More inference blueprints are also available in the AI on EKS https://github.com/awslabs/ai-on-eks/tree/main/blueprints/inference project. To get further hands-on experience with inference on EKS, register for our upcoming workshops https://awsworkshop.io/generative-ai . Acknowledgements The authors would like to convey special thanks to everyone involved in developing, validating, reviewing, and publishing this content, including but not limited to: Eliuth Triana, Jiahong Liu, Kshitiz Gupta, Abhijeet Patil, Mike Stefaniak, Chris Splinter, Sheetal Joshi, Aditya Ramakrishnan, Natarajan Kumar, Anish Maddipoti.