Build a Validated Kubernetes Runtime for NVIDIA Dynamo NVIDIA's AI Cluster Runtime (AICR), part of the NVIDIA DSX AI Factory Platform, now packages the NVIDIA Dynamo distributed inference framework and its Kubernetes dependencies into validated recipes with pinned component versions and compatibility checks. The AICR Dynamo recipe ships the dynamo-platform chart pinned to version 1.4.2, including the Dynamo operator, runtime images, and the nvidia.com/v1beta1 DynamoGraphDeployment CustomResourceDefinition, which replaces the former v1alpha1 API version. The recipe also bundles supporting infrastructure such as the NVIDIA GPU Operator, Node Feature Discovery, and Dynamic Resource Allocation for GPU allocation, giving platform teams a repeatable way to deploy and validate a Dynamo runtime. Teams building AI applications need a repeatable path from a model configuration to a working inference endpoint. NVIDIA Dynamo is a distributed inference framework that coordinates model serving across GPU workers. Deploying it on Kubernetes is only a first step. A successful model deployment requires more: GPU drivers, resource allocation, scheduling, networking, and monitoring must all work together. Resolving these dependencies takes time away from testing model behavior, serving performance, and application integration. The NVIDIA DSX AI Factory Platform https://docs.nvidia.com/dsx brings together technologies to help the ecosystem design, build, and operate more efficient AI factories, and Dynamo is one of them. To make management of those infrastructure dependencies easier, the platform contains NVIDIA AI Cluster Runtime AICR . It packages NVIDIA Dynamo and its supporting infrastructure into validated recipes. Each recipe defines a configuration for a specific environment and workload, including pinned component versions, compatibility constraints, and validation checks. AICR gives platform teams a repeatable way to deploy and check the runtime, providing developers with a consistent foundation for serving models and testing applications. This walkthrough covers the infrastructure that supports Dynamo on Kubernetes, compares two routing options, and shows how to deploy and validate the runtime with AICR. Why the underlying runtime matters for Dynamo Running a model with Dynamo on Kubernetes https://www.youtube.com/watch?v=JnsYIojzM5I starts with a DynamoGraphDeployment https://docs.nvidia.com/dynamo/latest/reference/api/kubernetes/dynamo-graph-deployment that specifies how vLLM https://github.com/vllm-project/vllm , SGLang https://github.com/sgl-project/sglang , or NVIDIA TensorRT LLM https://github.com/NVIDIA/TensorRT-LLM should run in pods on the cluster. But the graph is only the top of the stack. A production inference cluster also needs a runtime layer below it. It has to answer questions like: - Is the NVIDIA GPU Operator https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html installed with a driver stack that matches the nodes? - Are GPU nodes labeled and discoverable? - Does the cluster expose GPUs through the expected Dynamic Resource Allocation DRA driver? - Can the scheduler place GPU workers with the required topology and gang scheduling semantics? - Are node health signals available before workloads land on a degraded node? - Is Prometheus reachable for Dynamo and accelerator metrics? - Is the Gateway API / https://gateway-api-inference-extension.sigs.k8s.io/ Endpoint Picker https://gateway-api-inference-extension.sigs.k8s.io/ EPP https://gateway-api-inference-extension.sigs.k8s.io/ path configured when the deployment should be Kubernetes-native at ingress? AICR captures these runtime requirements in the recipe. What AICR ships for Dynamo An AICR Dynamo recipe packages both Dynamo and the cluster runtime components it depends on. AICR ships the dynamo-platform chart itself pinned to version 1.4.2 that includes the Dynamo operator, Dynamo runtime images, and, among others, the nvidia.com/v1beta1 DynamoGraphDeployment CustomResourceDefinition CRD https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ . v1beta1 replaces the former v1alpha1 API version. Table 1 lists the supporting infrastructure components included in the recipe. | Prerequisite | Requirement | Component in the AICR recipe | |---|---|---| | GPU driver stack | Driver and GPU software stack that matches the nodes | NVIDIA GPU Operator https://github.com/NVIDIA/gpu-operator | | GPU discovery | GPU nodes labeled and discoverable for placement | Node Feature Discovery https://github.com/kubernetes-sigs/node-feature-discovery | | GPU allocation | GPUs exposed through Kubernetes-native Dynamic Resource Allocation DRA | DRA Driver for NVIDIA GPUs https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu | | Scheduling | GPU workers placed with the required topology and gang scheduling | KAI Scheduler https://github.com/kai-scheduler/KAI-Scheduler | | Inference Workload Orchestration | Multi-component serving graphs started, placed, and scaled as coordinated groups | Grove https://github.com/ai-dynamo/grove | | Node health | Health signals available before workloads land on a degraded node | NVSentinel https://github.com/NVIDIA/NVSentinel and Nodewright https://github.com/NVIDIA/nodewright , where supported by the selected recipe | | Metrics | Cluster, accelerator, and Dynamo service metrics collected | Prometheus stack https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack | | Ingress | Gateway / EPP path configured when ingress is Kubernetes-native | agentgateway https://github.com/agentgateway/agentgateway and Gateway API resources, described in the routing sections | Table 1. Runtime prerequisites below Dynamo and the components the AICR recipe installs for each In addition, the recipe pins component versions and records configuration values, node selectors, tolerations, checksums, and validation expectations to make deployment reproducible. With Dynamo 1.4, frontends, workers, optional Gateway API Inference Extension GAIE / Endpoint Picker Plugin EPP components https://github.com/kubernetes-sigs/gateway-api-inference-extension , and other Dynamo services discover each other through Kubernetes resources. AICR’s default path therefore no longer uses NATS, and the dynamo-platform chart no longer installs it by default. NATS remains available as an opt-in for deployments that need it. Agentic workloads repeatedly extend long token prefixes, including system prompts, tool definitions, conversation history, and tool results. If a turn reaches a worker without that prefix in GPU memory, the system reprocesses intermediate tokens instead of reusing cached results. Dynamo routes requests to workers that already hold the relevant KV blocks, either through its frontend router or through GAIE/EPP integration at the Gateway API layer. Dynamo can also separate prompt processing prefill from token generation decode , an architecture known as disaggregated serving. On KV-cache misses, all or part of the prompt must be reprocessed by the prefill pool. Decode workers then take over token-by-token generation, which has different GPU, memory, latency, and batching characteristics. The cache miss rate depends on the workload. Long-running agentic sessions behave differently from one-off chat requests. Disaggregated serving enables operators to scale and tune prefill and decode capacity independently for a given request mix. The request plane and KV-cache event plane are separate. In Dynamo 1.4.2, the default AICR Kubernetes path uses TCP for requests and ZMQ for KV-cache events. Each vLLM worker engine emits raw KV-cache events through a local ZMQ https://zeromq.org/ publisher, for example, with the following vLLM configuration: --enable-prefix-caching --kv-transfer-config \ {"enable kv cache events": true, "publisher": "zmq", "endpoint": "tcp:// :5557"} Dynamo’s worker runtime then subscribes to that local vLLM event source, normalizes the events, and republishes them onto the Dynamo event plane. Both the native Dynamo Frontend router and the GAIE/EPP path use this Dynamo-managed KV-cache state to make routing decisions. What are the Dynamo routing paths on Kubernetes? Dynamo supports two routing paths on Kubernetes. The first uses the Dynamo Frontend router for worker selection. The second uses a Kubernetes Gateway API implementation for ingress and Dynamo EPP for endpoint selection. AICR ships and validates the Dynamo runtime for either topology. Path 1: Dynamo Frontend router The default path exposes a Dynamo Frontend behind a Kubernetes Service. Running with DYN ROUTER MODE=kv , the Frontend tokenizes each request, scores eligible workers using prefix overlap and current load, selects a worker, and dispatches the request through the Dynamo request plane, which uses TCP by default. This path is suitable when an application team owns a model-specific endpoint or an existing edge proxy only needs to forward traffic to the Dynamo Frontend Service. It is the smallest topology for deploying and validating a Dynamo graph and is the most production-proven serving option. It does not require the additional gateway and EPP components used in Path 2. Path 2: Gateway API Inference Extension and Dynamo EPP This path integrates Dynamo with the Gateway API and Gateway API Inference Extension GAIE . In the AICR example, agentgateway resolves an HTTPRoute to an InferencePool . The Dynamo operator creates that InferencePool from the DynamoGraphDeployment : its selector identifies eligible worker pods, while its endpointPickerRef identifies the Dynamo EPP Service. The gateway invokes the Dynamo EPP through GAIE’s endpoint-selection interface. The EPP tokenizes the request and uses Dynamo’s KV router to select an endpoint. It returns the selection and routing metadata to the gateway implementation, which is the actual HTTP proxy. In this mode, each routable worker pod includes a Dynamo Frontend sidecar that receives the request from the gateway and forwards it to the local worker container. The Frontend runs with --router-mode direct , so it applies the EPP selection without rerouting. This path is suitable when the platform team manages shared Kubernetes ingress and model endpoint worker selection, including timeouts, retries, and fallback, should be configured through Gateway API mechanisms. Note that both routing approaches can work behind a Gateway API HTTPRoute . The difference is which component manages the transport to the worker: the Dynamo Frontend or the Gateway itself. Note: This post covers Dynamo’s operator-managed DynamoGraphDeployment integration with GAIE; the experimental standalone vanilla vLLM on-ramp https://docs.nvidia.com/dynamo/v1.4.2/kubernetes/kv-aware-routing/vanilla-vllm-onramp is a separate integration. In Dynamo 1.4, this path uses GAIE’s Go EPP framework with Dynamo’s Rust KV router embedded through CGO and configured through eppConfig . Outlook: Starting with Dynamo 1.5, a native Rust EPP replaces the Go/CGO bridge. It implements GAIE’s Lightweight Endpoint Picker ext proc interface directly, runs the full Dynamo router in-process, and uses DYN environment variables in place of the eppConfig plugin and scheduling profiles. Aggregated and disaggregated routing are handled directly from the worker roles registered through Dynamo discovery. The surrounding Kubernetes topology remains unchanged. Removing the Go plugin framework and CGO boundary also reduces per-request EPP overhead. A client request enters through a gateway, such as agentgateway, and matches an HTTPRoute backed by an InferencePool . The Gateway invokes the Dynamo EPP, which performs just the token-aware KV selection and returns the selected endpoint and routing metadata. The Gateway then forwards the request to the selected worker’s Frontend sidecar in direct mode, which sends it to the vLLM worker over the Dynamo request plane. Worker KV-cache state reaches the EPP through the Dynamo event plane, while the Dynamo operator creates the EPP and InferencePool from the DynamoGraphDeployment . Figure 2. Gateway/EPP request routing through agentgateway, with live KV-cache events informing Dynamo endpoint selection before the request is forwarded to the selected vLLM worker. Choosing a path The routing path depends on whether the Gateway or the Dynamo Frontend should handle the forwarding of requests to the workers, including timeouts, retries, and fallback management. Table 2 compares the Dynamo Frontend and Gateway API/EPP paths. | Decision | Dynamo Router | GAIE / EPP | |---|---|---| | Ingress owner | Dynamo Frontend Service | Gateway and HTTPRoute | | Worker selection | The Frontend selects and dispatches to a worker | The EPP selects an endpoint; the Gateway forwards the request | | Request path | Client → Frontend → worker | Client → Gateway → worker Frontend sidecar → worker | | Best fit | A model-specific Dynamo endpoint with the smallest serving topology | A shared Kubernetes ingress managed through Gateway API resources | | Additional infrastructure | None beyond the Dynamo frontend and workers | GAIE-compatible Gateway, InferencePool, EPP, and worker frontend sidecars | | KV-cache behavior | Dynamo KV-aware routing from worker-published cache and load state | The same Dynamo KV-aware routing at the Gateway layer | Table 2. Decision criteria for the Dynamo router and GAIE / EPP routing paths Both paths use worker-published KV-cache events for precise cache-aware placement. Dynamo can instead be configured to predict KV-cache state from where it sent each request. Prediction cannot see evictions, so the router’s view can drift from what each worker actually holds, and routing quality can degrade, especially under memory pressure. The AICR configuration in this post enables live events, so both paths route using the workers’ reported KV-cache state. How the AICR recipe flow works AICR separates runtime setup from application deployment. First, it captures the existing Kubernetes cluster and generates a matching Dynamo recipe. The runtime bundle is then rendered, deployed, and validated before the application’s DynamoGraphDeployment is installed and its inference endpoints are tested. Step 1: Snapshot the cluster The snapshot captures the cluster state that AICR uses for recipe selection and validation: aicr snapshot \ --namespace aicr-validation \ --output snapshot.yaml When no placement flags are specified, AICR automatically targets nodes labeled nvidia.com/gpu.present=true and tolerates GPU-node taints. The --node-selector and --toleration flags provide more specific placement when required by the cluster. Step 2: Generate the Dynamo inference recipe AICR uses the snapshot to identify the Kubernetes distribution, GPU accelerator, and GPU-node operating system. The command specifies the workload intent and platform: aicr recipe \ --snapshot snapshot.yaml \ --intent inference \ --platform dynamo \ --output recipe.yaml The generated recipe records pinned component versions, environment-specific configuration, compatibility constraints, and validation checks. For the Amazon EKS Dynamo inference recipe for NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72 systems, AICR v1.0.0 selects the following Helm chart versions: | Component | Version | |---|---| | NVIDIA GPU Operator | v26.7.1 | | Node Feature Discovery | 0.19.0 | | cert-manager | v1.20.2 | | AWS EBS CSI Driver | 2.59.0 | | AWS EFA | v0.5.29 | | NVSentinel | v1.25.0 | | NodeWright Operator | v0.19.0 | | NVIDIA DRA GPU Driver | 0.5.0 | | KAI Scheduler | v0.16.9 | | Grove | v0.1.0-alpha.13 | | Dynamo Platform | 1.4.2 | | Agent Gateway | v1.5.0 | | Agent Gateway CRDs | v1.5.0 | | Kube Prometheus Stack | 84.4.0 | | Prometheus Operator CRDs | 28.0.1 | | Prometheus Adapter | 5.3.0 | | Kubernetes Ephemeral Storage Metrics | 1.19.2 | Table 3. Helm chart versions for the AICR v1.0.0 Amazon EKS Dynamo inference recipe The AICR team tested these component versions together on the specified platforms and hardware. The following validation steps verify that the cluster meets the selected recipe’s requirements. Optional offline check Recipe selection already uses these cluster facts in snapshot.yaml, so validating it against the same snapshot adds little. Offline validation is useful when testing a recipe against a different captured cluster snapshot without contacting Kubernetes. It evaluates declarative constraints, while behavioral checks are reported as skipped. aicr validate \ --recipe recipe.yaml \ --snapshot candidate-snapshot.yaml \ --no-cluster \ --output dry-run.json Step 3: Render the runtime bundle AICR renders the recipe into deployment-ready Helm values and manifests: aicr bundle \ --recipe recipe.yaml \ --accelerated-node-selector nodeGroup=gpu-worker \ --accelerated-node-toleration dedicated=worker-workload:NoSchedule \ --accelerated-node-toleration dedicated=worker-workload:NoExecute \ --system-node-selector nodeGroup=system-worker \ --system-node-toleration dedicated=system-workload:NoSchedule \ --system-node-toleration dedicated=system-workload:NoExecute \ --set-json agentgateway:allowedSourceRanges=' "