Optimizing GPU Resource Planning with NVIDIA Dynamo A developer published a guide for deploying NVIDIA Dynamo's SLA Planner on Kubernetes to automate GPU worker scaling for LLM inference. The setup uses Prometheus monitoring, DynamoGraphDeploymentRequest (DGDR) configuration for automated profiling and deployment, and Grafana dashboards to visualize scaling decisions against time-to-first-token and inter-token latency targets. GPU resource allocation for LLM inference presents complex optimization challenges. Workloads exhibit variable request patterns, unpredictable sequence lengths, and strict latency requirements, making static GPU allocation inefficient. Overprovisioning wastes expensive compute resources, while underprovisioning violates service level agreements SLAs and degrades user experience. NVIDIA Dynamo's SLA Planner https://docs.nvidia.com/dynamo/components/planner automates GPU resource management through SLA-driven profiling and intelligent autoscaling, analyzing performance requirements time-to-first-token, inter-token latency , profiling workload characteristics, and dynamically adjusting GPU worker replicas to meet latency targets while minimizing resource consumption. This guide covers deploying Dynamo's SLA Planner on Kubernetes, including infrastructure setup with Prometheus monitoring, DynamoGraphDeploymentRequest DGDR configuration for automated profiling and deployment, and Grafana dashboard integration for visualizing scaling decisions and performance metrics. By the end, you'll have a Kubernetes-based LLM deployment that automatically scales GPU workers to meet SLA targets. Before you begin, ensure you: This deployment uses several components that work together to provide automated GPU resource planning. 1. List GPU nodes in your cluster: bash $ kubectl get nodes -o json | jq '.items | select .status.capacity."nvidia.com/gpu" | {name: .metadata.name, gpus: .status.capacity."nvidia.com/gpu"}' The output displays nodes with available GPUs. { "name": "gpu-node-1", "gpus": "2" } { "name": "gpu-node-2", "gpus": "2" } 2. Verify GPU resources are allocatable: bash $ kubectl describe nodes | grep -A 5 "nvidia.com/gpu" Ensure at least one node shows nvidia.com/gpu: 1 or higher under the Allocatable section. 1. Check if GPU nodes are already labeled: bash $ kubectl get nodes --show-labels | grep nvidia.com/gpu.present If the output displays nodes with the nvidia.com/gpu.present=true label, skip to the next section. If no output appears, proceed with labeling. 2. Identify your GPU nodes: bash $ kubectl get nodes 3. Label each GPU node with the Dynamo GPU label. Replace NODE NAME with your GPU node name: bash $ kubectl label node NODE NAME nvidia.com/gpu.present=true Repeat this command for each GPU node in your cluster. 4. Verify the labels were applied: bash $ kubectl get nodes --show-labels | grep nvidia.com/gpu.present 1. Clone the Dynamo repository if you haven't already: bash $ git clone https://github.com/ai-dynamo/dynamo.git $ cd dynamo 2. Check out the latest stable release. Visit the Dynamo releases page https://github.com/ai-dynamo/dynamo/releases to find the latest version: bash $ git checkout release/0.9.0 3. Run the pre-deployment check script: $ ./deploy/pre-deployment/pre-deployment-check.sh The script validates kubectl connectivity, default StorageClass configuration, GPU resources, and GPU Operator installation. ======================================== Dynamo Pre-Deployment Check Script ======================================== ✅ kubectl Connectivity: PASSED ✅ Default StorageClass: PASSED ✅ Cluster GPU Resources: PASSED ✅ GPU Operator: PASSED All pre-deployment checks passed Your cluster is ready for Dynamo deployment. If any checks fail, review the error messages and resolve the issues before proceeding. 1. Add the Prometheus Helm repository: bash $ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts $ helm repo update 2. Install the kube-prometheus-stack with configuration for Dynamo metric scraping: bash $ helm install prometheus -n monitoring --create-namespace prometheus-community/kube-prometheus-stack \ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \ --set-json 'prometheus.prometheusSpec.podMonitorNamespaceSelector={}' \ --set prometheus.prometheusSpec.probeNamespaceSelector.matchLabels=null 3. Verify Prometheus is running: bash $ kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus 1. Set environment variables for the installation: bash $ export NAMESPACE=dynamo-system $ export RELEASE VERSION=0.9.0 2. Navigate to the home directory: bash $ cd ~ 3. Fetch the Dynamo CRDs Helm chart: bash $ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE VERSION}.tgz 4. Install the CRDs: bash $ helm install dynamo-crds dynamo-crds-${RELEASE VERSION}.tgz 5. Fetch the Dynamo Platform Helm chart: bash $ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE VERSION}.tgz 6. Install the Dynamo Platform with the operator enabled and the Prometheus endpoint configured: bash $ helm upgrade --install dynamo-platform dynamo-platform-${RELEASE VERSION}.tgz \ --namespace ${NAMESPACE} \ --create-namespace \ --set dynamo-operator.controllerManager.manager.image.tag=0.9.0 \ --set grove.enabled=true \ --set kai-scheduler.enabled=true \ --set nats.reloader.image.tag=0.22.3 \ --set prometheusEndpoint=http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090 The command configures the following components: --namespace ${NAMESPACE} and --create-namespace : Installs the platform in the dynamo-system namespace, creating it if it doesn't exist. --set dynamo-operator.controllerManager.manager.image.tag=0.9.0 : Specifies the Dynamo Operator version that manages DGD and DGDR lifecycle operations. --set grove.enabled=true : Enables Grove, the distributed runtime that coordinates communication between frontend and worker services. --set kai-scheduler.enabled=true : Enables the KAI Kubernetes AI scheduler for advanced GPU scheduling and resource allocation. --set nats.reloader.image.tag=0.22.3 : Specifies the NATS configuration reloader version that automatically reloads NATS server configuration changes without downtime. --set prometheusEndpoint=... : Configures the SLA Planner to query inference metrics from the Prometheus service in the monitoring namespace. If the installation fails with a Grove webhook certificate conflict error, uninstall the release and reinstall: bash $ helm uninstall dynamo-platform --namespace ${NAMESPACE} Then run the installation command again. The warning "tls: failed to find any PEM data in certificate input" is harmless and can be ignored. 7. Verify the operator is running: bash $ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/name=dynamo-operator NAME READY STATUS RESTARTS AGE dynamo-platform-dynamo-operator-controller-manager-77464cd6pcjt 2/2 Running 0 45s 1. Export your NGC API key. Replace YOUR NGC API KEY with your NGC API key from the NGC portal https://org.ngc.nvidia.com/setup/api-key : bash $ export NGC API KEY=YOUR NGC API KEY 2. Export your Hugging Face token. Replace YOUR HF TOKEN with your Hugging Face access token: bash $ export HF TOKEN=YOUR HF TOKEN 3. Create a Docker registry secret for NGC access using the exported variable: bash $ kubectl create secret docker-registry nvcr-imagepullsecret \ --docker-server=nvcr.io \ --docker-username='$oauthtoken' \ --docker-password=${NGC API KEY} \ -n ${NAMESPACE} 4. Create a Hugging Face token secret for model downloads using the exported variable: bash $ kubectl create secret generic hf-token-secret \ --from-literal=HF TOKEN=${HF TOKEN} \ -n ${NAMESPACE} The AI Configurator performs offline profiling to generate optimal deployment configurations without running actual workloads. The aicSystem parameter specifies your GPU hardware type, and the aicBackendVersion parameter specifies the inference backend version for profiling data. | System | Description | Supported Backends | |---|---|---| | a100 sxm | NVIDIA A100 SXM | TensorRT-LLM | | b200 sxm | NVIDIA B200 SXM | SGLang, TensorRT-LLM | | gb200 sxm | NVIDIA GB200 SXM | TensorRT-LLM | | h100 sxm | NVIDIA H100 SXM | vLLM, SGLang, TensorRT-LLM | | h200 sxm | NVIDIA H200 SXM | vLLM, SGLang, TensorRT-LLM | | l40s | NVIDIA L40S | SGLang, TensorRT-LLM | | GPU System | Supported Versions | |---|---| | H100 SXM | 0.12.0 | | H200 SXM | 0.12.0 | Example configuration for vLLM: sweep: useAiConfigurator: true aicSystem: h200 sxm aicBackendVersion: "0.12.0" | GPU System | Supported Versions | |---|---| | B200 SXM | 0.5.6.post2 | | H100 SXM | 0.5.6.post2 | | H200 SXM | 0.5.6.post2 | | L40S | 0.5.5.post3 | Example configuration for SGLang: sweep: useAiConfigurator: true aicSystem: h200 sxm aicBackendVersion: "0.5.6.post2" | GPU System | Supported Versions | |---|---| | A100 SXM | 1.0.0 | | B200 SXM | 1.0.0rc6 , 1.2.0rc5 | | GB200 SXM | 1.0.0rc6 , 1.2.0rc5 | | H100 SXM | 1.0.0rc3 , 1.2.0rc5 | | H200 SXM | 1.0.0rc3 , 1.2.0rc5 | | L40S | 1.0.0 | Example configuration for TensorRT-LLM: sweep: useAiConfigurator: true aicSystem: h200 sxm aicBackendVersion: "1.2.0rc5" Match your configuration to your cluster's GPU hardware: identify your GPU type with kubectl describe nodes , select the matching aicSystem code, and choose a compatible backend version from the tables above. An unsupported combination causes the profiling job to fail with an error indicating no profiling data is available for that configuration. 1. Create a DGDR manifest file with SLA specifications for the NVIDIA Nemotron Nano 4B model using the vLLM backend: bash $ cat << 'EOF' nemotron-nano-4b-dgdr.yaml apiVersion: nvidia.com/v1alpha1 kind: DynamoGraphDeploymentRequest metadata: name: nemotron-nano-sla spec: model: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 backend: vllm autoApply: false profilingConfig: profilerImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 config: sla: isl: 2000 osl: 100 ttft: 300.0 itl: 30.0 sweep: useAiConfigurator: true aicSystem: h200 sxm aicHfId: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 aicBackendVersion: "0.12.0" planner: plannerMinEndpoint: 1 plannerMaxGpuBudget: 4 plannerAdjustmentInterval: 30 deploymentOverrides: workersImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 name: vllm-disagg EOF The manifest specifies a Nemotron Nano 4B model with SLA targets ISL 2000, OSL 100, TTFT 300ms, ITL 30ms , AI Configurator offline profiling for H200 SXM GPUs 20-30 seconds versus 2-4 hours for online profiling , a planner budget of 1-4 GPUs with 30-second adjustment intervals, and autoApply: false so you can review the generated DGD before deployment. AI Configurator System Selection : The aicSystem parameter must match your GPU hardware type. Available options are: a100 sxm , b200 sxm , gb200 sxm , h100 sxm , h200 sxm , and l40s . The aicBackendVersion parameter must match the available version for your selected system and backend combination. For vLLM backend: H100/H200 SXM supports version 0.12.0 . For SGLang: H100/H200 SXM and B200 SXM support version 0.5.6.post2 , L40S supports 0.5.5.post3 . For TensorRT-LLM: consult the AI Configurator documentation https://docs.nvidia.com/dynamo/components/profiler/profiler-guide profiling-method for supported versions. 2. Apply the DGDR manifest: bash $ kubectl apply -f nemotron-nano-4b-dgdr.yaml -n ${NAMESPACE} 3. Monitor the DGDR status: bash $ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -w The DGDR transitions through states: Pending → Profiling → Ready . NAME MODEL BACKEND STATE DGD-STATE AGE nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Pending 5s nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Profiling 10s nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Ready 35s 4. View detailed DGDR status and events: bash $ kubectl describe dgdr nemotron-nano-sla -n ${NAMESPACE} The output displays profiling results and the generated DGD configuration in the status.generatedDeployment field. 1. Extract the generated DGD configuration to a file: bash $ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -o jsonpath='{.status.generatedDeployment}' nemotron-nano-dgd-raw.yaml 2. Add required Pod metadata environment variables and the --trust-remote-code flag to worker services using jq: bash $ cat nemotron-nano-dgd-raw.yaml | jq '.spec.services |= with entries .value.extraPodSpec.mainContainer.env = {"name":"POD UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}, {"name":"POD NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}}, {"name":"POD NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}, {"name":"POD IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}} | if .value.componentType == "worker" then if .value.extraPodSpec.mainContainer.args // | any . == "--trust-remote-code" | not then .value.extraPodSpec.mainContainer.args += "--trust-remote-code" else . end else . end ' nemotron-nano-dgd.yaml This injects Pod metadata environment variables into all services for Grove distributed runtime coordination and adds the --trust-remote-code flag only to worker service arguments. 3. Apply the modified DGD configuration to deploy the model: bash $ kubectl apply -f nemotron-nano-dgd.yaml -n ${NAMESPACE} 4. Verify the DynamoGraphDeployment was created: bash $ kubectl get dgd -n ${NAMESPACE} NAME AGE vllm-disagg 30s 5. Check that the frontend and worker pods are running: bash $ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/part-of=vllm-disagg Wait until all pods show Running status before proceeding. Initial deployment may take several minutes as worker pods download the model and load it into GPU memory; monitor with kubectl get pods -n ${NAMESPACE} -w . 1. Get the frontend service name: bash $ kubectl get svc -n ${NAMESPACE} -l nvidia.com/dynamo-discovery-backend=kubernetes | grep frontend vllm-disagg-frontend ClusterIP 10.43.69.246