# Optimizing GPU Resource Planning with NVIDIA Dynamo > Source: > Published: 2026-10-29 17:06:22+00:00 GPU resource allocation for LLM inference presents complex optimization challenges. Workloads exhibit variable request patterns, unpredictable sequence lengths, and strict latency requirements, making static GPU allocation inefficient. Overprovisioning wastes expensive compute resources, while underprovisioning violates service level agreements (SLAs) and degrades user experience. [NVIDIA Dynamo's SLA Planner](https://docs.nvidia.com/dynamo/components/planner) automates GPU resource management through SLA-driven profiling and intelligent autoscaling, analyzing performance requirements (time-to-first-token, inter-token latency), profiling workload characteristics, and dynamically adjusting GPU worker replicas to meet latency targets while minimizing resource consumption. This guide covers deploying Dynamo's SLA Planner on Kubernetes, including infrastructure setup with Prometheus monitoring, DynamoGraphDeploymentRequest (DGDR) configuration for automated profiling and deployment, and Grafana dashboard integration for visualizing scaling decisions and performance metrics. By the end, you'll have a Kubernetes-based LLM deployment that automatically scales GPU workers to meet SLA targets. Before you begin, ensure you: This deployment uses several components that work together to provide automated GPU resource planning. **1. List GPU nodes in your cluster:** ``` bash $ kubectl get nodes -o json | jq '.items[] | select(.status.capacity."nvidia.com/gpu") | {name: .metadata.name, gpus: .status.capacity."nvidia.com/gpu"}' ``` The output displays nodes with available GPUs. ``` { "name": "gpu-node-1", "gpus": "2" } { "name": "gpu-node-2", "gpus": "2" } ``` **2. Verify GPU resources are allocatable:** ``` bash $ kubectl describe nodes | grep -A 5 "nvidia.com/gpu" ``` Ensure at least one node shows `nvidia.com/gpu: 1` or higher under the Allocatable section. **1. Check if GPU nodes are already labeled:** ``` bash $ kubectl get nodes --show-labels | grep nvidia.com/gpu.present ``` If the output displays nodes with the `nvidia.com/gpu.present=true` label, skip to the next section. If no output appears, proceed with labeling. **2. Identify your GPU nodes:** ``` bash $ kubectl get nodes ``` **3. Label each GPU node with the Dynamo GPU label. Replace `NODE_NAME` with your GPU node name:** ``` bash $ kubectl label node NODE_NAME nvidia.com/gpu.present=true ``` Repeat this command for each GPU node in your cluster. **4. Verify the labels were applied:** ``` bash $ kubectl get nodes --show-labels | grep nvidia.com/gpu.present ``` **1. Clone the Dynamo repository if you haven't already:** ``` bash $ git clone https://github.com/ai-dynamo/dynamo.git $ cd dynamo ``` **2. Check out the latest stable release. Visit the [Dynamo releases page](https://github.com/ai-dynamo/dynamo/releases) to find the latest version:** ``` bash $ git checkout release/0.9.0 ``` **3. Run the pre-deployment check script:** ``` $ ./deploy/pre-deployment/pre-deployment-check.sh ``` The script validates kubectl connectivity, default StorageClass configuration, GPU resources, and GPU Operator installation. ``` ======================================== Dynamo Pre-Deployment Check Script ======================================== ✅ kubectl Connectivity: PASSED ✅ Default StorageClass: PASSED ✅ Cluster GPU Resources: PASSED ✅ GPU Operator: PASSED All pre-deployment checks passed! Your cluster is ready for Dynamo deployment. ``` If any checks fail, review the error messages and resolve the issues before proceeding. **1. Add the Prometheus Helm repository:** ``` bash $ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts $ helm repo update ``` **2. Install the kube-prometheus-stack with configuration for Dynamo metric scraping:** ``` bash $ helm install prometheus -n monitoring --create-namespace prometheus-community/kube-prometheus-stack \ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \ --set-json 'prometheus.prometheusSpec.podMonitorNamespaceSelector={}' \ --set prometheus.prometheusSpec.probeNamespaceSelector.matchLabels=null ``` **3. Verify Prometheus is running:** ``` bash $ kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus ``` **1. Set environment variables for the installation:** ``` bash $ export NAMESPACE=dynamo-system $ export RELEASE_VERSION=0.9.0 ``` **2. Navigate to the home directory:** ``` bash $ cd ~ ``` **3. Fetch the Dynamo CRDs Helm chart:** ``` bash $ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz ``` **4. Install the CRDs:** ``` bash $ helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz ``` **5. Fetch the Dynamo Platform Helm chart:** ``` bash $ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz ``` **6. Install the Dynamo Platform with the operator enabled and the Prometheus endpoint configured:** ``` bash $ helm upgrade --install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz \ --namespace ${NAMESPACE} \ --create-namespace \ --set dynamo-operator.controllerManager.manager.image.tag=0.9.0 \ --set grove.enabled=true \ --set kai-scheduler.enabled=true \ --set nats.reloader.image.tag=0.22.3 \ --set prometheusEndpoint=http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090 ``` The command configures the following components: `--namespace ${NAMESPACE}` and `--create-namespace`: Installs the platform in the `dynamo-system` namespace, creating it if it doesn't exist.`--set dynamo-operator.controllerManager.manager.image.tag=0.9.0`: Specifies the Dynamo Operator version that manages DGD and DGDR lifecycle operations.`--set grove.enabled=true`: Enables Grove, the distributed runtime that coordinates communication between frontend and worker services.`--set kai-scheduler.enabled=true`: Enables the KAI (Kubernetes AI) scheduler for advanced GPU scheduling and resource allocation.`--set nats.reloader.image.tag=0.22.3`: Specifies the NATS configuration reloader version that automatically reloads NATS server configuration changes without downtime.`--set prometheusEndpoint=...`: Configures the SLA Planner to query inference metrics from the Prometheus service in the monitoring namespace. If the installation fails with a Grove webhook certificate conflict error, uninstall the release and reinstall: ``` bash $ helm uninstall dynamo-platform --namespace ${NAMESPACE} ``` Then run the installation command again. The warning `"tls: failed to find any PEM data in certificate input"` is harmless and can be ignored. **7. Verify the operator is running:** ``` bash $ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/name=dynamo-operator NAME READY STATUS RESTARTS AGE dynamo-platform-dynamo-operator-controller-manager-77464cd6pcjt 2/2 Running 0 45s ``` **1. Export your NGC API key. Replace `YOUR_NGC_API_KEY` with your NGC API key from the [NGC portal](https://org.ngc.nvidia.com/setup/api-key):** ``` bash $ export NGC_API_KEY=YOUR_NGC_API_KEY ``` **2. Export your Hugging Face token. Replace `YOUR_HF_TOKEN` with your Hugging Face access token:** ``` bash $ export HF_TOKEN=YOUR_HF_TOKEN ``` **3. Create a Docker registry secret for NGC access using the exported variable:** ``` bash $ kubectl create secret docker-registry nvcr-imagepullsecret \ --docker-server=nvcr.io \ --docker-username='$oauthtoken' \ --docker-password=${NGC_API_KEY} \ -n ${NAMESPACE} ``` **4. Create a Hugging Face token secret for model downloads using the exported variable:** ``` bash $ kubectl create secret generic hf-token-secret \ --from-literal=HF_TOKEN=${HF_TOKEN} \ -n ${NAMESPACE} ``` The AI Configurator performs offline profiling to generate optimal deployment configurations without running actual workloads. The `aicSystem` parameter specifies your GPU hardware type, and the `aicBackendVersion` parameter specifies the inference backend version for profiling data. | System | Description | Supported Backends | |---|---|---| | `a100_sxm` | NVIDIA A100 SXM | TensorRT-LLM | | `b200_sxm` | NVIDIA B200 SXM | SGLang, TensorRT-LLM | | `gb200_sxm` | NVIDIA GB200 SXM | TensorRT-LLM | | `h100_sxm` | NVIDIA H100 SXM | vLLM, SGLang, TensorRT-LLM | | `h200_sxm` | NVIDIA H200 SXM | vLLM, SGLang, TensorRT-LLM | | `l40s` | NVIDIA L40S | SGLang, TensorRT-LLM | | GPU System | Supported Versions | |---|---| | H100 SXM | `0.12.0` | | H200 SXM | `0.12.0` | **Example configuration for vLLM:** ``` sweep: useAiConfigurator: true aicSystem: h200_sxm aicBackendVersion: "0.12.0" ``` | GPU System | Supported Versions | |---|---| | B200 SXM | `0.5.6.post2` | | H100 SXM | `0.5.6.post2` | | H200 SXM | `0.5.6.post2` | | L40S | `0.5.5.post3` | **Example configuration for SGLang:** ``` sweep: useAiConfigurator: true aicSystem: h200_sxm aicBackendVersion: "0.5.6.post2" ``` | GPU System | Supported Versions | |---|---| | A100 SXM | `1.0.0` | | B200 SXM | `1.0.0rc6` ,`1.2.0rc5` | | GB200 SXM | `1.0.0rc6` ,`1.2.0rc5` | | H100 SXM | `1.0.0rc3` ,`1.2.0rc5` | | H200 SXM | `1.0.0rc3` ,`1.2.0rc5` | | L40S | `1.0.0` | **Example configuration for TensorRT-LLM:** ``` sweep: useAiConfigurator: true aicSystem: h200_sxm aicBackendVersion: "1.2.0rc5" ``` Match your configuration to your cluster's GPU hardware: identify your GPU type with `kubectl describe nodes`, select the matching `aicSystem` code, and choose a compatible backend version from the tables above. An unsupported combination causes the profiling job to fail with an error indicating no profiling data is available for that configuration. **1. Create a DGDR manifest file with SLA specifications for the NVIDIA Nemotron Nano 4B model using the vLLM backend:** ``` bash $ cat << 'EOF' > nemotron-nano-4b-dgdr.yaml apiVersion: nvidia.com/v1alpha1 kind: DynamoGraphDeploymentRequest metadata: name: nemotron-nano-sla spec: model: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 backend: vllm autoApply: false profilingConfig: profilerImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 config: sla: isl: 2000 osl: 100 ttft: 300.0 itl: 30.0 sweep: useAiConfigurator: true aicSystem: h200_sxm aicHfId: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 aicBackendVersion: "0.12.0" planner: plannerMinEndpoint: 1 plannerMaxGpuBudget: 4 plannerAdjustmentInterval: 30 deploymentOverrides: workersImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 name: vllm-disagg EOF ``` The manifest specifies a Nemotron Nano 4B model with SLA targets (ISL 2000, OSL 100, TTFT 300ms, ITL 30ms), AI Configurator offline profiling for H200 SXM GPUs (20-30 seconds versus 2-4 hours for online profiling), a planner budget of 1-4 GPUs with 30-second adjustment intervals, and `autoApply: false` so you can review the generated DGD before deployment. **AI Configurator System Selection**: The `aicSystem` parameter must match your GPU hardware type. Available options are: `a100_sxm`, `b200_sxm`, `gb200_sxm`, `h100_sxm`, `h200_sxm`, and `l40s`. The `aicBackendVersion` parameter must match the available version for your selected system and backend combination. For vLLM backend: H100/H200 SXM supports version `0.12.0`. For SGLang: H100/H200 SXM and B200 SXM support version `0.5.6.post2`, L40S supports `0.5.5.post3`. For TensorRT-LLM: consult the [AI Configurator documentation](https://docs.nvidia.com/dynamo/components/profiler/profiler-guide#profiling-method) for supported versions. **2. Apply the DGDR manifest:** ``` bash $ kubectl apply -f nemotron-nano-4b-dgdr.yaml -n ${NAMESPACE} ``` **3. Monitor the DGDR status:** ``` bash $ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -w ``` The DGDR transitions through states: `Pending` → `Profiling` → `Ready`. ``` NAME MODEL BACKEND STATE DGD-STATE AGE nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Pending 5s nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Profiling 10s nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Ready 35s ``` **4. View detailed DGDR status and events:** ``` bash $ kubectl describe dgdr nemotron-nano-sla -n ${NAMESPACE} ``` The output displays profiling results and the generated DGD configuration in the `status.generatedDeployment` field. **1. Extract the generated DGD configuration to a file:** ``` bash $ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -o jsonpath='{.status.generatedDeployment}' > nemotron-nano-dgd-raw.yaml ``` **2. Add required Pod metadata environment variables and the `--trust-remote-code` flag to worker services using jq:** ``` bash $ cat nemotron-nano-dgd-raw.yaml | jq '.spec.services |= with_entries( .value.extraPodSpec.mainContainer.env = [ {"name":"POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}, {"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}}, {"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}, {"name":"POD_IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}} ] | if .value.componentType == "worker" then if (.value.extraPodSpec.mainContainer.args // []) | any(. == "--trust-remote-code") | not then .value.extraPodSpec.mainContainer.args += ["--trust-remote-code"] else . end else . end )' > nemotron-nano-dgd.yaml ``` This injects Pod metadata environment variables into all services (for Grove distributed runtime coordination) and adds the `--trust-remote-code` flag only to worker service arguments. **3. Apply the modified DGD configuration to deploy the model:** ``` bash $ kubectl apply -f nemotron-nano-dgd.yaml -n ${NAMESPACE} ``` **4. Verify the DynamoGraphDeployment was created:** ``` bash $ kubectl get dgd -n ${NAMESPACE} NAME AGE vllm-disagg 30s ``` **5. Check that the frontend and worker pods are running:** ``` bash $ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/part-of=vllm-disagg ``` Wait until all pods show `Running` status before proceeding. Initial deployment may take several minutes as worker pods download the model and load it into GPU memory; monitor with `kubectl get pods -n ${NAMESPACE} -w`. **1. Get the frontend service name:** ``` bash $ kubectl get svc -n ${NAMESPACE} -l nvidia.com/dynamo-discovery-backend=kubernetes | grep frontend vllm-disagg-frontend ClusterIP 10.43.69.246 8000/TCP 2m ``` **2. Forward the frontend service to your local machine:** ``` bash $ kubectl port-forward -n ${NAMESPACE} svc/vllm-disagg-frontend 8000:8000 ``` Keep this terminal session running. **3. In a new terminal, send a test inference request:** ``` bash $ curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "messages": [ { "role": "user", "content": "Explain GPU resource planning in one sentence." } ], "max_tokens": 50, "temperature": 0.7 }' ``` **4. Press `Ctrl+C` in the port-forward terminal to stop forwarding.** **Production Deployments**: Port-forwarding is suitable for local testing only. For production deployments with secure external access, configure Gateway API with TLS encryption to expose the frontend service with HTTPS ingress, load balancing, and certificate management. **1. Apply the Grafana dashboard ConfigMap:** ``` bash $ kubectl apply -n monitoring -f https://raw.githubusercontent.com/ai-dynamo/dynamo/release/0.9.0/deploy/observability/k8s/grafana-planner-dashboard-configmap.yaml ``` **2. Expose the Grafana service using a NodePort for quick testing:** ``` bash $ kubectl patch svc prometheus-grafana \ -n monitoring \ -p '{"spec": {"type": "NodePort"}}' ``` NodePort is for testing only. For production deployments with secure external access, configure Gateway API with TLS encryption to expose Grafana with HTTPS ingress and certificate management. **3. Get the NodePort assigned to Grafana:** ``` bash $ kubectl get svc prometheus-grafana -n monitoring -o jsonpath='{.spec.ports[0].nodePort}'; echo ``` **4. Retrieve the Grafana admin password:** ``` bash $ kubectl get secret prometheus-grafana -n monitoring -o jsonpath='{.data.admin-password}' | base64 -d; echo ``` **5. Access Grafana in your browser. Replace `NODE_IP` with your cluster node's IP address and `NODE_PORT` with the port from the previous step:** ``` http://NODE_IP:NODE_PORT ``` Log in with username `admin` and the password retrieved in the previous step, then navigate to **Dashboards** → **Dynamo Planner Dashboard** to view planner metrics. The SLA Planner continuously monitors request rates, input/output sequence lengths, TTFT, and ITL metrics from Prometheus, then adjusts prefill and decode worker replicas every adjustment interval (30 seconds in this configuration) to meet SLA targets. As your application receives inference requests, the planner: The dashboard visualizes worker counts, cumulative GPU-hours consumed, observed request/TTFT/ITL metrics, load predictions, correction factors, and a timeline of scaling decisions. Use the **Namespace** dropdown to filter metrics for your specific deployment namespace (`dynamo-system` in this guide). **Monitor planner logs:** ``` bash $ kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=vllm-disagg-0-planner,app.kubernetes.io/part-of=vllm-disagg --tail=50 -f 2026-02-28T00:01:53.952354Z INFO planner_core.run: New adjustment interval started! 2026-02-28T00:01:53.962025Z INFO planner_core.observe_metrics: Observed num_req: 0.00 isl: 0.00 osl: 0.00 2026-02-28T00:01:53.962037Z INFO planner_core.observe_metrics: Observed ttft: 0.00ms itl: 0.00ms 2026-02-28T00:01:53.962463Z INFO planner_core.predict_load: Predicted load: num_req=0.00, isl=0.00, osl=0.00 2026-02-28T00:01:53.962592Z INFO planner_core._compute_replica_requirements: Predicted number of engine replicas: prefill=1, decode=1 2026-02-28T00:01:53.966200Z INFO kubernetes_connector.set_component_replicas: prefill component VllmPrefillWorker already at desired replica count 1, skipping ``` Press `Ctrl+C` to stop following the logs. The SLA Planner uses load predictors to forecast future request patterns and proactively scale resources. Choose from four prediction models based on your workload's traffic patterns. **Constant** — assumes the next interval's load equals the current load; use for stable workloads with long adjustment intervals. ``` planner: loadPredictor: constant ``` **ARIMA** — uses auto-ARIMA to fit optimal model parameters for time-series data with trends and seasonality. ``` planner: loadPredictor: arima loadPredictorLog1p: true ``` **Kalman** — provides fast online forecasting with smooth adaptation using a local linear trend Kalman filter. ``` planner: loadPredictor: kalman kalmanQLevel: 0.1 kalmanQTrend: 0.01 kalmanR: 1.0 ``` **Prophet** — handles complex seasonal patterns and trend changes using Facebook's Prophet model. ``` planner: loadPredictor: prophet prophetWindowSize: 1000 loadPredictorLog1p: true ``` `plannerMaxGpuBudget` and `plannerAdjustmentInterval` for your real traffic patterns instead of the fast demo values used here. For the full guide with additional tips, visit the original article on **[Vultr Docs](https://docs.vultr.com/how-to-optimize-gpu-resource-planning-with-nvidia-dynamo)**.