GPU resource allocation for LLM inference presents complex optimization challenges. Workloads exhibit variable request patterns, unpredictable sequence lengths, and strict latency requirements, making static GPU allocation inefficient. Overprovisioning wastes expensive compute resources, while underprovisioning violates service level agreements (SLAs) and degrades user experience. NVIDIA Dynamo's SLA Planner automates GPU resource management through SLA-driven profiling and intelligent autoscaling, analyzing performance requirements (time-to-first-token, inter-token latency), profiling workload characteristics, and dynamically adjusting GPU worker replicas to meet latency targets while minimizing resource consumption. This guide covers deploying Dynamo's SLA Planner on Kubernetes, including infrastructure setup with Prometheus monitoring, DynamoGraphDeploymentRequest (DGDR) configuration for automated profiling and deployment, and Grafana dashboard integration for visualizing scaling decisions and performance metrics. By the end, you'll have a Kubernetes-based LLM deployment that automatically scales GPU workers to meet SLA targets.
Before you begin, ensure you:
This deployment uses several components that work together to provide automated GPU resource planning.
1. List GPU nodes in your cluster:
$ kubectl get nodes -o json | jq '.items[] | select(.status.capacity."nvidia.com/gpu") | {name: .metadata.name, gpus: .status.capacity."nvidia.com/gpu"}'
The output displays nodes with available GPUs.
{
"name": "gpu-node-1",
"gpus": "2"
}
{
"name": "gpu-node-2",
"gpus": "2"
}
2. Verify GPU resources are allocatable:
$ kubectl describe nodes | grep -A 5 "nvidia.com/gpu"
Ensure at least one node shows nvidia.com/gpu: 1 or higher under the Allocatable section.
1. Check if GPU nodes are already labeled:
$ kubectl get nodes --show-labels | grep nvidia.com/gpu.present
If the output displays nodes with the nvidia.com/gpu.present=true label, skip to the next section. If no output appears, proceed with labeling.
2. Identify your GPU nodes:
$ kubectl get nodes
3. Label each GPU node with the Dynamo GPU label. Replace NODE_NAME with your GPU node name:
$ kubectl label node NODE_NAME nvidia.com/gpu.present=true
Repeat this command for each GPU node in your cluster.
4. Verify the labels were applied:
$ kubectl get nodes --show-labels | grep nvidia.com/gpu.present
1. Clone the Dynamo repository if you haven't already:
$ git clone https://github.com/ai-dynamo/dynamo.git
$ cd dynamo
2. Check out the latest stable release. Visit the Dynamo releases page to find the latest version:
$ git checkout release/0.9.0
3. Run the pre-deployment check script:
$ ./deploy/pre-deployment/pre-deployment-check.sh
The script validates kubectl connectivity, default StorageClass configuration, GPU resources, and GPU Operator installation.
✅ kubectl Connectivity: PASSED ✅ Default StorageClass: PASSED ✅ Cluster GPU Resources: PASSED ✅ GPU Operator: PASSED
All pre-deployment checks passed! Your cluster is ready for Dynamo deployment.
If any checks fail, review the error messages and resolve the issues before proceeding.
**1. Add the Prometheus Helm repository:**
``` bash
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
2. Install the kube-prometheus-stack with configuration for Dynamo metric scraping:
$ helm install prometheus -n monitoring --create-namespace prometheus-community/kube-prometheus-stack \
--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
--set-json 'prometheus.prometheusSpec.podMonitorNamespaceSelector={}' \
--set prometheus.prometheusSpec.probeNamespaceSelector.matchLabels=null
3. Verify Prometheus is running:
$ kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus
1. Set environment variables for the installation:
$ export NAMESPACE=dynamo-system
$ export RELEASE_VERSION=0.9.0
2. Navigate to the home directory:
$ cd ~
3. Fetch the Dynamo CRDs Helm chart:
$ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz
4. Install the CRDs:
$ helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz
5. Fetch the Dynamo Platform Helm chart:
$ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz
6. Install the Dynamo Platform with the operator enabled and the Prometheus endpoint configured:
$ helm upgrade --install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz \
--namespace ${NAMESPACE} \
--create-namespace \
--set dynamo-operator.controllerManager.manager.image.tag=0.9.0 \
--set grove.enabled=true \
--set kai-scheduler.enabled=true \
--set nats.re.image.tag=0.22.3 \
--set prometheusEndpoint=http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
The command configures the following components:
--namespace ${NAMESPACE} and --create-namespace: Installs the platform in the dynamo-system namespace, creating it if it doesn't exist.--set dynamo-operator.controllerManager.manager.image.tag=0.9.0: Specifies the Dynamo Operator version that manages DGD and DGDR lifecycle operations.--set grove.enabled=true: Enables Grove, the distributed runtime that coordinates communication between frontend and worker services.--set kai-scheduler.enabled=true: Enables the KAI (Kubernetes AI) scheduler for advanced GPU scheduling and resource allocation.--set nats.re.image.tag=0.22.3: Specifies the NATS configuration re version that automatically reloads NATS server configuration changes without downtime.--set prometheusEndpoint=...: Configures the SLA Planner to query inference metrics from the Prometheus service in the monitoring namespace.
If the installation fails with a Grove webhook certificate conflict error, uninstall the release and reinstall:
$ helm uninstall dynamo-platform --namespace ${NAMESPACE}
Then run the installation command again. The warning "tls: failed to find any PEM data in certificate input" is harmless and can be ignored.
7. Verify the operator is running:
$ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/name=dynamo-operator
NAME READY STATUS RESTARTS AGE
dynamo-platform-dynamo-operator-controller-manager-77464cd6pcjt 2/2 Running 0 45s
1. Export your NGC API key. Replace YOUR_NGC_API_KEY with your NGC API key from the NGC portal:
$ export NGC_API_KEY=YOUR_NGC_API_KEY
2. Export your Hugging Face token. Replace YOUR_HF_TOKEN with your Hugging Face access token:
$ export HF_TOKEN=YOUR_HF_TOKEN
3. Create a Docker registry secret for NGC access using the exported variable:
$ kubectl create secret docker-registry nvcr-imagepullsecret \
--docker-server=nvcr.io \
--docker-username='$oauthtoken' \
--docker-password=${NGC_API_KEY} \
-n ${NAMESPACE}
4. Create a Hugging Face token secret for model downloads using the exported variable:
$ kubectl create secret generic hf-token-secret \
--from-literal=HF_TOKEN=${HF_TOKEN} \
-n ${NAMESPACE}
The AI Configurator performs offline profiling to generate optimal deployment configurations without running actual workloads. The aicSystem parameter specifies your GPU hardware type, and the aicBackendVersion parameter specifies the inference backend version for profiling data.
| System | Description | Supported Backends |
|---|---|---|
a100_sxm |
NVIDIA A100 SXM | TensorRT-LLM |
b200_sxm |
NVIDIA B200 SXM | SGLang, TensorRT-LLM |
gb200_sxm |
NVIDIA GB200 SXM | TensorRT-LLM |
h100_sxm |
NVIDIA H100 SXM | vLLM, SGLang, TensorRT-LLM |
h200_sxm |
NVIDIA H200 SXM | vLLM, SGLang, TensorRT-LLM |
l40s |
NVIDIA L40S | SGLang, TensorRT-LLM |
| GPU System | Supported Versions |
|---|---|
| H100 SXM | 0.12.0 |
| H200 SXM | 0.12.0 |
Example configuration for vLLM:
sweep:
useAiConfigurator: true
aicSystem: h200_sxm
aicBackendVersion: "0.12.0"
| GPU System | Supported Versions |
|---|---|
| B200 SXM | 0.5.6.post2 |
| H100 SXM | 0.5.6.post2 |
| H200 SXM | 0.5.6.post2 |
| L40S | 0.5.5.post3 |
Example configuration for SGLang:
sweep:
useAiConfigurator: true
aicSystem: h200_sxm
aicBackendVersion: "0.5.6.post2"
| GPU System | Supported Versions |
|---|---|
| A100 SXM | 1.0.0 |
| B200 SXM | 1.0.0rc6 ,1.2.0rc5 |
| GB200 SXM | 1.0.0rc6 ,1.2.0rc5 |
| H100 SXM | 1.0.0rc3 ,1.2.0rc5 |
| H200 SXM | 1.0.0rc3 ,1.2.0rc5 |
| L40S | 1.0.0 |
Example configuration for TensorRT-LLM:
sweep:
useAiConfigurator: true
aicSystem: h200_sxm
aicBackendVersion: "1.2.0rc5"
Match your configuration to your cluster's GPU hardware: identify your GPU type with kubectl describe nodes, select the matching aicSystem code, and choose a compatible backend version from the tables above. An unsupported combination causes the profiling job to fail with an error indicating no profiling data is available for that configuration.
1. Create a DGDR manifest file with SLA specifications for the NVIDIA Nemotron Nano 4B model using the vLLM backend:
$ cat << 'EOF' > nemotron-nano-4b-dgdr.yaml
apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeploymentRequest
metadata:
name: nemotron-nano-sla
spec:
model: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1
backend: vllm
autoApply: false
profilingConfig:
profilerImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
config:
sla:
isl: 2000
osl: 100
ttft: 300.0
itl: 30.0
sweep:
useAiConfigurator: true
aicSystem: h200_sxm
aicHfId: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1
aicBackendVersion: "0.12.0"
planner:
plannerMinEndpoint: 1
plannerMaxGpuBudget: 4
plannerAdjustmentInterval: 30
deploymentOverrides:
workersImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
name: vllm-disagg
EOF
The manifest specifies a Nemotron Nano 4B model with SLA targets (ISL 2000, OSL 100, TTFT 300ms, ITL 30ms), AI Configurator offline profiling for H200 SXM GPUs (20-30 seconds versus 2-4 hours for online profiling), a planner budget of 1-4 GPUs with 30-second adjustment intervals, and autoApply: false so you can review the generated DGD before deployment.
AI Configurator System Selection: The aicSystem parameter must match your GPU hardware type. Available options are: a100_sxm, b200_sxm, gb200_sxm, h100_sxm, h200_sxm, and l40s. The aicBackendVersion parameter must match the available version for your selected system and backend combination. For vLLM backend: H100/H200 SXM supports version 0.12.0. For SGLang: H100/H200 SXM and B200 SXM support version 0.5.6.post2, L40S supports 0.5.5.post3. For TensorRT-LLM: consult the AI Configurator documentation for supported versions.
2. Apply the DGDR manifest:
$ kubectl apply -f nemotron-nano-4b-dgdr.yaml -n ${NAMESPACE}
3. Monitor the DGDR status:
$ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -w
The DGDR transitions through states: Pending → Profiling → Ready.
NAME MODEL BACKEND STATE DGD-STATE AGE
nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Pending 5s
nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Profiling 10s
nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Ready 35s
4. View detailed DGDR status and events:
$ kubectl describe dgdr nemotron-nano-sla -n ${NAMESPACE}
The output displays profiling results and the generated DGD configuration in the status.generatedDeployment field.
1. Extract the generated DGD configuration to a file:
$ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -o jsonpath='{.status.generatedDeployment}' > nemotron-nano-dgd-raw.yaml
2. Add required Pod metadata environment variables and the --trust-remote-code flag to worker services using jq:
$ cat nemotron-nano-dgd-raw.yaml | jq '.spec.services |= with_entries(
.value.extraPodSpec.mainContainer.env = [
{"name":"POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},
{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},
{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},
{"name":"POD_IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}}
] |
if .value.componentType == "worker" then
if (.value.extraPodSpec.mainContainer.args // []) | any(. == "--trust-remote-code") | not then
.value.extraPodSpec.mainContainer.args += ["--trust-remote-code"]
else
.
end
else
.
end
)' > nemotron-nano-dgd.yaml
This injects Pod metadata environment variables into all services (for Grove distributed runtime coordination) and adds the --trust-remote-code flag only to worker service arguments.
3. Apply the modified DGD configuration to deploy the model:
$ kubectl apply -f nemotron-nano-dgd.yaml -n ${NAMESPACE}
4. Verify the DynamoGraphDeployment was created:
$ kubectl get dgd -n ${NAMESPACE}
NAME AGE
vllm-disagg 30s
5. Check that the frontend and worker pods are running:
$ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/part-of=vllm-disagg
Wait until all pods show Running status before proceeding. Initial deployment may take several minutes as worker pods download the model and load it into GPU memory; monitor with kubectl get pods -n ${NAMESPACE} -w.
1. Get the frontend service name:
$ kubectl get svc -n ${NAMESPACE} -l nvidia.com/dynamo-discovery-backend=kubernetes | grep frontend
vllm-disagg-frontend ClusterIP 10.43.69.246 <none> 8000/TCP 2m
2. Forward the frontend service to your local machine:
$ kubectl port-forward -n ${NAMESPACE} svc/vllm-disagg-frontend 8000:8000
Keep this terminal session running.
3. In a new terminal, send a test inference request:
$ curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
"messages": [
{
"role": "user",
"content": "Explain GPU resource planning in one sentence."
}
],
"max_tokens": 50,
"temperature": 0.7
}'
4. Press Ctrl+C in the port-forward terminal to stop forwarding.
Production Deployments: Port-forwarding is suitable for local testing only. For production deployments with secure external access, configure Gateway API with TLS encryption to expose the frontend service with HTTPS ingress, load balancing, and certificate management.
1. Apply the Grafana dashboard ConfigMap:
$ kubectl apply -n monitoring -f https://raw.githubusercontent.com/ai-dynamo/dynamo/release/0.9.0/deploy/observability/k8s/grafana-planner-dashboard-configmap.yaml
2. Expose the Grafana service using a NodePort for quick testing:
$ kubectl patch svc prometheus-grafana \
-n monitoring \
-p '{"spec": {"type": "NodePort"}}'
NodePort is for testing only. For production deployments with secure external access, configure Gateway API with TLS encryption to expose Grafana with HTTPS ingress and certificate management.
3. Get the NodePort assigned to Grafana:
$ kubectl get svc prometheus-grafana -n monitoring -o jsonpath='{.spec.ports[0].nodePort}'; echo
4. Retrieve the Grafana admin password:
$ kubectl get secret prometheus-grafana -n monitoring -o jsonpath='{.data.admin-password}' | base64 -d; echo
5. Access Grafana in your browser. Replace NODE_IP with your cluster node's IP address and NODE_PORT with the port from the previous step:
http://NODE_IP:NODE_PORT
Log in with username admin and the password retrieved in the previous step, then navigate to Dashboards → Dynamo Planner Dashboard to view planner metrics.
The SLA Planner continuously monitors request rates, input/output sequence lengths, TTFT, and ITL metrics from Prometheus, then adjusts prefill and decode worker replicas every adjustment interval (30 seconds in this configuration) to meet SLA targets.
As your application receives inference requests, the planner:
The dashboard visualizes worker counts, cumulative GPU-hours consumed, observed request/TTFT/ITL metrics, load predictions, correction factors, and a timeline of scaling decisions. Use the Namespace dropdown to filter metrics for your specific deployment namespace (dynamo-system in this guide).
Monitor planner logs:
$ kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=vllm-disagg-0-planner,app.kubernetes.io/part-of=vllm-disagg --tail=50 -f
2026-02-28T00:01:53.952354Z INFO planner_core.run: New adjustment interval started!
2026-02-28T00:01:53.962025Z INFO planner_core.observe_metrics: Observed num_req: 0.00 isl: 0.00 osl: 0.00
2026-02-28T00:01:53.962037Z INFO planner_core.observe_metrics: Observed ttft: 0.00ms itl: 0.00ms
2026-02-28T00:01:53.962463Z INFO planner_core.predict_load: Predicted load: num_req=0.00, isl=0.00, osl=0.00
2026-02-28T00:01:53.962592Z INFO planner_core._compute_replica_requirements: Predicted number of engine replicas: prefill=1, decode=1
2026-02-28T00:01:53.966200Z INFO kubernetes_connector.set_component_replicas: prefill component VllmPrefillWorker already at desired replica count 1, skipping
Press Ctrl+C to stop following the logs.
The SLA Planner uses load predictors to forecast future request patterns and proactively scale resources. Choose from four prediction models based on your workload's traffic patterns.
Constant — assumes the next interval's load equals the current load; use for stable workloads with long adjustment intervals.
planner:
loadPredictor: constant
ARIMA — uses auto-ARIMA to fit optimal model parameters for time-series data with trends and seasonality.
planner:
loadPredictor: arima
loadPredictorLog1p: true
Kalman — provides fast online forecasting with smooth adaptation using a local linear trend Kalman filter.
planner:
loadPredictor: kalman
kalmanQLevel: 0.1
kalmanQTrend: 0.01
kalmanR: 1.0
Prophet — handles complex seasonal patterns and trend changes using Facebook's Prophet model.
planner:
loadPredictor: prophet
prophetWindowSize: 1000
loadPredictorLog1p: true
plannerMaxGpuBudget and plannerAdjustmentInterval for your real traffic patterns instead of the fast demo values used here.
For the full guide with additional tips, visit the original article on Vultr Docs.