This article is Part of a series on running vLLM on AKS and walks through creating an AKS cluster with a GPU node pool, deploying vLLM onto it, and wiring up Prometheus and Grafana for visibility.
Companion pieces:
Choosing the right GPU | Why your autoscaler flaps| Source
Standard_NV36ads_A10_v5
(1× A10, 24 GB)vllm/vllm-openai:latest
serving Qwen/Qwen2.5-7B-Instruct-AWQ
All commands below are bash. The steps are ordered and each one depends on the previous.
The build order follows one chain: model → VRAM requirement → GPU SKU → region availability → quota.
GPU quota. Request through Portal → Quotas → Compute →
This article: Requested Standard NVADSA10v5 Family vCPUs = 108
in westus
(108 = 3 nodes × 36 vCPUs, matching the autoscaler's max-count 3
set in step 3).
Quota is granted per-subscription and survives resource group deletion, so this step happens once, not on every rebuild.
A quota is Azure's per-subscription limit on how much of a resource (here, GPU vCPUs in a specific VM family) you're allowed to provision at once. New subscriptions start at 0 for GPU families since it's expensive and can be abused.
You need it because without an approval, az aks nodepool add for a GPU will fail outright. The request goes through manual Azure approval, so it has to happen before you plan to build.
**Prerequisites **
Local tooling:
Bash Variables to set for use through the setup
RG=<resource-group-name>
CLUSTER=<cluster-name>
LOCATION=<preferred-location>
az group create -n $RG -l $LOCATION
az aks create -g $RG -n $CLUSTER \
--node-count 1 --node-vm-size Standard_D2s_v5 \
--generate-ssh-keys
The GPU does not go on this pool. Every AKS cluster requires a system node pool for cluster-critical pods (CoreDNS, metrics-server), and system pools cannot scale to zero — so a GPU placed here runs, and bills, 24/7 regardless of load. A
D2s_v5
CPU node covers the system pods cheaply; the GPU pool created in step 3 is where scale-to-zero actually happens.
az aks nodepool add \
-g $RG --cluster-name $CLUSTER \
-n gpu \
--node-vm-size Standard_NV36ads_A10_v5 \
--node-count 0 --enable-cluster-autoscaler --min-count 0 --max-count 3 \
--node-taints sku=gpu:NoSchedule --labels sku=gpu
What each flag does:
--node-count 0
--enable-cluster-autoscaler
--min-count 0 --max-count 3
--node-taints sku=gpu:NoSchedule
--labels sku=gpu
nodeSelector
targets in step 7.Taint, toleration, and nodeSelector do three separate jobs: the taint repels pods by default, a toleration permits a specific pod to ignore that taint, and a nodeSelector steers a pod toward a specific node. A toleration alone doesn't guarantee placement — it only lifts the block. The vLLM pod spec in step 7 carries both the toleration and the nodeSelector because both are required.
The above scaling sets the --node-count and --min-count to 0; this may or may not be desirable for your use case. Keeping a node or 2 warm can help with latency, but there is cost associated with that. Choose whichever best fits your use case.
az aks get-credentials -g $RG -n $CLUSTER
kubectl get nodes # expect only the system node — the GPU pool is still at 0
AKS does not install this by default. Without it, a GPU node never advertises nvidia.com/gpu
as an allocatable resource, and any pod requesting nvidia.com/gpu: "1"
stays Pending
indefinitely with no error.
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml
The upstream DaemonSet only tolerates the standard nvidia.com/gpu
taint, not the custom sku=gpu
taint set in step 3, so it won't schedule onto the GPU node without a patch:
kubectl patch daemonset nvidia-device-plugin-daemonset -n kube-system --type=json \
-p='[{"op":"add","path":"/spec/template/spec/tolerations/-","value":{"operator":"Exists"}}]'
A Pending
vLLM pod looks identical whether the device plugin is missing, mis-scheduled, or the node just hasn't scaled up yet. kubectl describe node -l sku=gpu
and checking for nvidia.com/gpu
under Allocatable
distinguishes between the three.
NOTE: Installed before the GPU node scales up, so the stack builds on the free CPU pool. Values file: observability/kps-values.yaml
— 6-hour retention, an 8 Gi PV on managed-csi
, Grafana on ClusterIP
, Alertmanager disabled.
No StorageClass to create beforehand — AKS ships a built-in managed-csi
class (provisioner disk.csi.azure.com
, WaitForFirstConsumer
binding), which the Prometheus PVC above uses directly.
Add the chart repos first (one-time per workstation):
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm upgrade --install kps prometheus-community/kube-prometheus-stack \
-n monitoring --create-namespace \
-f observability/kps-values.yaml --timeout 10m
helm upgrade --install keda kedacore/keda -n keda --create-namespace --timeout 5m
Verify:
kubectl get pods -n monitoring # prometheus, grafana, kube-state-metrics Running
kubectl get pods -n keda # keda-operator + metrics-apiserver Running
kubectl api-resources | grep scaledobject # confirms KEDA's CRDs landed
prometheus-node-exporter
is configured to tolerate sku=gpu
(operator: Exists
) in the values file, so it lands on the GPU node automatically once it scales up — no separate install step needed for CPU/memory/disk metrics from that node.
Grafana's admin password is admin
, set in the values file. Acceptable for a cluster torn down daily; not for anything long-lived.
kubectl apply -f deployment.yaml
kubectl get pods -w
Expected sequence: Pending
→ cluster autoscaler provisions an A10 node (~3–5 min) → ContainerCreating
→ image pull (~1 min, 8.8 GB) → model weights load → 1/1 Running
.
The pod spec ([deployment.yaml](https://github.com/JDoornink/vLLM_on_K8s/blob/main/deployment.yaml)
) is where the taint/toleration/nodeSelector from step 3 get consumed:
spec:
tolerations:
- key: sku
operator: Equal
value: gpu
effect: NoSchedule # matches --node-taints sku=gpu:NoSchedule on the GPU nodepool
nodeSelector:
sku: gpu # matches --labels sku=gpu on the GPU nodepool
containers:
- name: vllm-gpu
image: vllm/vllm-openai:latest
args:
- --model
- Qwen/Qwen2.5-7B-Instruct-AWQ
- --quantization
- awq
- --gpu-memory-utilization
- "0.85"
- --max-num-seqs
- "32"
resources:
limits:
nvidia.com/gpu: "1" # ensures only one pod per GPU node
Two settings worth explaining:
--gpu-memory-utilization 0.85
, not the vLLM default of 0.92.nvidia.com/gpu: "1"
resources.limits
is what makes one-pod-per-node a scheduling constraint rather than a convention: Kubernetes tracks the node's GPU as consumed once this pod is placed, so a second replica can't land on the same node and the autoscaler brings up a new one instead.
kubectl apply -f [service.yaml](https://github.com/JDoornink/vLLM_on_K8s/blob/main/service.yaml)
kubectl get endpoints vllm-openai-gpu # must show pod IP:8080, confirming the selector matched
kubectl port-forward svc/vllm-openai-gpu 8080:8080 &
curl -s localhost:8080/v1/models | jq # should list id "vllm-openai-gpu"
The Service is ClusterIP
— reachable only via port-forward
, no public IP. Switch to type: LoadBalancer
only if the endpoint needs to be reached from outside the cluster (e.g., load-testing from a separate machine).
vLLM's /metrics
endpoint reports request/queue stats but nothing about the GPU itself — no utilization, VRAM, temperature, or power. NVIDIA's DCGM exporter is a DaemonSet that reads the GPU directly and exposes it to Prometheus. It requires the GPU node to already be up (step 7) and, like the device plugin, must tolerate the sku=gpu
taint to schedule there.
helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update gpu-helm-charts
helm upgrade --install dcgm-exporter gpu-helm-charts/dcgm-exporter \
-n monitoring -f observability/dcgm-values.yaml --timeout 5m
kubectl rollout status ds/dcgm-exporter -n monitoring --timeout=120s
Two failure modes, both fixed in observability/dcgm-values.yaml
:
1Gi
limit.scrapeTimeout
must be ≤ interval
.scrapeTimeout
to 25s; with interval: 15s
, that combination makes the generated ServiceMonitor
invalid, and the Prometheus operator drops the target with no visible error — the ServiceMonitor
and Service
objects both exist, Prometheus is healthy, but the target never appears. Fixed by setting serviceMonitor.scrapeTimeout: 10s
.DCGM ships its own ServiceMonitor
, discovered automatically because kps-values.yaml
sets serviceMonitorSelectorNilUsesHelmValues: false
(Prometheus picks up every monitor object in the cluster, not just ones with a specific release label). vLLM needs a PodMonitor
instead, since it's scraped directly on the pod's metrics port:
kubectl apply -f observability/vllm-podmonitor.yaml
Verify both targets are up
, not just present:
kubectl port-forward -n monitoring svc/kps-kube-prometheus-stack-prometheus 9090:9090 &
curl -s http://localhost:9090/api/v1/targets \
| jq -r '.data.activeTargets[] | select(.labels.job|test("vllm|dcgm";"i")) | "\(.health) \(.labels.job)"'
curl -s 'http://localhost:9090/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL' # GPU utilization series
curl -s 'http://localhost:9090/api/v1/query?query=vllm:num_requests_running' # vLLM's own series
kubectl port-forward -n monitoring svc/kps-grafana 3000:80 &
GPU-level and application-level metrics now land in the same Prometheus, on the same time axis — the data the capacity-planning math in GPU sizing and the autoscaler experiments in the flapping article are built from.
kubectl get pods -n kube-system -o wide | grep -i nvidia
kubectl describe node -l sku=gpu | grep -A8 Allocatable # expect: nvidia.com/gpu: 1
kubectl logs -f -l app=vllm-openai-gpu # look for "Application startup complete"
kubectl get pods -o wide # confirm one pod per GPU node
If any of these fail, work backward through steps 5 → 3 rather than re-running the deployment. A Pending
vLLM pod is almost always caused upstream of vLLM itself.
az group delete -n $RG --yes --no-wait
Deletes the cluster, both node pools, and the auto-created node resource group (MC_*
) that holds the managed disks, including the Prometheus PV. The GPU quota grant from step 0 is untouched and persists for the next rebuild.
To keep the cluster but stop GPU spend without a full teardown:
kubectl scale deploy vllm-openai-gpu --replicas=0 # the GPU pool's autoscaler drains the node 1→0
nvidia.com/gpu
taint by default.--gpu-memory-utilization 0.85
, not the 0.92 defaultscrapeTimeout
must be ≤ its scrape interval
This gets the environment running. The remaining questions — how large a GPU the model actually needs, and what signal the autoscaler should watch — are covered in the companion articles GPU sizing and the flapping article.