Standing Up a GPU Cluster on AKS for vLLM

Josef Doornink, an engineer, published a guide to standing up a GPU cluster on Azure Kubernetes Service (AKS) for serving vLLM models. The walkthrough covers requesting GPU quota, creating a cluster with a GPU node pool, and installing the NVIDIA device plugin to enable GPU scheduling. The guide emphasizes a methodical approach from model selection to infrastructure setup.

This article is Part of a series on running vLLM on AKS and walks through creating an AKS cluster with a GPU node pool, deploying vLLM onto it, and wiring up Prometheus and Grafana for visibility. Companion pieces: Choosing the right GPU https://dev.to/josef doornink 930b2caf1c/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess-4fe5 | Why your autoscaler flaps https://dev.to/josef doornink 930b2caf1c/your-vllm-autoscaler-is-flapping-because-you-picked-the-wrong-signal-not-the-wrong-number-24lf | Source https://github.com/JDoornink/vLLM on K8s/blob/main/write-up-infrastructure.md Standard NV36ads A10 v5 1× A10, 24 GB vllm/vllm-openai:latest serving Qwen/Qwen2.5-7B-Instruct-AWQ All commands below are bash. The steps are ordered and each one depends on the previous. The build order follows one chain: model → VRAM requirement → GPU SKU → region availability → quota. GPU quota. Request through Portal → Quotas → Compute → This article: Requested Standard NVADSA10v5 Family vCPUs = 108 in westus 108 = 3 nodes × 36 vCPUs, matching the autoscaler's max-count 3 set in step 3 . Quota is granted per-subscription and survives resource group deletion, so this step happens once, not on every rebuild. A quota is Azure's per-subscription limit on how much of a resource here, GPU vCPUs in a specific VM family you're allowed to provision at once. New subscriptions start at 0 for GPU families since it's expensive and can be abused. You need it because without an approval, az aks nodepool add for a GPU will fail outright. The request goes through manual Azure approval, so it has to happen before you plan to build. Prerequisites Local tooling: Bash Variables to set for use through the setup RG=<resource-group-name CLUSTER=<cluster-name LOCATION=<preferred-location az group create -n $RG -l $LOCATION az aks create -g $RG -n $CLUSTER \ --node-count 1 --node-vm-size Standard D2s v5 \ --generate-ssh-keys The GPU does not go on this pool. Every AKS cluster requires a system node pool for cluster-critical pods CoreDNS, metrics-server , and system pools cannot scale to zero — so a GPU placed here runs, and bills, 24/7 regardless of load. A D2s v5 CPU node covers the system pods cheaply; the GPU pool created in step 3 is where scale-to-zero actually happens. az aks nodepool add \ -g $RG --cluster-name $CLUSTER \ -n gpu \ --node-vm-size Standard NV36ads A10 v5 \ --node-count 0 --enable-cluster-autoscaler --min-count 0 --max-count 3 \ --node-taints sku=gpu:NoSchedule --labels sku=gpu What each flag does: --node-count 0 + --enable-cluster-autoscaler --min-count 0 --max-count 3 --node-taints sku=gpu:NoSchedule --labels sku=gpu nodeSelector targets in step 7.Taint, toleration, and nodeSelector do three separate jobs: the taint repels pods by default, a toleration permits a specific pod to ignore that taint, and a nodeSelector steers a pod toward a specific node. A toleration alone doesn't guarantee placement — it only lifts the block. The vLLM pod spec in step 7 carries both the toleration and the nodeSelector because both are required. The above scaling sets the --node-count and --min-count to 0; this may or may not be desirable for your use case. Keeping a node or 2 warm can help with latency, but there is cost associated with that. Choose whichever best fits your use case. az aks get-credentials -g $RG -n $CLUSTER kubectl get nodes expect only the system node — the GPU pool is still at 0 AKS does not install this by default. Without it, a GPU node never advertises nvidia.com/gpu as an allocatable resource, and any pod requesting nvidia.com/gpu: "1" stays Pending indefinitely with no error. kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml The upstream DaemonSet only tolerates the standard nvidia.com/gpu taint, not the custom sku=gpu taint set in step 3, so it won't schedule onto the GPU node without a patch: kubectl patch daemonset nvidia-device-plugin-daemonset -n kube-system --type=json \ -p=' {"op":"add","path":"/spec/template/spec/tolerations/-","value":{"operator":"Exists"}} ' A Pending vLLM pod looks identical whether the device plugin is missing, mis-scheduled, or the node just hasn't scaled up yet. kubectl describe node -l sku=gpu and checking for nvidia.com/gpu under Allocatable distinguishes between the three. NOTE: Installed before the GPU node scales up, so the stack builds on the free CPU pool. Values file: observability/kps-values.yaml — 6-hour retention, an 8 Gi PV on managed-csi , Grafana on ClusterIP , Alertmanager disabled. No StorageClass to create beforehand — AKS ships a built-in managed-csi class provisioner disk.csi.azure.com , WaitForFirstConsumer binding , which the Prometheus PVC above uses directly. Add the chart repos first one-time per workstation : helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo add kedacore https://kedacore.github.io/charts helm repo update helm upgrade --install kps prometheus-community/kube-prometheus-stack \ -n monitoring --create-namespace \ -f observability/kps-values.yaml --timeout 10m helm upgrade --install keda kedacore/keda -n keda --create-namespace --timeout 5m Verify: kubectl get pods -n monitoring prometheus, grafana, kube-state-metrics Running kubectl get pods -n keda keda-operator + metrics-apiserver Running kubectl api-resources | grep scaledobject confirms KEDA's CRDs landed prometheus-node-exporter is configured to tolerate sku=gpu operator: Exists in the values file, so it lands on the GPU node automatically once it scales up — no separate install step needed for CPU/memory/disk metrics from that node. Grafana's admin password is admin , set in the values file. Acceptable for a cluster torn down daily; not for anything long-lived. kubectl apply -f deployment.yaml kubectl get pods -w Expected sequence: Pending → cluster autoscaler provisions an A10 node ~3–5 min → ContainerCreating → image pull ~1 min, 8.8 GB → model weights load → 1/1 Running . The pod spec deployment.yaml https://github.com/JDoornink/vLLM on K8s/blob/main/deployment.yaml is where the taint/toleration/nodeSelector from step 3 get consumed: spec: tolerations: - key: sku operator: Equal value: gpu effect: NoSchedule matches --node-taints sku=gpu:NoSchedule on the GPU nodepool nodeSelector: sku: gpu matches --labels sku=gpu on the GPU nodepool containers: - name: vllm-gpu image: vllm/vllm-openai:latest args: - --model - Qwen/Qwen2.5-7B-Instruct-AWQ - --quantization - awq - --gpu-memory-utilization - "0.85" - --max-num-seqs - "32" resources: limits: nvidia.com/gpu: "1" ensures only one pod per GPU node Two settings worth explaining: --gpu-memory-utilization 0.85 , not the vLLM default of 0.92. nvidia.com/gpu: "1" resources.limits is what makes one-pod-per-node a scheduling constraint rather than a convention: Kubernetes tracks the node's GPU as consumed once this pod is placed, so a second replica can't land on the same node and the autoscaler brings up a new one instead. kubectl apply -f service.yaml https://github.com/JDoornink/vLLM on K8s/blob/main/service.yaml kubectl get endpoints vllm-openai-gpu must show pod IP:8080, confirming the selector matched kubectl port-forward svc/vllm-openai-gpu 8080:8080 & curl -s localhost:8080/v1/models | jq should list id "vllm-openai-gpu" The Service is ClusterIP — reachable only via port-forward , no public IP. Switch to type: LoadBalancer only if the endpoint needs to be reached from outside the cluster e.g., load-testing from a separate machine . vLLM's /metrics endpoint reports request/queue stats but nothing about the GPU itself — no utilization, VRAM, temperature, or power. NVIDIA's DCGM exporter is a DaemonSet that reads the GPU directly and exposes it to Prometheus. It requires the GPU node to already be up step 7 and, like the device plugin, must tolerate the sku=gpu taint to schedule there. helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts helm repo update gpu-helm-charts helm upgrade --install dcgm-exporter gpu-helm-charts/dcgm-exporter \ -n monitoring -f observability/dcgm-values.yaml --timeout 5m kubectl rollout status ds/dcgm-exporter -n monitoring --timeout=120s Two failure modes, both fixed in observability/dcgm-values.yaml : 1Gi limit. scrapeTimeout must be ≤ interval . scrapeTimeout to 25s; with interval: 15s , that combination makes the generated ServiceMonitor invalid, and the Prometheus operator drops the target with no visible error — the ServiceMonitor and Service objects both exist, Prometheus is healthy, but the target never appears. Fixed by setting serviceMonitor.scrapeTimeout: 10s .DCGM ships its own ServiceMonitor , discovered automatically because kps-values.yaml sets serviceMonitorSelectorNilUsesHelmValues: false Prometheus picks up every monitor object in the cluster, not just ones with a specific release label . vLLM needs a PodMonitor instead, since it's scraped directly on the pod's metrics port: kubectl apply -f observability/vllm-podmonitor.yaml Verify both targets are up , not just present: kubectl port-forward -n monitoring svc/kps-kube-prometheus-stack-prometheus 9090:9090 & curl -s http://localhost:9090/api/v1/targets \ | jq -r '.data.activeTargets | select .labels.job|test "vllm|dcgm";"i" | "\ .health \ .labels.job "' expect: up default/vllm-openai-gpu AND up dcgm-exporter curl -s 'http://localhost:9090/api/v1/query?query=DCGM FI DEV GPU UTIL' GPU utilization series curl -s 'http://localhost:9090/api/v1/query?query=vllm:num requests running' vLLM's own series kubectl port-forward -n monitoring svc/kps-grafana 3000:80 & browse http://localhost:3000 admin / admin GPU-level and application-level metrics now land in the same Prometheus, on the same time axis — the data the capacity-planning math in GPU sizing https://github.com/JDoornink/vLLM on K8s/blob/main/write-up-gpu-sizing.md and the autoscaler experiments in the flapping article https://github.com/JDoornink/vLLM on K8s/blob/main/write-up-flapping.md are built from. device plugin landed on the GPU node and is Running kubectl get pods -n kube-system -o wide | grep -i nvidia GPU is now advertised as an allocatable resource kubectl describe node -l sku=gpu | grep -A8 Allocatable expect: nvidia.com/gpu: 1 vLLM is actually serving kubectl logs -f -l app=vllm-openai-gpu look for "Application startup complete" kubectl get pods -o wide confirm one pod per GPU node If any of these fail, work backward through steps 5 → 3 rather than re-running the deployment. A Pending vLLM pod is almost always caused upstream of vLLM itself. az group delete -n $RG --yes --no-wait Deletes the cluster, both node pools, and the auto-created node resource group MC that holds the managed disks, including the Prometheus PV. The GPU quota grant from step 0 is untouched and persists for the next rebuild. To keep the cluster but stop GPU spend without a full teardown: kubectl scale deploy vllm-openai-gpu --replicas=0 the GPU pool's autoscaler drains the node 1→0 nvidia.com/gpu taint by default. --gpu-memory-utilization 0.85 , not the 0.92 default scrapeTimeout must be ≤ its scrape interval This gets the environment running. The remaining questions — how large a GPU the model actually needs, and what signal the autoscaler should watch — are covered in the companion articles GPU sizing //write-up-gpu-sizing.md and the flapping article //write-up-flapping.md .