{"slug": "standing-up-a-gpu-cluster-on-aks-for-vllm", "title": "Standing Up a GPU Cluster on AKS for vLLM", "summary": "Josef Doornink, an engineer, published a guide to standing up a GPU cluster on Azure Kubernetes Service (AKS) for serving vLLM models. The walkthrough covers requesting GPU quota, creating a cluster with a GPU node pool, and installing the NVIDIA device plugin to enable GPU scheduling. The guide emphasizes a methodical approach from model selection to infrastructure setup.", "body_md": "This article is *Part of a series on running vLLM on AKS* and walks through creating an AKS cluster with a GPU node pool, deploying vLLM onto it, and wiring up Prometheus and Grafana for visibility.\n\nCompanion pieces:\n\n[Choosing the right GPU](https://dev.to/josef_doornink_930b2caf1c/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess-4fe5) | [Why your autoscaler flaps](https://dev.to/josef_doornink_930b2caf1c/your-vllm-autoscaler-is-flapping-because-you-picked-the-wrong-signal-not-the-wrong-number-24lf)| [Source](https://github.com/JDoornink/vLLM_on_K8s/blob/main/write-up-infrastructure.md)\n\n`Standard_NV36ads_A10_v5`\n\n(1× A10, 24 GB)`vllm/vllm-openai:latest`\n\nserving `Qwen/Qwen2.5-7B-Instruct-AWQ`\n\nAll commands below are bash. The steps are ordered and each one depends on the previous.\n\nThe build order follows one chain: **model → VRAM requirement → GPU SKU → region availability → quota.**\n\n**GPU quota.** Request through Portal → Quotas → Compute →\n\nThis article: Requested `Standard NVADSA10v5 Family vCPUs = 108`\n\nin `westus`\n\n(108 = 3 nodes × 36 vCPUs, matching the autoscaler's `max-count 3`\n\nset in step 3).\n\nQuota is granted per-subscription and survives resource group deletion, so this step happens once, not on every rebuild.\n\nA quota is Azure's per-subscription limit on how much of a resource (here, GPU vCPUs in a specific VM family) you're allowed to provision at once. New subscriptions start at 0 for GPU families since it's expensive and can be abused.\n\nYou need it because without an approval, az aks nodepool add for a GPU will fail outright. The request goes through manual Azure approval, so it has to happen before you plan to build.\n\n**Prerequisites **\n\nLocal tooling:\n\nBash Variables to set for use through the setup\n\n```\nRG=<resource-group-name>\nCLUSTER=<cluster-name>\nLOCATION=<preferred-location>\naz group create -n $RG -l $LOCATION\naz aks create -g $RG -n $CLUSTER \\\n  --node-count 1 --node-vm-size Standard_D2s_v5 \\\n  --generate-ssh-keys\n```\n\nThe GPU does not go on this pool. Every AKS cluster requires a system node pool for cluster-critical pods (CoreDNS, metrics-server), and system pools cannot scale to zero — so a GPU placed here runs, and bills, 24/7 regardless of load. A\n\n`D2s_v5`\n\nCPU node covers the system pods cheaply; the GPU pool created in step 3 is where scale-to-zero actually happens.\n\n```\naz aks nodepool add \\\n  -g $RG --cluster-name $CLUSTER \\\n  -n gpu \\\n  --node-vm-size Standard_NV36ads_A10_v5 \\\n  --node-count 0 --enable-cluster-autoscaler --min-count 0 --max-count 3 \\\n  --node-taints sku=gpu:NoSchedule --labels sku=gpu\n```\n\nWhat each flag does:\n\n`--node-count 0`\n\n+ `--enable-cluster-autoscaler`\n\n`--min-count 0 --max-count 3`\n\n`--node-taints sku=gpu:NoSchedule`\n\n`--labels sku=gpu`\n\n`nodeSelector`\n\ntargets in step 7.Taint, toleration, and nodeSelector do three separate jobs: the taint repels pods by default, a toleration permits a specific pod to ignore that taint, and a nodeSelector steers a pod toward a specific node. A toleration alone doesn't guarantee placement — it only lifts the block. The vLLM pod spec in step 7 carries both the toleration and the nodeSelector because both are required.\n\nThe above scaling sets the --node-count and --min-count to 0; this may or may not be desirable for your use case. Keeping a node or 2 warm can help with latency, but there is cost associated with that. Choose whichever best fits your use case.\n\n```\naz aks get-credentials -g $RG -n $CLUSTER\nkubectl get nodes          # expect only the system node — the GPU pool is still at 0\n```\n\nAKS does not install this by default. Without it, a GPU node never advertises `nvidia.com/gpu`\n\nas an allocatable resource, and any pod requesting `nvidia.com/gpu: \"1\"`\n\nstays `Pending`\n\nindefinitely with no error.\n\n```\nkubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml\n```\n\nThe upstream DaemonSet only tolerates the standard `nvidia.com/gpu`\n\ntaint, not the custom `sku=gpu`\n\ntaint set in step 3, so it won't schedule onto the GPU node without a patch:\n\n```\nkubectl patch daemonset nvidia-device-plugin-daemonset -n kube-system --type=json \\\n  -p='[{\"op\":\"add\",\"path\":\"/spec/template/spec/tolerations/-\",\"value\":{\"operator\":\"Exists\"}}]'\n```\n\nA `Pending`\n\nvLLM pod looks identical whether the device plugin is missing, mis-scheduled, or the node just hasn't scaled up yet. `kubectl describe node -l sku=gpu`\n\nand checking for `nvidia.com/gpu`\n\nunder `Allocatable`\n\ndistinguishes between the three.\n\nNOTE: Installed before the GPU node scales up, so the stack builds on the free CPU pool. Values file: `observability/kps-values.yaml`\n\n— 6-hour retention, an 8 Gi PV on `managed-csi`\n\n, Grafana on `ClusterIP`\n\n, Alertmanager disabled.\n\nNo StorageClass to create beforehand — AKS ships a built-in `managed-csi`\n\nclass (provisioner `disk.csi.azure.com`\n\n, `WaitForFirstConsumer`\n\nbinding), which the Prometheus PVC above uses directly.\n\nAdd the chart repos first (one-time per workstation):\n\n```\nhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts\nhelm repo add kedacore https://kedacore.github.io/charts\nhelm repo update\nhelm upgrade --install kps prometheus-community/kube-prometheus-stack \\\n  -n monitoring --create-namespace \\\n  -f observability/kps-values.yaml --timeout 10m\n\nhelm upgrade --install keda kedacore/keda -n keda --create-namespace --timeout 5m\n```\n\nVerify:\n\n```\nkubectl get pods -n monitoring          # prometheus, grafana, kube-state-metrics Running\nkubectl get pods -n keda                # keda-operator + metrics-apiserver Running\nkubectl api-resources | grep scaledobject   # confirms KEDA's CRDs landed\n```\n\n`prometheus-node-exporter`\n\nis configured to tolerate `sku=gpu`\n\n(`operator: Exists`\n\n) in the values file, so it lands on the GPU node automatically once it scales up — no separate install step needed for CPU/memory/disk metrics from that node.\n\nGrafana's admin password is `admin`\n\n, set in the values file. Acceptable for a cluster torn down daily; not for anything long-lived.\n\n```\nkubectl apply -f deployment.yaml\nkubectl get pods -w\n```\n\nExpected sequence: `Pending`\n\n→ cluster autoscaler provisions an A10 node (~3–5 min) → `ContainerCreating`\n\n→ image pull (~1 min, 8.8 GB) → model weights load → `1/1 Running`\n\n.\n\nThe pod spec (`[deployment.yaml](https://github.com/JDoornink/vLLM_on_K8s/blob/main/deployment.yaml)`\n\n) is where the taint/toleration/nodeSelector from step 3 get consumed:\n\n```\nspec:\n  tolerations:\n    - key: sku\n      operator: Equal\n      value: gpu\n      effect: NoSchedule   # matches --node-taints sku=gpu:NoSchedule on the GPU nodepool\n  nodeSelector:\n    sku: gpu               # matches --labels sku=gpu on the GPU nodepool\n  containers:\n    - name: vllm-gpu\n      image: vllm/vllm-openai:latest\n      args:\n        - --model\n        - Qwen/Qwen2.5-7B-Instruct-AWQ\n        - --quantization\n        - awq\n        - --gpu-memory-utilization\n        - \"0.85\"\n        - --max-num-seqs\n        - \"32\"\n      resources:\n        limits:\n          nvidia.com/gpu: \"1\"    # ensures only one pod per GPU node\n```\n\nTwo settings worth explaining:\n\n`--gpu-memory-utilization 0.85`\n\n, not the vLLM default of 0.92.`nvidia.com/gpu: \"1\"`\n\n`resources.limits`\n\nis what makes one-pod-per-node a scheduling constraint rather than a convention: Kubernetes tracks the node's GPU as consumed once this pod is placed, so a second replica can't land on the same node and the autoscaler brings up a new one instead.\n\n```\nkubectl apply -f [service.yaml](https://github.com/JDoornink/vLLM_on_K8s/blob/main/service.yaml)\nkubectl get endpoints vllm-openai-gpu        # must show pod IP:8080, confirming the selector matched\n\nkubectl port-forward svc/vllm-openai-gpu 8080:8080 &\ncurl -s localhost:8080/v1/models | jq        # should list id \"vllm-openai-gpu\"\n```\n\nThe Service is `ClusterIP`\n\n— reachable only via `port-forward`\n\n, no public IP. Switch to `type: LoadBalancer`\n\nonly if the endpoint needs to be reached from outside the cluster (e.g., load-testing from a separate machine).\n\nvLLM's `/metrics`\n\nendpoint reports request/queue stats but nothing about the GPU itself — no utilization, VRAM, temperature, or power. NVIDIA's DCGM exporter is a DaemonSet that reads the GPU directly and exposes it to Prometheus. It requires the GPU node to already be up (step 7) and, like the device plugin, must tolerate the `sku=gpu`\n\ntaint to schedule there.\n\n```\nhelm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts\nhelm repo update gpu-helm-charts\nhelm upgrade --install dcgm-exporter gpu-helm-charts/dcgm-exporter \\\n  -n monitoring -f observability/dcgm-values.yaml --timeout 5m\n\nkubectl rollout status ds/dcgm-exporter -n monitoring --timeout=120s\n```\n\nTwo failure modes, both fixed in `observability/dcgm-values.yaml`\n\n:\n\n`1Gi`\n\nlimit.`scrapeTimeout`\n\nmust be ≤ `interval`\n\n.`scrapeTimeout`\n\nto 25s; with `interval: 15s`\n\n, that combination makes the generated `ServiceMonitor`\n\ninvalid, and the Prometheus operator drops the target with no visible error — the `ServiceMonitor`\n\nand `Service`\n\nobjects both exist, Prometheus is healthy, but the target never appears. Fixed by setting `serviceMonitor.scrapeTimeout: 10s`\n\n.DCGM ships its own `ServiceMonitor`\n\n, discovered automatically because `kps-values.yaml`\n\nsets `serviceMonitorSelectorNilUsesHelmValues: false`\n\n(Prometheus picks up every monitor object in the cluster, not just ones with a specific release label). vLLM needs a `PodMonitor`\n\ninstead, since it's scraped directly on the pod's metrics port:\n\n```\nkubectl apply -f observability/vllm-podmonitor.yaml\n```\n\nVerify both targets are `up`\n\n, not just present:\n\n```\nkubectl port-forward -n monitoring svc/kps-kube-prometheus-stack-prometheus 9090:9090 &\ncurl -s http://localhost:9090/api/v1/targets \\\n  | jq -r '.data.activeTargets[] | select(.labels.job|test(\"vllm|dcgm\";\"i\")) | \"\\(.health)  \\(.labels.job)\"'\n# expect:  up  default/vllm-openai-gpu   AND   up  dcgm-exporter\n\ncurl -s 'http://localhost:9090/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL'      # GPU utilization series\ncurl -s 'http://localhost:9090/api/v1/query?query=vllm:num_requests_running' # vLLM's own series\nkubectl port-forward -n monitoring svc/kps-grafana 3000:80 &\n# browse http://localhost:3000  (admin / admin)\n```\n\nGPU-level and application-level metrics now land in the same Prometheus, on the same time axis — the data the capacity-planning math in [GPU sizing](https://github.com/JDoornink/vLLM_on_K8s/blob/main/write-up-gpu-sizing.md) and the autoscaler experiments in [the flapping article](https://github.com/JDoornink/vLLM_on_K8s/blob/main/write-up-flapping.md) are built from.\n\n```\n# device plugin landed on the GPU node and is Running\nkubectl get pods -n kube-system -o wide | grep -i nvidia\n\n# GPU is now advertised as an allocatable resource\nkubectl describe node -l sku=gpu | grep -A8 Allocatable      # expect: nvidia.com/gpu: 1\n\n# vLLM is actually serving\nkubectl logs -f -l app=vllm-openai-gpu                       # look for \"Application startup complete\"\nkubectl get pods -o wide                                     # confirm one pod per GPU node\n```\n\nIf any of these fail, work backward through steps 5 → 3 rather than re-running the deployment. A `Pending`\n\nvLLM pod is almost always caused upstream of vLLM itself.\n\n```\naz group delete -n $RG --yes --no-wait\n```\n\nDeletes the cluster, both node pools, and the auto-created node resource group (`MC_*`\n\n) that holds the managed disks, including the Prometheus PV. The GPU quota grant from step 0 is untouched and persists for the next rebuild.\n\nTo keep the cluster but stop GPU spend without a full teardown:\n\n```\nkubectl scale deploy vllm-openai-gpu --replicas=0    # the GPU pool's autoscaler drains the node 1→0\n```\n\n`nvidia.com/gpu`\n\ntaint by default.`--gpu-memory-utilization 0.85`\n\n, not the 0.92 default`scrapeTimeout`\n\nmust be ≤ its scrape `interval`\n\nThis gets the environment running. The remaining questions — how large a GPU the model actually needs, and what signal the autoscaler should watch — are covered in the companion articles [GPU sizing](//write-up-gpu-sizing.md) and [the flapping article](//write-up-flapping.md).", "url": "https://wpnews.pro/news/standing-up-a-gpu-cluster-on-aks-for-vllm", "canonical_source": "https://dev.to/josef_doornink_930b2caf1c/standing-up-a-gpu-cluster-on-aks-for-vllm-bif", "published_at": "2026-08-30 00:07:52+00:00", "updated_at": "2026-08-30 00:19:19.546884+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "developer-tools"], "entities": ["Josef Doornink", "Azure Kubernetes Service", "AKS", "vLLM", "NVIDIA", "Qwen", "Prometheus", "Grafana"], "alternates": {"html": "https://wpnews.pro/news/standing-up-a-gpu-cluster-on-aks-for-vllm", "markdown": "https://wpnews.pro/news/standing-up-a-gpu-cluster-on-aks-for-vllm.md", "text": "https://wpnews.pro/news/standing-up-a-gpu-cluster-on-aks-for-vllm.txt", "jsonld": "https://wpnews.pro/news/standing-up-a-gpu-cluster-on-aks-for-vllm.jsonld"}}