Kubernetes GPU Autoscaling: Scale GPU Capacity to Real Demand

Kubernetes GPU autoscaling adjusts GPU node and pod capacity to match spiky AI demand, scaling up for training and inference bursts and down to zero when idle. With average GPU utilization at just 5% across production clusters, autoscaling can dramatically cut costs, as idle H100s cost roughly $12.30 per GPU-hour on AWS. Tools like Karpenter and KEDA enable rapid provisioning and scale-to-zero, addressing the structural inefficiency that drives most GPU waste.

GPU autoscaling in Kubernetes adjusts GPU node and pod capacity to match spiky AI demand, scaling up for training and inference bursts and back down even to zero when idle. Because GPUs sit at about 5% utilization and cost far more than CPU, autoscaling GPU capacity is one of the highest-impact ways to cut AI spend. Key takeaways - GPU utilization averages 5% across production Kubernetes clusters, per the Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ - An idle H100 on AWS p5 costs ~$12.30/GPU-hr on-demand p5.48xlarge at ~$98.32/hr for 8 H100s ; 20 always-on idle H100s cost roughly $177K/month - GPU prices are rising: AWS raised H200 Capacity Block pricing 15% in January 2026 - Karpenter makes the cloud API call in under 60 seconds; the node reaches GPU-ready state in 3-5 minutes with a pre-baked GPU AMI - KEDA with minReplicaCount: 0 enables true scale-to-zero for GPU inference pods - Full cold-start from zero takes 8-15 minutes for large models; pod-only restarts run 30-60 seconds - Spot GPU instances offer 60-91% savings, but fewer than 2% of GPU workloads ran on Spot in 2025 Why GPU autoscaling matters You are paying for 20 GPUs and using one. That is not a figure of speech. The Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ found that average GPU utilization across production Kubernetes clusters sits at 5%. Therefore, 95% of provisioned GPU capacity is idle at any given moment. An idle H100 on AWS p5 costs roughly $12.30 per GPU per hour on-demand the p5.48xlarge lists at ~$98.32/hr for 8 H100s . At that rate, 20 always-on idle H100s burn around $177,000 per month before a single model finishes training. GPU prices are also heading in the wrong direction. AWS raised H200 Capacity Block prices 15% in January 2026, breaking a two-decade trend of falling compute costs. The good news is that this is mostly a technique problem, not a hardware problem. The best-performing cluster in the Cast AI dataset runs 136 H200s at 49% average GPU utilization. However, the fleet average is 5%. That 10x gap is almost entirely attributable to autoscaling, bin-packing, and GPU sharing. Better hardware does not close it. GPU autoscaling is the structural fix. It provisions GPU nodes when workloads need them and terminates those nodes when the work is done. For more on the full optimization picture, see Kubernetes GPU optimization https://cast.ai/blog/kubernetes-gpu-optimization/ . Node-level GPU autoscaling Karpenter vs. Cluster Autoscaler Two tools dominate node-level GPU autoscaling in Kubernetes: Karpenter and Cluster Autoscaler. Each responds to the same trigger, pending pods with nvidia.com/gpu resource requests, but they behave differently once that trigger fires. Karpenter is faster and more flexible. It calls cloud APIs directly, making the provisioning request in under 60 seconds; with a pre-baked GPU AMI, the node reaches GPU-ready state in 3-5 minutes. Additionally, Karpenter selects the optimal GPU instance type at scheduling time from a pool of options, so you do not need a separate node group per GPU SKU. Karpenter’s consolidation policy WhenUnderutilized reclaims idle GPU nodes after a configurable window, typically five minutes. Cluster Autoscaler takes a node-group approach. It adds nodes to a pre-defined group when pending pods appear. However, it cannot scale down based on GPU utilization alone. A node with near-zero GPU usage but any running pods stays up. For inference workloads with long idle gaps between requests, this is a meaningful cost leak. Karpenter GPU NodePool The following NodePool targets AWS GPU instance families, prefers Spot with on-demand fallback, and uses aggressive five-minute consolidation: apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: gpu-pool spec: template: spec: requirements: - key: karpenter.k8s.aws/instance-family operator: In values: "g5", "p4d", "p5" - key: karpenter.sh/capacity-type operator: In values: "spot", "on-demand" taints: - key: nvidia.com/gpu value: "true" effect: NoSchedule nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: gpu-node-class limits: nvidia.com/gpu: 16 disruption: consolidationPolicy: WhenUnderutilized consolidateAfter: 5m The NoSchedule taint is critical. It prevents CPU-only workloads from landing on GPU nodes and consuming expensive capacity. Only pods with a matching toleration will be scheduled there. For reliable Spot operation, configure Karpenter’s interruption handler: create an SQS queue named after your cluster and an EventBridge rule targeting it see the --interruption-queue flag . Without this, Spot termination signals are missed and the node is terminated without graceful drain. On GCP, use a2 or g2 instance families for A100/L4 nodes; on Azure, use ND H100 v5. The NodePool structure is identical, with only the instance family names differing. Also set a PodDisruptionBudget with minAvailable: 1 on inference Deployments to prevent Karpenter’s WhenUnderutilized consolidation from terminating your only warm replica during a compaction event. Spot GPU fallback Spot GPU instances offer 60-91% savings over on-demand pricing. However, fewer than 2% of GPU workloads ran on Spot in 2025. Availability is expanding, particularly for T4s in select regions. For training jobs and batch inference, Spot with on-demand fallback is practical. The Karpenter NodePool above handles this automatically through the capacity-type requirement listing both values. Karpenter tries Spot first and falls back to on-demand if Spot capacity is unavailable. Pod-level and event-driven autoscaling Before you begin The HPA and KEDA configurations below require three components already running in your cluster. Without them, the YAML examples are dead ends: NVIDIA GPU Operator installed on GPU nodes, exposing nvidia.com/gpu as a schedulable resource and activating the device plugin DCGM Exporter DaemonSet running on GPU nodes and scraped by Prometheus, making GPU metrics available at the Prometheus endpoint KEDA operator installed in the cluster required for the KEDA ScaledObject section HPA on DCGM GPU metrics Kubernetes HPA can scale GPU pods using custom metrics from NVIDIA DCGM Exporter. DCGM runs as a DaemonSet on GPU nodes and exports per-GPU metrics to Prometheus. The primary metric for compute utilization is DCGM FI PROF GR ENGINE ACTIVE , which measures the fraction of time the GPU’s Graphics/Compute engine is active, making it a reliable proxy for inference saturation. For finer-grained SM utilization tracking, use DCGM FI PROF SM ACTIVE instead. Bridging DCGM metrics into HPA requires the Prometheus Adapter, which translates Prometheus metrics into the Kubernetes custom metrics API. The following ConfigMap wires DCGM FI PROF GR ENGINE ACTIVE into a custom metric named dcgm gr engine active that the HPA can consume: prometheus-adapter-config.yaml partial apiVersion: v1 kind: ConfigMap metadata: name: adapter-config namespace: monitoring data: config.yaml: | rules: - seriesQuery: 'DCGM FI PROF GR ENGINE ACTIVE{namespace ="",pod =""}' resources: overrides: namespace: {resource: "namespace"} pod: {resource: "pod"} name: matches: "DCGM FI PROF GR ENGINE ACTIVE" as: "dcgm gr engine active" metricsQuery: 'avg < {< } by < ' With that ConfigMap applied, the following HPA scales a GPU inference deployment on engine utilization, targeting 70% average across pods: apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: gpu-inference-hpa namespace: inference spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: gpu-inference minReplicas: 1 maxReplicas: 8 metrics: - type: Pods pods: metric: name: dcgm gr engine active target: type: AverageValue averageValue: "0.70" Why 70%? Research shows GPU SM utilization above 70-75% correlates with inference latency degradation for transformer models. Adjust based on your P99 latency SLO. Note the hard constraint: minReplicas cannot go below 1 with native HPA. So HPA alone cannot scale GPU pods to zero. Also, Kubernetes allows only one custom metrics API server per cluster, which can conflict if other tools already use the Prometheus Adapter. KEDA for GPU inference KEDA Kubernetes Event-Driven Autoscaling is the stronger choice for most GPU inference workloads. Its Prometheus scaler queries DCGM metrics directly from your existing Prometheus instance, with no separate adapter layer required. More importantly, minReplicaCount: 0 enables true scale-to-zero. For vLLM inference, scaling on request queue depth is more semantically accurate than raw GPU utilization. GPU utilization can appear moderate even under heavy request queuing. Therefore, queue depth is the right signal. The following ScaledObject scales a vLLM deployment to zero when idle: apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: vllm-scaledobject namespace: inference spec: scaleTargetRef: name: vllm-llama3 minReplicaCount: 0 maxReplicaCount: 8 cooldownPeriod: 300 pollingInterval: 15 triggers: - type: prometheus metadata: serverAddress: http://prometheus.monitoring:9090 metricName: vllm queue depth vLLM exports vllm num requests waiting underscore separator . The query below assumes a recording rule: vllm:num requests waiting = vllm num requests waiting Use vllm num requests waiting directly if no recording rule is configured. query: | avg vllm:num requests waiting{namespace="inference",pod=~"vllm-llama3. "} threshold: "5" activationThreshold: "1" Why 5? Match this to your model’s per-replica inference capacity. Start low and tune upward if replicas cycle too frequently. The activationThreshold: "1" controls when KEDA activates the deployment: with 0 replicas idle, KEDA wakes the Deployment only when queue depth exceeds 1 pending request. Scale-down is governed separately by the main threshold value and the cooldownPeriod: 300 , which holds replicas up for five minutes after demand drops before scaling back down. Verify your setup Once deployed, confirm each layer is functioning before relying on autoscaling in production: Confirm GPU nodes are registered kubectl get nodes -l kubernetes.io/instance-type=p3.2xlarge Check HPA is reading GPU metrics kubectl describe hpa gpu-inference-hpa -n inference Inspect KEDA ScaledObject status kubectl get scaledobject vllm-scaler -n inference -o yaml | grep -A5 "conditions:" Confirm DCGM metrics flow into Prometheus curl http://prometheus:9090/api/v1/query?query=DCGM FI PROF GR ENGINE ACTIVE Choosing the right approach The table below compares the three autoscaling layers. In practice, node-level and pod-level autoscaling run together, not as alternatives. | Approach | Trigger | Min Replicas | Scale-to-Zero | Best For | |---|---|---|---|---| | Node Autoscaler Karpenter | Pending pods with GPU requests | 0 nodes | Yes node level | Node provisioning and termination | | Pod Autoscaling HPA | DCGM GPU metric via Prometheus Adapter | 1 pod | No | Steady-state inference serving scale-to-zero not required | | Event-Driven KEDA | Prometheus query, queue depth, external events | 0 pods | Yes | Inference serving, batch workloads | For broader Kubernetes autoscaling patterns beyond GPU workloads, the Kubernetes autoscaling for cloud cost optimization guide https://cast.ai/blog/guide-to-kubernetes-autoscaling-for-cloud-cost-optimization/ covers HPA, VPA, and KEDA across CPU and memory workloads. Scale-to-zero for inference True scale-to-zero means zero pods and zero running GPU nodes when demand is absent. For low-traffic or overnight inference endpoints, it eliminates idle GPU cost entirely. The tradeoff is cold-start latency, and it varies significantly depending on what is already warm. Cold-start timeline The full cold-start sequence from zero works as follows. First, KEDA detects demand and creates the pod, which stays Pending until a GPU node exists. Next, the node autoscaler provisions a GPU node: Karpenter makes the cloud API call in under 60 seconds, and the node reaches GPU-ready state in 3-5 minutes with a pre-baked GPU AMI. After that, the GPU Operator initializes drivers 1-2 minutes with a GPU-optimized AMI; 3-7 minutes on generic AMIs requiring kernel module compilation , the container image pulls 1-3 minutes , and model weights load into VRAM. For large models on fresh nodes, the total is typically 8-15 minutes. For a 7B parameter model with cached weights and a pre-warmed node already running, pod-only cold start runs 30-60 seconds. Node provisioning is the dominant cost in the full path. Mitigating cold-start latency Three techniques reduce cold-start time substantially. Pre-pulling container images onto nodes using a DaemonSet eliminates the image pull step. Using GPU-optimized AMIs with drivers pre-installed removes the driver initialization step. Caching model weights on a ReadOnlyMany PVC or object storage mount speeds up weight loading across restarts. For latency-sensitive endpoints, set minReplicaCount: 1 to hold one warm pod at all times. Reserve scale-to-zero for batch inference, overnight jobs, and endpoints where a 10-minute startup is acceptable. How automation handles the full loop Managing this stack manually is an ongoing job. Karpenter NodePools need tuning as models change. KEDA thresholds drift. Spot pools require region and capacity monitoring. The tools above each solve one layer well, but the integration work between them, including Spot selection, bin-packing, consolidation timing, and GPU sharing configuration, accumulates quickly. That work does not end at initial setup. Cast AI automates the full GPU node lifecycle: provisioning GPU instances when pods are pending, terminating GPU nodes when idle not just draining them , selecting optimal instance types across AWS and GCP, and applying Spot with on-demand fallback automatically. Beyond instance selection, Cast AI integrates with GPU time-slicing and MIG configurations to improve per-GPU utilization before adding new capacity. For a deeper look at those techniques, see Kubernetes GPU optimization https://cast.ai/blog/kubernetes-gpu-optimization/ . ALLEN Digital ran 7 inference models on SageMaker and moved them to Kubernetes with GPU time-slicing and Spot instances. The resulting savings came from three compounding factors: Spot instance pricing, GPU time-slicing across models, and bin-packing through model consolidation. Together, those techniques cut total GPU infrastructure cost by more than 70% compared to their SageMaker baseline, with no latency regression. The best-performing cluster in the Cast AI dataset runs 136 H200s at 49% GPU utilization, compared to the 5% fleet average. The gap is technique, not hardware, and it compounds at scale. Full benchmarking data is available in the State of Kubernetes Optimization report https://cast.ai/reports/state-of-kubernetes-optimization/ . Conclusion GPU autoscaling in Kubernetes is not optional at scale. At 5% average utilization and $12.30/GPU-hr on-demand for an H100 on AWS p5, idle capacity destroys AI infrastructure budgets quickly. GPU prices are also rising, not falling. The tools exist: Karpenter, KEDA, and DCGM Exporter each solve a real problem at their layer. Used together, they close most of the gap between 5% and 49% utilization. The remaining work, continuous Spot selection, bin-packing, and GPU sharing, is where automation earns its keep. Frequently Asked Questions What is GPU autoscaling in Kubernetes? GPU autoscaling in Kubernetes dynamically adjusts the number of GPU-backed pods and GPU nodes based on workload demand. At the node level, tools like Karpenter and Cluster Autoscaler provision GPU instances when pods with nvidia.com/gpu resource requests are pending, and terminate those nodes when idle. At the pod level, HPA with the Prometheus Adapter or KEDA with direct Prometheus queries adjusts replica counts based on GPU utilization, VRAM usage, or inference queue depth. Because GPUs average only 5% utilization in production Kubernetes clusters and cost $3-12+ per GPU-hour on-demand, GPU autoscaling is one of the highest-impact cost controls available to platform and ML ops teams. Can you scale GPUs to zero in Kubernetes? Yes. GPU workloads can scale to zero, but it requires two coordinated layers. First, KEDA with minReplicaCount: 0 scales pods to zero when demand drops. Next, the node autoscaler Karpenter with consolidationPolicy: WhenUnderutilized , or Cluster Autoscaler with scale-down enabled terminates the now-empty GPU node. The key tradeoff is cold-start latency: full provisioning from zero takes 8-15 minutes for large model workloads node boot plus driver init plus image pull plus model load . For latency-sensitive serving, keep minReplicaCount: 1 to hold one warm pod. Reserve scale-to-zero for batch workloads, overnight jobs, and non-critical inference endpoints. How do you autoscale GPU nodes in Kubernetes? First, install the NVIDIA GPU Operator to expose nvidia.com/gpu as a schedulable Kubernetes resource. Next, configure a node autoscaler: Karpenter preferred for provisioning speed and instance flexibility with a NodePool targeting GPU instance families, or Cluster Autoscaler with a dedicated GPU node group at min-nodes=0. Apply a NoSchedule taint on GPU nodes so only GPU-requesting pods land there. Finally, set a consolidation policy to terminate empty or underutilized GPU nodes. Karpenter uses consolidationPolicy: WhenUnderutilized with a short consolidateAfter window five minutes is a practical default . The autoscaler responds to pending pods with GPU requests by provisioning, and to empty GPU nodes by terminating. HPA vs KEDA for GPU autoscaling: which should I use? Use HPA when you already have the Prometheus Adapter installed, do not need scale-to-zero, and your scaling logic maps to a single GPU metric. Use KEDA when you want to query Prometheus directly without a separate adapter, need minReplicaCount: 0 for true scale-to-zero, or need queue-based scaling signals. For GPU inference workloads, KEDA is almost always the better choice. Inference demand is spiky and benefits from scale-to-zero. Beyond raw GPU utilization, request queue depth is a more accurate scaling signal, and KEDA eliminates Prometheus Adapter operational overhead. HPA remains a reasonable choice for inference Deployments with predictable, sustained resource consumption where scale-to-zero is not required.