Your Kubernetes bill is large. Your utilization data tells a different story than you expect. Cast AI’s 2026 analysis of tens of thousands of production clusters on AWS, GCP, and Azure found average CPU utilization sitting at 8% fleet-wide, down from 10% the prior year. That gap between provisioned and consumed is where cloud budget disappears.
This post maps The 7 Kubernetes Cost Drivers, a diagnostic framework covering the seven categories responsible for most overspend. For each driver you get the symptom, the root cause, a detection command, and the fix. If you want the remediation playbook after this, the implementation guide lives at Cast AI’s Kubernetes cost optimization guide.
Key Takeaways #
Idle and underutilized nodes– Symptom: CPU utilization near 8% but node count holds steady. Fix: enable node consolidation with Karpenter or Cast AI, and tune scale-down thresholds.Full remediation guide.Over-requested pods– Symptom: pods request 3–8x more memory than they consume. Fix: run VPA in Auto mode and set requests to p95 actual plus 20% headroom.Untuned autoscaling– Symptom: replica counts swing erratically under steady load. Fix: separate HPA (replicas) from VPA (sizing), tune CA thresholds, use KEDA for event-driven scaling.On-demand instances instead of spot– Symptom: nearly every node is on-demand despite stateless workloads. Fix: use Karpenter or Cast AI to run spot with automated on-demand fallback.Hidden storage and egress costs– Symptom: orphaned PVCs and load balancers appear on bills after namespace deletion. Fix: set PVC reclaim policy to Delete, audit LBs monthly, enable Topology Aware Routing.Control plane fees– Symptom: EKS/GKE billing line items for clusters that hold only dev workloads. Fix: consolidate dev/test environments into namespaces on fewer clusters.Idle GPUs– Symptom: 5% average GPU utilization while H100 instances bill at $12.29/GPU/hour. Fix: enable GPU time-slicing and MIG partitioning, move eligible workloads to spot.
Driver 1: Idle and Underutilized Nodes #
Symptom
Node count stays flat or grows while actual CPU consumption hovers in single digits. Cloud bills reflect provisioned capacity, not consumed capacity.
Root Cause
Cluster Autoscaler (CA) scales out based on pod requests, not actual usage. Scale-down requires a node to stay underutilized for 10 minutes by default, a threshold most teams never tighten. Result: nodes added quickly, nodes removed slowly, utilization permanently depressed. According to the Cast AI 2026 State of Kubernetes Resource Optimization report, 8% average CPU utilization now represents fleet-wide reality across tens of thousands of clusters.
Detection
kubectl top nodes
kubectl describe nodes | grep -A5 'Allocated resources'
Fix
Enable bin-packing and consolidation. Karpenter’s consolidationPolicy: WhenUnderutilized
handles this natively. For Cluster Autoscaler, lower scale-down-utilization-threshold
to 0.5 or below and reduce the scale-down delay. Cast AI handles this autonomously, continuously, without manual threshold tuning, and without requiring you to choose the right instance type in advance.
Driver 2: Over-Requested Pods (Resource Over-Provisioning) #
Symptom
Pods consume a fraction of what they request. The scheduler reserves capacity that workloads never touch.
Root Cause
Engineers pad resource requests to avoid OOMKill events. Helm charts ship with conservative defaults that rarely match production reality. Because Kubernetes cost is invisible to the developer writing the deployment manifest, there is no feedback loop. The Cast AI 2026 optimization press release puts CPU overprovisioning at 69% of clusters (up from 40% year-over-year) and memory overprovisioning at 79%. Cast AI’s own telemetry across customer clusters found 68% of pods wasting 3–8x more memory than they actually consume.
Detection
kubectl top pods --all-namespaces
Run VPA in Off mode first to collect recommendations without changing requests:
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: my-app-vpa
namespace: default
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
updatePolicy:
updateMode: "Off" # Recommendation only, no automatic restarts
Fix
Switch VPA to Auto or Recreate mode once you have enough data. Target requests at p95 actual consumption plus 20% headroom rather than a fixed multiplier of some worst-case estimate. Tools like VPA in Recreate mode or Goldilocks (an open-source VPA recommendation tool) track actual usage and adjust requests in-band, which also reduces OOMKill frequency because sizing decisions use real data instead of engineer guesses.
Driver 3: Untuned Autoscaling (HPA/VPA/CA Misconfiguration) #
Symptom
Replica counts fluctuate under stable load. Nodes provision and deprovision repeatedly within minutes. Alerts fire for instability that has no clear workload trigger.
Root Cause
Running HPA and VPA together without coordination creates a feedback loop. VPA changes pod resource requests; HPA watches CPU utilization as a percentage of those requests. When VPA raises requests, HPA sees utilization drop and scales in. When VPA lowers requests, HPA sees utilization spike and scales out. Neither controller knows the other changed the baseline. Cluster Autoscaler compounds the problem with its conservative 10-minute scale-down window.
Detection
kubectl get hpa --all-namespaces
kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml | grep -i scale-down
Fix
Separate concerns clearly: HPA controls replica count, VPA controls pod sizing, and they should not watch the same metric. Switch HPA to custom or external metrics (request rate, queue depth) rather than CPU percentage of requests. For event-driven workloads, KEDA gives you signal-based scaling without the HPA/VPA conflict. Tighten CA’s scale-down window to match your workload’s variance profile.
Note: VPA in Auto or Recreate mode and HPA targeting the same CPU/memory metrics are mutually exclusive – running both on the same Deployment causes the feedback loop described above. Use VPA for sizing and KEDA (not HPA) for replica scaling when combining vertical and horizontal autoscaling.
Driver 4: On-Demand Instances Instead of Spot/Preemptible #
Symptom
Your node inventory is almost entirely on-demand instances. Stateless, fault-tolerant workloads run on the most expensive instance class available.
Root Cause
Interruption risk feels too high without tooling to handle it. A spot interruption without automated rescheduling causes visible downtime. So teams default to on-demand and pay 60–90% more than necessary.
Detection
kubectl get nodes -L node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType
Fix
Automated interruption handling eliminates the risk. Karpenter handles spot-to-on-demand fallback natively. AWS Node Termination Handler gives Cluster Autoscaler-based clusters the same ability. According to Cast AI’s 2025 benchmark, mixed on-demand plus spot clusters average 59% savings; spot-only clusters reach 77%. Configure PodDisruptionBudgets so interruptions drain gracefully rather than abruptly.
Driver 5: Hidden Storage and Egress Costs #
Symptom (Storage)
PVCs appear in Released state long after the namespace that owned them was deleted. Load balancer line items accumulate in billing for services that no longer exist.
Root Cause (Storage)
PVC reclaim policy defaults to Retain. When a namespace is deleted, the PVC resource disappears from Kubernetes but the underlying cloud volume persists and keeps billing. The same happens with LoadBalancer services: the cloud-side NLB or CLB survives namespace deletion. A single orphaned NLB costs roughly $16/month; 30 orphaned load balancers reach $500/month.
Detection (Storage)
kubectl get pvc --all-namespaces | grep -v Bound
kubectl get pv | grep Released
kubectl get svc --all-namespaces | grep LoadBalancer
Fix (Storage)
Set the PVC reclaim policy to Delete for non-critical storage classes. Enforce namespace lifecycle policies and PVC audit scripts to catch orphaned volumes before they accumulate. Automate LB audits as part of your GitOps pipeline. Build namespace teardown runbooks that explicitly include cloud resource verification.
Symptom (Egress)
Cloud bills show consistent cross-AZ transfer charges that correlate with inter-pod traffic, not user-facing requests.
Root Cause (Egress)
Default Kubernetes scheduling ignores availability zone affinity. Pods for the same service land across AZs, and every RPC between them crosses an AZ boundary at $0.01/GB per direction ($0.02 round-trip). At scale, microservice architectures with frequent inter-service calls turn this into a meaningful monthly number. Internet egress rates compound the problem: AWS at $0.09/GB, Azure at $0.087/GB, GCP at $0.085/GB.
Detection (Egress)
Enable VPC Flow Logs and filter for cross-AZ traffic. Cilium Hubble and Istio both provide east-west traffic topology views that break down traffic by availability zone.
aws logs filter-log-events --log-group-name /aws/vpc/flowlogs \
--filter-pattern '[version, account, intf, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, start, end, action, status]' \
--query 'events[?contains(message, `az`) == `true`].message'
kubectl get pods --all-namespaces -o wide --show-labels | awk '{print $1, $8}' | sort | uniq -c
Fix (Egress)
Enable Topology Aware Routing (formerly Topology Aware Hints) to prefer same-AZ endpoints. Use topologySpreadConstraints
to co-locate communicating pods in the same zone. Move internal service-to-service traffic to private endpoints to eliminate internet egress charges entirely.
kubectl annotate svc my-service service.kubernetes.io/topology-mode=Auto
Driver 6: Control Plane Fees #
Symptom
Billing shows EKS or GKE management fees for clusters running mostly idle dev and test workloads.
Root Cause
Managed control plane pricing is a flat per-cluster fee regardless of workload volume. EKS charges $0.10/hour per cluster with no free tier, which totals roughly $72/month. GKE charges the same rate, though one zonal cluster qualifies for a billing credit. AKS offers a free control plane on the standard tier. Ten EKS clusters running dev environments cost $720/month before a single workload pod runs.
Detection
aws eks list-clusters --output table
gcloud container clusters list
az aks list --output table
Fix
Consolidate dev, test, and staging environments into namespaces on shared clusters rather than separate clusters for each environment. Delete clusters that have been idle for more than a sprint cycle. GitOps-driven cluster lifecycle management prevents idle clusters from persisting because cluster creation and deletion become pull-request-gated operations.
Driver 7: Idle GPUs #
Symptom
GPU instances appear fully allocated in Kubernetes but DCGM metrics show single-digit utilization. Each idle H100 on an AWS p5 instance bills at approximately $12.29/GPU/hour, or roughly $8,849/month.*
- AWS p5.48xlarge on-demand rate, January 2026. Spot pricing averages 60–70% lower.
Root Cause
The NVIDIA device plugin allocates one whole GPU per pod by default. Inference workloads are bursty: they spike during a request and sit idle between requests. Teams typically run one model per GPU instance rather than sharing the hardware across models. Cast AI’s 2026 report measured average GPU utilization at 5% across 23,000+ clusters. Under 2% of GPUs ran on spot instances in 2025, despite spot availability for GPU instance types on all three major clouds. AWS raised H200 GPU instance prices 15% in January 2026, breaking a 20-year trend of declining cloud compute prices.
Detection
kubectl describe nodes | grep -A10 'nvidia.com/gpu'
Fix
GPU time-slicing via nvidia.com/gpu.shared
lets multiple pods share a single GPU, which is the fastest path to higher utilization for inference workloads. For H100s, Multi-Instance GPU (MIG) partitioning provides hardware-level isolation with predictable memory and compute allocation per partition. Note: MIG partitioning is supported on NVIDIA A100 and H100 GPUs only (Ampere and Hopper architectures). T4 and V100 fleets should use GPU time-slicing instead. Right-size model server instances: a 7B parameter model does not need an 80GB A100. Cast AI’s GPU optimization combines time-slicing, bin-packing, and spot scheduling together.
ALLEN Digital provides a concrete reference point: they moved 7 AI models from SageMaker to Kubernetes with time-slicing and spot instances. Initial savings reached 20%. After consolidation, savings grew to 30–40%. Total savings versus SageMaker exceeded 70%, with latency maintained throughout.
Driver-to-Fix Reference Table #
| Cost Driver | Symptom | Quick Fix | Automation Option |
|---|---|---|---|
| Idle and underutilized nodes | 8% avg CPU, steady node count | Lower scale-down-utilization-threshold |
Cast AI continuous node consolidation |
| Over-requested pods | Requests 3–8x actual usage | VPA Auto mode, p95 + 20% headroom | VPA / Goldilocks open-source rightsizing |
| Untuned autoscaling | Erratic replica counts under stable load | Separate HPA/VPA concerns, KEDA | Cast AI intelligent autoscaling |
| On-demand instead of spot | 100% on-demand nodes, stateless workloads | Karpenter spot fallback + PDBs | Karpenter + spot (59–77% savings per Cast AI 2025 benchmark) |
| Hidden storage and egress | Orphaned PVCs, LBs, cross-AZ traffic charges | Reclaim policy Delete, Topology Aware Routing | Namespace lifecycle policies and PVC audit scripts |
| Control plane fees | EKS/GKE fees for dev clusters | Consolidate envs into namespaces | GitOps-driven cluster lifecycle |
| Idle GPUs | 5% avg GPU utilization, bursty inference | GPU time-slicing, MIG for H100s/A100s | Cast AI GPU optimization |
Conclusion #
Each driver compounds the others: overprovisioned pods fill underutilized nodes, mistuned autoscalers prevent consolidation, and on-demand spend crowds out spot capacity. The diagnostic framework above identifies where your cluster’s cost profile diverges from benchmarks.
Diagnosis is the first step. Remediation requires implementation depth on each driver. The step-by-step implementation guide at Cast AI’s Kubernetes cost optimization guide covers the remediation loop in detail, including how to sequence these fixes and measure savings as you go.
Frequently Asked Questions #
What are the biggest Kubernetes cost drivers? The seven biggest kubernetes cost drivers are: idle nodes, over-provisioned pod requests, untuned autoscaling, on-demand instances, hidden storage and egress fees, managed control plane charges, and idle GPUs. Cast AI’s 2026 analysis of tens of thousands of production clusters found average CPU utilization at just 8% fleet-wide, which means the majority of provisioned compute goes unused.
Where does the money go in Kubernetes? The largest share goes to compute nodes, most sitting unused. CPU utilization averages 8% fleet-wide and memory averages 20% according to Cast AI’s 2026 data. Beyond compute, the next biggest drains are storage orphans (PVCs and load balancers that survive namespace deletion), cross-AZ and internet egress, and control plane fees (EKS at $0.10/hr per cluster with no free tier).
How much can I save by switching to spot instances? Spot instances cost 60–90% less than on-demand. Cast AI’s 2025 benchmark across production clusters found mixed on-demand plus spot clusters averaged 59% savings; spot-only configurations averaged 77% savings. The key enabler is automated interruption handling so workloads reschedule without causing downtime when a spot instance is reclaimed.
Why is Kubernetes GPU utilization so low? Cast AI’s 2026 report measured GPU utilization at 5% across 23,000+ clusters. The primary cause is the default NVIDIA device plugin, which allocates an entire GPU per pod regardless of how much of that GPU the workload actually uses. Inference workloads are also bursty by nature: they spike during inference requests and sit idle between them. GPU time-slicing and MIG partitioning (for A100 and H100 GPUs) are the primary technical fixes.
What is the cheapest managed Kubernetes control plane? AKS offers a free control plane on the standard tier, making it the lowest-cost option for managed Kubernetes control plane fees. GKE charges $0.10/hr but provides one free zonal cluster per billing account via a $74.40/month billing credit. EKS charges $0.10/hr per cluster with no free tier, totaling approximately $72/month per cluster before any workload costs.