Key Takeaways #
- Average CPU utilization across production Kubernetes clusters sits at 8% in 2026 – down from 10% the prior year. Memory sits at 20%. Efficiency is moving in the wrong direction despite more tooling and awareness. ( Cast AI 2026 State of Kubernetes Optimization Report) - 69% of clusters over-provision CPU; 79% over-provision memory. The root cause is structural: inflated requests, slow scale-down, and no cost ownership at the team level. (Cast AI 2026 State of Kubernetes Optimization Report)
- 68% of pods waste 3-8x more memory than they consume. That waste compounds at every node the autoscaler adds in response to inflated requests. (Cast AI 2026 State of Kubernetes Optimization Report)
- Only 14% of teams implement chargeback for Kubernetes costs. Without ownership, no one fixes what no one sees. (FinOps Foundation 2025 State of FinOps Survey)
- EKS charges $0.10/hr per cluster – roughly $72/month per cluster, no free tier. Ten dev clusters waste $720/month before a single pod runs.
- Typical savings from systematic optimization reach 30-70%. Most of that gap lives in over-requested resource settings that have never been measured against actual usage. (Cast AI customer data, stateless workloads on EKS/GKE)
- For a SaaS company at $10M ARR with $200K/month in Kubernetes costs, a 40% reduction moves infrastructure from 24% to 14% of revenue – a line item a CFO will notice in a quarterly review.
Group 1: Visibility and Cost Monitoring (Checks 1–6) #
You cannot fix what you cannot see. Before touching any resource setting, establish per-workload, per-namespace cost attribution. Without a baseline, optimization actions produce no measurable outcome.
Check 1: Deploy kube-state-metrics and metrics-server
Both tools are prerequisites for HPA, VPA, and Prometheus-based cost visibility. Many clusters run neither by default. Verify first – every downstream check in this list depends on them.
kubectl get pods -n kube-system | grep -E 'metrics-server|kube-state-metrics'
Check 2: Install OpenCost or Kubecost for per-workload attribution
Cloud billing shows node-level totals. Workload-level attribution requires cluster-native tooling. OpenCost is CNCF-incubating and free.
helm repo add opencost https://opencost.github.io/opencost-helm-chart
helm install opencost opencost/opencost -n monitoring --create-namespace
Check 3: Audit CPU and memory utilization vs. requests
Fleet-wide CPU averages 8% (Cast AI 2026 State of Kubernetes Optimization Report). Knowing your cluster’s ratio is the starting data point. Start with real-time data, then move to historical percentiles over a 14-day window.
kubectl top nodes
kubectl top pods --all-namespaces --sort-by=cpu
sum(rate(container_cpu_usage_seconds_total{container!='', container!='POD'}[5m])) by (namespace) /
sum(kube_pod_container_resource_requests{resource='cpu', container!='POD'}) by (namespace)
Checks 4–6: Labels, Dashboards, and Waste Ranking
Check 4: Enforce labeling standards. Every workload must carry app
, team
, and env
labels. Missing labels mean costs roll up to “unallocated” – a number that grows silently.
kubectl get pods --all-namespaces -o json | \
jq '.items[] | select(.metadata.labels.team == null) | .metadata.name'
Check 5: Set up a cost dashboard with a 14–30 day rolling window. A single-day snapshot misses batch spikes and CI runs. Rightsizing from one day of data causes OOM kills during infrequent but normal load spikes.
quantile_over_time(0.95, rate(container_cpu_usage_seconds_total{container!='', container!='POD'}[5m])[14d:5m])
Check 6: Identify the top namespaces by estimated CPU waste – requests minus actual utilization rate. Prioritize by dollar waste, not alphabetically.
sort_desc(
sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)
- sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)
)
Cast AI: Cast AI provides real-time cost visibility across namespaces, workloads, and teams from the moment it installs – typically two minutes via a single script. Allocation Groups slice costs by any label combination without custom tooling or Prometheus query maintenance. Historical cost data is available immediately, not after a multi-day setup cycle.
Group 2: Pod Rightsizing (Checks 7–13) #
Over-requested pods are the single largest cost driver. 69% of clusters over-provision CPU; 79% over-provision memory (Cast AI 2026 State of Kubernetes Optimization Report). Rightsizing matches resource requests to actual measured consumption – not worst-case guesses from Helm chart defaults.
Check 7: Find pods with no resource requests
BestEffort pods have no requests set. They are the first evicted under node pressure, causing unexpected downtime. Find them before they cause problems.
kubectl get pods --all-namespaces -o json | \
jq '.items[] | select(.spec.containers[].resources.requests == null) |
{ns: .metadata.namespace, pod: .metadata.name}'
Check 8: Run VPA in Off mode first
Run VPA in recommendation-only mode for 7–14 days before applying changes. This collects actual usage data without modifying running pods. Applying VPA Auto on day one without this phase causes too-aggressive cuts and OOM kills.
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: my-app-vpa
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
updatePolicy:
updateMode: "Off"
Checks 9–11: Setting Requests and Limits Correctly
Check 9: Set CPU requests to p95 of actual usage plus 20% headroom. Set memory requests to p99 plus 25% headroom. CPU is compressible – the kernel throttles it. Memory is not – the kernel OOM-kills the container. Use different percentiles for each resource type.
quantile_over_time(0.95, rate(container_cpu_usage_seconds_total{container!='POD'}[5m])[14d:5m]) * 1.2
quantile_over_time(0.99, container_memory_working_set_bytes{container!='POD'}[14d:5m]) * 1.25
Check 10: Remove or raise CPU limits on latency-sensitive services. A pod with a 500m request and a 500m limit gets throttled at exactly 500m – even when the node has idle cores. The metric to watch is the throttling ratio: what fraction of scheduling periods was this container actually throttled?
sum(rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])) by (pod, namespace)
/
sum(rate(container_cpu_cfs_periods_total{container!=""}[5m])) by (pod, namespace)
Check 11: Always set memory limits. An uncapped pod can exhaust node memory and evict every neighboring pod. Never remove memory limits to improve performance – that creates a time bomb.
Checks 12–13: Namespace-Wide Recommendations and HPA Recalibration
Check 12: Use Goldilocks to surface namespace-level VPA recommendations via a web UI. Running VPA manually per deployment across 100+ workloads is not sustainable at scale.
helm install goldilocks fairwinds-stable/goldilocks -n goldilocks --create-namespace
Check 13: Recalibrate HPA minReplicas and maxReplicas after every rightsizing pass. Rightsizing changes the per-pod footprint. Therefore, HPA replica-to-utilization math shifts – and the scaler fires at wrong thresholds if targets are not updated.
Cast AI: Teams using Cast AI Workload Optimization see OOM kill events drop to near zero within 48 hours of activation because requests are continuously recalibrated against actual usage, not set-and-forgotten at deployment. In-place pod resizing avoids restarts for most workloads. For stateful workloads that cannot be evicted, Live Migration moves them between nodes without downtime.
Group 3: Autoscaling Configuration (Checks 14–20) #
Default autoscaling settings add nodes aggressively and remove them conservatively. Untuned HPA, VPA, and Cluster Autoscaler configurations are a primary source of excess node spend. For a deeper configuration reference, see the Kubernetes autoscaling guide for cost optimization.
Checks 14–16: HPA Coverage, Custom Metrics, and Conflict Avoidance
Check 14: Verify HPA is deployed on all stateless deployments with traffic variance. Static replica counts provision for peak capacity even during off-peak hours.
kubectl get hpa --all-namespaces
Check 15: Configure custom or external metrics for queue-driven workloads. CPU is a lagging indicator for event-driven systems. By the time CPU rises, the queue has already backed up. Use KEDA for queue depth or business metrics instead.
kubectl get scaledobject --all-namespaces # KEDA ScaledObject CRD
Check 16: Do not run HPA and VPA on the same metric. HPA adds replicas while VPA reduces per-pod requests. Together, they oscillate and produce unstable replica counts that are difficult to debug.
The recommended pattern: run VPA in Off mode to generate CPU/memory recommendations, apply those requests manually once every 1-2 weeks, and let HPA scale replica count on CPU utilization. Since Kubernetes 1.30, HPA and VPA can coexist on the same deployment when each targets a different signal – VPA on resource requests, HPA on replicas.
Checks 17–20: Scale-Down Tuning, Karpenter, PDBs, and Scale-to-Zero
Check 17: For Cluster Autoscaler, lower scale-down-utilization-threshold
to 0.5 and reduce scale-down-delay-after-add
from the default 10 minutes. Default settings accumulate idle node-hours after every traffic spike.
Check 18: If using Karpenter, enable consolidationPolicy: WhenUnderutilized
. Without this, Karpenter provisions efficiently but does not actively reclaim idle nodes after traffic drops. Note: consolidateAfter
is only valid for WhenEmpty
– for WhenUnderutilized
, consolidation is event-driven and fires automatically on new scheduling opportunities.
Note: Karpenter v1 GA (released mid-2024) added limited consolidateAfter support for WhenUnderutilized as well. If you are running Karpenter v1+, test carefully – the behavior differs from WhenEmpty.
spec:
disruption:
consolidationPolicy: WhenUnderutilized
budgets:
- nodes: "10%"
Check 19: Set PodDisruptionBudgets for all production workloads. Without PDBs, the autoscaler will evict pods without any availability guardrails – this is dangerous in production. Set maxUnavailable: 1
for production deployments, which allows the consolidation engine to drain nodes one pod at a time. Setting maxUnavailable: 0
blocks all evictions and will stall Karpenter consolidation entirely.
Check 20: Deploy KEDA for batch jobs, CI workloads, and queue-driven services. KEDA scales to zero replicas during idle periods, eliminating idle replica cost entirely for non-continuous workloads.
Cast AI: The manual alternative – tuning CA thresholds and Karpenter policies check by check – requires 2-4 weeks of platform engineering time per major cluster change and drifts continuously without automation. With Cast AI Autopilot, HPA parameters adjust automatically based on workload patterns, eliminating manual threshold tuning. Cast AI for Karpenter adds ML-based Spot prediction and real-time cost visibility on top of Karpenter. Cast AI integrates with KEDA without conflicts because it operates at the node provisioning layer while KEDA manages replica counts independently.
What this means in practice: A 50-node cluster on AWS (mixed m5.2xlarge at ~$0.38/hr) runs about $13,680/month. At 8% CPU utilization, roughly $12,600 of that spend is provisioned but mostly idle. Recovering 40% through rightsizing and bin-packing returns $5,040/month, or $60,480 annualized – before any Spot savings are counted. That is the math a FinOps champion brings to the CFO review.
Group 4: Node Efficiency (Checks 21–26) #
Nodes are where compute cost is billed. Most clusters run too many on-demand nodes, at the wrong size, with too much headroom. For teams running GPU workloads, also review the Kubernetes GPU optimization guide – a single idle H100 GPU costs approximately $8,850/month on-demand (AWS p5.48xlarge, divided across 8 GPUs). At 5% fleet utilization, that is $8,407 wasted per GPU per month.
Check 21: Audit nodes below 50% allocation
Nodes below 50% combined CPU and memory allocation are prime consolidation candidates. Teams add nodes to solve scheduling issues but rarely remove the ones left underutilized after the bottleneck resolves.
kubectl describe nodes | grep -A5 'Allocated resources'
kubectl top nodes
1 - (sum(kube_pod_container_resource_requests{resource='cpu'}) by (node) /
sum(kube_node_status_allocatable{resource='cpu'}) by (node))
Checks 22–24: Spot, Savings Plans, and Instance Diversity
Check 22: Enable Spot instances for stateless, fault-tolerant, and batch workloads. Spot saves 60–90% over on-demand. Regional interruption variance is large – T4 GPUs in eu-west-3 sustain 90%+ survival over 30 minutes, while the same type in us-east-1 sees sub-20% over 24 hours. Measure your specific region before assuming high risk.
Check 23: Cover 60–80% of baseline compute with Savings Plans or Reserved Instances. RI and SP discounts reach up to 72% for 1-year or 3-year terms. Apply these to predictable baseline workloads – not variable capacity.
Check 24: Use multi-instance-type node pools. Diversifying across 5–10 compatible instance types dramatically improves Spot availability. Single-instance-type Spot pools get fully interrupted simultaneously during capacity events.
kubectl get nodes -o custom-columns=\
NODE:.metadata.name,\
INSTANCE:.metadata.labels.'node\.kubernetes\.io/instance-type',\
SPOT:.metadata.labels.'karpenter\.sh/capacity-type'
Checks 25–26: Cluster Consolidation and Bin-Packing Ratio
Check 25: Consolidate dev, test, and staging workloads into namespaces on fewer clusters. EKS charges $0.10/hr per cluster – roughly $72/month per cluster with no free tier. Ten dev clusters spend $720/month on control plane fees before a single application pod runs.
Check 26: Verify bin-packing ratio. Nodes with 70%+ headroom are waste. Fragmented nodes – one or two pods per node – are common after repeated scale-up events without subsequent consolidation. Bin-packing resolves this without adding capacity.
Cast AI: Cast AI analyzes real-time Spot interruption signals across all availability zones and instance families before committing to a node – not just at provisioning time. It autonomously selects the optimal instance type across all available pools on AWS, GCP, Azure, and OKE – no manual instance selection required. Node consolidation continuously bin-packs pods onto fewer nodes and terminates empty ones. Live Migration handles stateful workloads that standard autoscalers cannot move.
Group 5: Storage and Network Costs (Checks 27–31) #
Storage and network costs accumulate invisibly until the monthly bill arrives. Orphaned PVCs, oversized load balancers, cross-AZ traffic, and unnecessary service mesh overhead compound silently. This group surfaces and eliminates them.
Checks 27–28: Orphaned PVCs and Reclaim Policies
Check 27: Audit all PersistentVolumeClaims. PVCs in Released or Pending state still accrue storage charges. After namespace deletion, PVCs with Retain reclaim policy survive and continue billing. Each orphaned gp3 volume costs approximately $0.08/GB/month indefinitely.
kubectl get pvc --all-namespaces | grep -v Bound
kubectl get pv | grep -E 'Released|Available'
Check 28: Set PVC StorageClass reclaim policy to Delete for non-critical ephemeral data. Retain creates orphan volumes at scale after deployments churn. Keep Retain only for stateful databases where data loss is unacceptable.
Checks 29–31: Cross-AZ Traffic, Load Balancers, and Storage Classes
Check 29: Enable Topology Aware Routing to keep pod-to-pod traffic within the same availability zone. Cross-AZ data transfer costs $0.01/GB on AWS. In addition, service meshes with mTLS often double inter-pod traffic volume – cross-AZ mesh traffic compounds that cost significantly.
kubectl annotate service my-service service.kubernetes.io/topology-mode=auto
Check 30: Audit all LoadBalancer Services for orphaned resources. Each AWS NLB costs approximately $16/month regardless of traffic. Orphaned LBs from deleted namespaces continue to bill. A cluster with 30 orphaned LBs wastes nearly $500/month.
kubectl get svc --all-namespaces --field-selector spec.type=LoadBalancer
Check 31: Match storage class to workload requirements. Premium SSD (Azure) or io2 (AWS) costs 3–5x more than gp3. Non-critical workloads – logs, cache, scratch storage – do not require provisioned IOPS. Moving them to sc1 or Standard_LRS reduces storage costs 40–60% for those workloads.
Cast AI: Cast AI surfaces orphaned PVCs and unused LoadBalancer resources during cluster analysis. Node consolidation reduces cross-AZ traffic by co-locating communicating pods. Zonal awareness in node provisioning reduces cross-AZ scheduling of tightly coupled workloads – a structural fix rather than a per-service workaround.
Group 6: Governance, Quotas, and FinOps Process (Checks 32–37) #
Without a governance loop, savings from the previous five groups erode within weeks. Governance enforces cost discipline at the API layer and creates the accountability structure that sustains optimization over time.
Check 32: ResourceQuota on every production namespace
ResourceQuotas prevent any team from monopolizing shared cluster resources. They also force teams to set requests – quota enforcement blocks pod creation unless requests are explicitly declared.
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-backend-quota
namespace: team-backend
spec:
hard:
requests.cpu: "8"
requests.memory: 16Gi
limits.cpu: "16"
limits.memory: 32Gi
pods: "50"
persistentvolumeclaims: "10"
kubectl get resourcequota --all-namespaces # Verify quota presence
Check 33: LimitRange on every namespace
LimitRange injects default resource values at admission time. Every pod in the namespace gets a sensible starting point, even when the developer did not specify resource fields. A LimitRange without a ResourceQuota is insufficient – both objects are required together.
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
namespace: team-backend
spec:
limits:
- type: Container
default:
memory: 256Mi
cpu: 200m
defaultRequest:
memory: 128Mi
cpu: 100m
max:
memory: 4Gi
cpu: "2"
Checks 34–35: Cost Ownership and Monthly Review
Check 34: Assign cost ownership. Label every namespace with team
and cost-center
. Only 14% of teams implement chargeback (FinOps Foundation 2025 State of FinOps Survey). Showback – informational reporting – is a safe starting point. However, showback data with no named owner has the same effect as no showback.
kubectl label namespace team-backend team=backend cost-center=eng-123
Check 35: Run a monthly cluster cost review. Compare actual spend versus budget, review the top five namespace movers, and confirm no new zombie resources appeared. Kubernetes cost drift is continuous – a monthly review catches it before it compounds into a significant bill surprise.
kubectl get limitrange --all-namespaces # Verify LimitRange coverage
Checks 36–37: Budget Alerts and Chargeback Maturity
Check 36: Set namespace-level budget alerts via Kubecost or OpenCost. Configure alerts to fire at 80% of each namespace’s monthly budget ceiling – not at 100%. By the time a namespace hits its ceiling, the month’s overage is already locked in. Alerting at 80% gives a team lead time to act before the bill closes.
Check 37: Define a chargeback maturity ladder: showback first, then allocated showback (costs tied to teams via labels), then full chargeback (costs reflected in team budgets or P&Ls). Run a monthly “top 5 namespace cost mover” report and escalate to team leads who exceed a 20% month-over-month increase. The escalation does not need to be punitive – a Slack notification with the delta is enough to make cost visible at the ownership layer. Visibility changes behavior faster than policy does.
Cast AI: Showback data changes team behavior. Cast AI’s cost allocation dashboard makes namespace-level spend visible to every team lead without requiring a dedicated FinOps analyst. Allocation Groups slice cost data by any label combination – team, environment, cost center, or application – without custom query maintenance. Cost reports are available daily. Historical comparison is built in, enabling the monthly review in Check 35 and the cost mover analysis in Check 37. The governance loop (Measure, Allocate, Govern, Review) maps directly to Cast AI’s monitoring, Allocation Groups, policy enforcement, and reporting – turning a recurring manual checklist into a continuous automated process.
How to Use This Checklist #
Run the groups in order for a fresh audit. Use this k8s cost checklist as a recurring audit – not a one-time exercise. Visibility comes first because every downstream decision depends on accurate data. Rightsizing and autoscaling come next because they address the largest cost drivers. Node efficiency, storage, and governance follow once pod-level waste is under control.
The cost optimization steps below are grouped by impact, not by complexity. Prioritize by dollar waste – use Check 6 (top namespaces by CPU waste) and Check 21 (underutilized nodes) to identify where to focus first. A namespace wasting 30% of its allocated CPU is a higher-priority fix than an orphaned PVC costing $5/month.
Automate what you can. VPA in Off mode, Goldilocks, and Karpenter consolidation all reduce ongoing manual work. However, the monthly review in Check 35 and the budget alerts in Check 36 require human judgment – a dashboard alone does not catch organizational drift or budget creep from forgotten workloads.
Re-run the full checklist monthly. Kubernetes clusters drift continuously as teams deploy new workloads and leave orphaned resources behind. A one-time audit erodes within one or two deployment cycles without a recurring review process.
Conclusion #
CPU utilization at 8% and falling is not a tooling problem. It is a structural problem: inflated requests, conservative defaults, and no cost ownership at the team level. The six groups in this checklist address each structural cause directly. Visibility creates the baseline. Rightsizing eliminates the padding. Autoscaling matches capacity to demand. Node efficiency reduces the per-hour cost of that capacity. Storage and network checks recover invisible spend. Governance keeps the savings from eroding as the cluster evolves.
After completing the audit, the next step is implementing a repeatable optimization loop. The full Kubernetes cost optimization guide covers the six-step process (Measure, Allocate, Rightsize, Autoscale, Govern, Review) with a 30/60/90-day roadmap for teams moving from audit to continuous practice.
Frequently Asked Questions #
What should a Kubernetes cost checklist cover?
A complete Kubernetes cost checklist covers six areas:
(1) visibility and cost attribution by namespace and team,
(2) pod rightsizing based on measured CPU and memory usage,
(3) autoscaling configuration for HPA, VPA, KEDA, and Cluster Autoscaler or Karpenter,
(4) node efficiency including Spot adoption and bin-packing,
(5) storage and network costs including orphaned PVCs and cross-AZ traffic
(6) governance through ResourceQuotas, LimitRanges, budget alerts, and a monthly review cycle. A checklist that skips any one group leaves money on the table.
How often should I audit my Kubernetes cluster for cost issues?
Run a full cost audit at least monthly. Kubernetes clusters drift continuously as teams deploy new workloads, increase resource requests, and leave orphaned resources behind. A monthly review catches drift before it compounds. In between reviews, continuous automated tools (VPA, Karpenter consolidation, Cast AI) handle ongoing adjustments. For rapidly growing clusters or those undergoing major refactoring, run an audit before and after each major release cycle.