Kubernetes Rightsizing: A Practical Workflow for CPU and Memory Optimization

Kubernetes rightsizing matches pod resource requests to actual usage, closing an overprovisioning gap where 69% of requested CPU goes unused, according to the 2026 Cast AI report. A five-step workflow—Observe, Set Requests, Set Limits, Verify, Repeat—can reduce compute costs by 40-70% without code changes. Kubernetes 1.33+ supports zero-downtime in-place resizing, and GPU waste, averaging 5% utilization, can be cut by 70% via fractional sharing.

Kubernetes rightsizing is the practice of matching pod CPU and memory requests and limits to actual usage, so you stop paying for capacity you never use without risking throttling or OOM kills. Most clusters show 8% average CPU utilization: pods consume only a small fraction of what they allocate. The 2026 Cast AI report quantifies the gap: 69% of requested CPU across production clusters goes entirely unused. That is the overprovisioning gap that rightsizing closes. Key Takeaways CPU is wasted by default: Average CPU utilization across production Kubernetes clusters is 8%; 69% of requested CPU goes entirely unused Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ . Rightsizing is the safest cost lever: Correcting resource requests requires no code changes, no replica adjustments, and no architectural redesign only configuration fixes. Follow the five-step workflow: Observe real usage 2+ weeks → Set CPU requests at p95 typical → Set limits to protect neighbors → Verify with throttling and OOM checks → Repeat as workloads evolve. Zero-downtime resizing is available: Kubernetes 1.33+ supports in-place pod resizing InPlacePodVerticalScaling, beta in 1.33, GA in 1.35 ; CPU adjustments apply without a container restart. GPU waste is the most expensive gap: Average GPU utilization is 5%; fractional GPU sharing via time-slicing, MIG, or NVIDIA MPS can cut GPU costs by 70% or more without changing model code. Expected savings: 40–70% compute cost reduction is realistic for clusters with no previous rightsizing; automated rightsizing Cast AI Workload Autoscaler has delivered up to 93% utilization improvement. What Rightsizing Is and Why It Is the Safest Lever Requests define the CPU and memory Kubernetes reserves on a node for a pod. Limits define how much the pod can consume before it gets throttled CPU or killed memory . The scheduler uses requests to place pods. Nodes fill up based on requests, not actual usage. The gap between those two numbers is where your money disappears. The Cast AI 2026 State of Kubernetes Optimization Report measured tens of thousands of production clusters across AWS, GCP, and Azure. Average CPU utilization: 8%. Average memory utilization: 20%. CPU overprovisioning sits at 69% up from 40% year over year . Memory overprovisioning is 79%. Those are two distinct measurements. The 8% figure is average CPU utilization: how much of reserved capacity pods actually consume at runtime. The 69% figure is the overprovisioning ratio: how much of CPU requests is never consumed across production clusters. Together, they confirm that most clusters provision many times more CPU than their workloads actually need. No autoscaler can fix this at the node level if pods hold oversized requests. Node autoscalers see a “full” cluster and add nodes. The waste scales with you. Rightsizing is the safest lever because it reduces waste without changing what your application does. You are not modifying code, not changing replicas, not touching network policy. You are correcting a configuration error that most teams introduced when they set resources by gut feel during initial deployment and never revisited. The Rightsizing Workflow The Rightsizing Workflow is a five-step cycle: Observe, Set Requests to Typical, Set Limits to Protect, Verify, Repeat. It applies whether you are doing this manually, with VPA, or with an automated platform. The cycle does not end after the first pass. Observe Start with real usage data. Two weeks minimum. One week misses weekly batch jobs. One day misses end-of-month spikes. The longer the observation window, the more accurate the recommendation. For a quick baseline, run these two commands: Check actual usage vs requests kubectl top pod -n <namespace --containers Check what's allocated kubectl describe pod <pod-name -n <namespace | grep -A 10 "Requests:" The first command shows live usage. The second shows what Kubernetes reserved. If the gap is large, rightsizing will have a meaningful impact. For statistical sizing over longer windows, use these PromQL queries directly against Prometheus: Compute p95 CPU usage over the past 2 weeks for requests sizing quantile over time 0.95, rate container cpu usage seconds total{container =""} 5m 2w:5m by namespace, pod, container Compute p99 memory usage over 2 weeks for limits sizing quantile over time 0.99, container memory working set bytes{container =""} 2w:5m by namespace, pod, container These queries work best with 2+ weeks of Prometheus history. For longer retention, use Thanos or Cortex. KRR is designed for exactly this use case and queries those backends natively. For CPU throttling, you need Prometheus. Throttling is invisible at the pod level: no log, no event, no status change. The container keeps running, but latency increases silently. Detect it with: sum rate container cpu cfs throttled periods total 5m by pod, container / sum rate container cpu cfs periods total 5m by pod, container 0.25 A result above 0.25 means more than 25% of CPU scheduling periods are throttled. That is a performance problem hiding in your metrics. Set Requests to Typical Requests should reflect typical workload behavior, not peak. Use the p95 or p99 of observed CPU usage over your observation window. Sizing to the absolute peak means you overprovision for rare spikes that last seconds. Here is what that looks like in practice. A service that was provisioned by hand at deployment: Before over-provisioned resources: requests: cpu: "2" memory: "4Gi" limits: cpu: "4" memory: "8Gi" After observing p95 usage over two weeks: After rightsized via p95 observed usage resources: requests: cpu: "200m" memory: "512Mi" limits: cpu: "600m" memory: "1Gi" The CPU request dropped from 2000m to 200m. The memory request dropped from 4Gi to 512Mi. The node scheduler now sees this pod as ten times smaller, freeing capacity for other workloads or smaller nodes. Set Limits to Protect Limits exist to prevent one pod from consuming all node capacity and affecting neighbors. For CPU, set limits to 2-3x the request. CPU is compressible: exceeding the limit throttles the container, it does not kill it. For memory, be more conservative. For memory-sensitive workloads, set request equal to limit Guaranteed QoS . This prevents eviction under node memory pressure. For workloads where occasional bursting is acceptable, set request below limit Burstable QoS . Never omit memory limits on multi-tenant clusters. For CPU on most production workloads, consider omitting limits entirely and relying on requests for scheduling. CPU limits with tight settings cause more throttling incidents than the overprovisioning they prevent. JVM workloads need special attention. The JVM heap size is separate from the container memory limit. Java applications will try to expand the heap up to the limit, which means your container memory request should account for: JVM heap -Xmx plus roughly 25% overhead for metaspace, thread stacks, and off-heap buffers. If the limit is too tight, you get OOMKills. If requests are too loose, you pay for memory the JVM never uses. Set -Xmx to 75% of the container memory limit as a starting point. Verify After applying new requests and limits, watch for two failure signals. First, check the Prometheus throttling query again 24 hours after the change. If throttling increased significantly, the new CPU request is too low. Second, watch for OOMKilled events: kubectl get events -n <namespace --field-selector reason=OOMKilling kubectl describe pod <pod-name -n <namespace | grep -i oom A single OOMKill after rightsizing means the memory request landed too low. Increase the memory request and limit, then re-observe. If p99 latency increases or error rates rise after applying new limits, the cause is likely CPU throttling limits too tight or OOM kills memory limits too tight . Rollback immediately: kubectl set resources deployment <name -c <container --requests=cpu=<original --limits=cpu=<original For automated platforms, temporarily suspend rightsizing for that workload until you identify the safe limit floor. Repeat Workloads change. A new feature ships. Traffic patterns shift. A library version increases baseline memory footprint. Static rightsizing decays. Treat it as a recurring process, not a one-time task. For manual processes: set a calendar reminder every two to four weeks, or after every major release. For automated platforms: Cast AI’s Workload Autoscaler refreshes recommendations every 30 minutes and reacts immediately to OOM events or usage spikes exceeding 50% above the current recommendation. CPU vs Memory Failure Modes CPU and memory fail differently. Knowing the difference changes how you set limits and how urgently you respond to threshold breaches. Throttling vs OOMKilled CPU is a compressible resource. When a container exceeds its CPU limit, the kernel throttles it. The process keeps running. Your application slows down. This is silent: no Kubernetes event, no log entry, no status change. Latency increases, timeouts creep up, SLOs degrade. You may not connect the cause to a CPU limit for days. Memory is incompressible . When a container exceeds its memory limit, the kernel kills it immediately with an OOMKill exit code 137 . Kubernetes restarts the container. Depending on your readiness probe configuration, this causes a visible service interruption. On StatefulSets or pods with warm caches, each restart is expensive. See the detailed breakdown of OOMKilled behavior and exit codes: OOMKilled Exit Code 137 explained https://cast.ai/blog/oomkilled-exit-code-137/ . One production cluster in the 2026 report recorded 40-50 OOM kills per 30-minute window measured across a representative production cluster with generous resource padding; the interval is the platform’s 30-minute recommendation cycle . After applying recommendations with OOM detection overhead added automatically, OOM kills dropped to near zero while provisioned CPUs were cut by roughly half. How to Set Each The sizing principles follow the workflow steps above. For CPU, use p95 observed usage for requests and 2-3x the request for limits, or omit limits entirely to avoid silent throttling. For memory, use p99 or peak for requests; set limits equal to the request for Guaranteed QoS on critical workloads, or up to 1.5x the request for Burstable QoS on batch workloads. JVM services require the additional heap headroom calculation described in the Set Limits to Protect step. Manual vs VPA vs Automated Three approaches exist for getting recommendations into production. Each has a different cost/benefit profile. | Approach | Detection | Application | Downtime | Accuracy | Ops Burden | |---|---|---|---|---|---| | Manual | Engineer runs kubectl top / PromQL | PR + deployment | Rolling restart | Low point-in-time | High | | VPA Recreate | VPA Recommender 8-day window | Updater evicts pods | Pod eviction + restart | Medium | Low | | Automated Cast AI | Continuous, every 30 min + anomaly detection | In-place or deferred | None in-place or controlled | High OOM-aware | Very low | Goldilocks Goldilocks Fairwinds wraps VPA in recommendation-only mode. It creates a VPA object per Deployment in namespaces you label with goldilocks.fairwinds.com/enabled=true . No pods change. You view recommendations in a web dashboard that shows both Guaranteed QoS request equals limit and Burstable QoS request below limit options. Setup: install VPA first, then Goldilocks, then label the namespace. The VPA Updater mode stays Off throughout. This makes Goldilocks safe to run alongside production workloads with zero risk of pod changes. The trade-off: you still apply changes manually. KRR KRR Robusta queries Prometheus directly, bypassing the VPA API entirely. This matters because VPA retains only 8 days of history. KRR can query Thanos, Cortex, or Mimir for longer windows, giving more accurate recommendations for workloads with weekly or monthly traffic patterns. pip install robusta-krr krr simple --prometheus-url http://prometheus:9090 KRR ships with customizable strategies: percentile default p99 for CPU, peak for memory. No admission webhook means it is safe to run on any cluster without side effects. Like Goldilocks, it produces recommendations. You apply them. VPA Modes VPA ships with five update modes. Each has different disruption characteristics: Off: Recommendations only. No pods changed. Safe starting point for any cluster. Initial: Sets requests at pod creation only. Existing pods unchanged. Low risk. Recreate: The Updater evicts and recreates pods to apply new values. Disrupts StatefulSets with caches, JIT-warmed code, or leader elections. Use with caution. InPlaceOrRecreate: Attempts live resize first, falls back to eviction. Requires Kubernetes 1.33+ InPlacePodVerticalScaling beta or 1.35+ for GA and default-on behavior. InPlace: Never evicts. Alpha in VPA 1.7.0. Requires Kubernetes 1.33+. VPA limitations to know before enabling Recreate mode: VPA needs 24-48 hours of data minimum and retains only 8 days. When HPA is also active, VPA lowering requests causes HPA utilization percentages to rise, which can trigger unintended horizontal scale-out. Run HPA off CPU utilization and VPA together only with care. The mitigation: configure HPA to scale on external or custom metrics such as request rate or queue depth rather than CPU utilization when VPA is active. That way, VPA adjusts requests without affecting HPA’s scaling trigger. KEDA is a strong option here. It scales on message queue length, HTTP request rate, or any Prometheus metric, avoiding the feedback loop entirely. Platforms That Act The Cast AI Workload Autoscaler moves beyond recommendation into continuous automated application. It supports Deployments, StatefulSets, DaemonSets, Rollouts, ReplicaSets, and CronJobs. Recommendations regenerate every 30 minutes. If usage spikes more than 50% above the current recommendation, regeneration triggers immediately. OOM detection runs by monitoring pod container statuses. When a container OOMKills, Cast AI adds overhead automatically to the memory recommendation before the next apply cycle. This is the mechanism behind the 40-50 OOM kills per interval dropping to near zero in the 2026 report cluster data. Apply modes are Immediate changes applied as soon as a recommendation is ready and Deferred changes queued for the next maintenance window or pod restart . High-availability mode with two replicas is available from v0.30.0. In-Place vs Restart The in-place pod resize feature eliminates the need to evict and recreate pods when changing requests. The timeline: alpha in Kubernetes 1.27, beta in Kubernetes 1.33 May 2025 , GA and default-on in Kubernetes 1.35 December 2025 . Cast AI Workload Autoscaler supports in-place resizing from v0.53.0. For clusters still below 1.33, restart-based application remains the default. PodDisruptionBudgets covered below limit the blast radius. Rightsizing Without Downtime The most common objection to rightsizing in production is disruption risk. Two mechanisms address this directly. In-Place Pod Resize In-place pod resizing InPlacePodVerticalScaling lets the kubelet update a running container’s CPU and memory resources without restarting it. The kernel adjusts the cgroup limits live. Without evictions, restarts, or service interruptions. The feature timeline: alpha in Kubernetes 1.27, beta in Kubernetes 1.33 May 2025 , GA and default-on in Kubernetes 1.35 December 2025 . On clusters running 1.33+, you can use InPlace or InPlaceOrRecreate modes with the feature gate enabled. Whereas on 1.35+, it is on by default with no gate required. On managed clusters EKS, GKE, AKS , check whether your control plane version exposes InPlacePodVerticalScaling. Managed versions often lag upstream by 2-4 months. Verify with: kubectl get node -o jsonpath='{.items 0 .status.nodeInfo.kubeletVersion}' and your cloud provider’s feature gate documentation. This is the biggest operational change for rightsizing since VPA shipped. StatefulSets with warm caches, JVM workloads with compiled code, and leader-elected controllers all benefit immediately because pod churn disappears. Control per-container resize behavior with the resizePolicy field: spec: containers: - name: myapp resizePolicy: - resourceName: cpu restartPolicy: NotRequired CPU resize without restart - resourceName: memory restartPolicy: RestartContainer Memory decrease needs restart The default if unset is RestartContainer for all resources. Set NotRequired for CPU safe in both directions and leave memory as RestartContainer . Most container runtimes as of mid-2025 require a restart for memory resize down; memory resize up is live. PodDisruptionBudgets When in-place resize is not available clusters below 1.33, or memory downsizing , PodDisruptionBudgets PDBs limit how many pods VPA or Cast AI can evict simultaneously. A PDB with maxUnavailable: 1 allows at most one replica to be taken down during the resize cycle while keeping the rest available. apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: my-service-pdb namespace: production spec: maxUnavailable: 1 selector: matchLabels: app: my-service Avoid maxUnavailable: 0 on pods subject to rightsizing. That configuration prevents any voluntary disruption, blocking VPA in Recreate mode and in-place restarts from completing. Set PDBs before enabling VPA Recreate mode or Cast AI Immediate mode on any service with an SLO. The combination of PDB plus rolling eviction gives you safe restart-based rightsizing on any Kubernetes version. Governing Requests Rightsizing individual workloads is necessary but not sufficient. Without namespace-level guardrails, a single misconfigured deployment can consume all cluster resources. Use ResourceQuotas and LimitRanges to enforce governance. ResourceQuotas ResourceQuotas cap the total CPU and memory that all pods in a namespace can request and limit. They prevent any single team or workload from crowding out others. apiVersion: v1 kind: ResourceQuota metadata: name: team-quota namespace: team-a spec: hard: requests.cpu: "10" requests.memory: 20Gi limits.cpu: "20" limits.memory: 40Gi Once a ResourceQuota exists in a namespace, every pod must declare requests and limits. Pods without resource declarations fail admission. This forces teams to think about sizing at deploy time. Cast AI Workload Autoscaler is ResourceQuota-aware from v0.82.0 and auto-adjusts recommendations to stay within quota constraints. LimitRanges LimitRanges set default requests and limits for containers that do not specify them, and enforce min/max bounds. They complement ResourceQuotas by operating at the container level rather than the namespace aggregate level. apiVersion: v1 kind: LimitRange metadata: name: default-limits namespace: team-a spec: limits: - type: Container default: cpu: "500m" memory: "512Mi" defaultRequest: cpu: "100m" memory: "128Mi" max: cpu: "4" memory: "8Gi" Without a LimitRange, containers that omit resource requests will have no requests set. In namespaces with ResourceQuotas, this makes them unschedulable because quota enforcement requires explicit requests. A LimitRange solves this by injecting defaultRequest values automatically. Cast AI Workload Autoscaler is LimitRange-aware and auto-adjusts to comply with namespace constraints. For teams on AWS using Karpenter: Karpenter’s NodePool resource constraints act as a cluster-level ceiling on what pods can request. Ensure your NodePool resource limits are set above your namespace ResourceQuotas, otherwise pod scheduling will fail when quota-allowed requests exceed NodePool limits. GPU Rightsizing and Fractional GPUs GPU waste is the most expensive rightsizing problem. The Cast AI 2026 report found average GPU utilization at 5%. A cluster full of A100s running at 5% is not an AI cluster. It is a $10,000-per-node cost center that happens to run some models. Three approaches share a single GPU across multiple workloads: Time-slicing: The GPU switches between workloads temporally. No memory isolation. Simple to configure. Best for inference workloads that do not need dedicated VRAM. MIG Multi-Instance GPU : Hardware partitioning available on A100 and H100. Provides memory and fault isolation. Each MIG instance looks like a separate GPU to the container. Best for production inference with strict isolation requirements. NVIDIA MPS Multi-Process Service : Process-level sharing for CUDA workloads. Best for training jobs that need to share compute without full isolation overhead. ALLEN Digital moved 7 models from SageMaker to Kubernetes using GPU time-slicing plus Spot instances case study https://cast.ai/case-studies/allen-digital/ . Time-slicing alone delivered 20% savings. Node consolidation added another 30-40%. Total savings exceeded 70%. How Much You Can Save The 2026 Cast AI data puts CPU overprovisioning at 69% across tens of thousands of clusters. That is the addressable savings pool from CPU rightsizing alone, before touching node autoscaling or Spot usage. Memory overprovisioning at 79% adds to this, though memory savings flow through differently depending on whether you run spot-heavy or reserved fleets. In practice, savings depend on how overprovisioned you start. The Bud case study saw up to 93% improvement in CPU and memory utilization via Cast AI automated rightsizing. The OOM cluster in the report cut provisioned CPUs by approximately half while eliminating nearly all kill events. A useful frame: rightsizing reduces the size of the bin each pod occupies. Node autoscaling then packs more workloads per node and removes underutilized nodes. The two work together. Rightsizing first makes every subsequent autoscaling decision more efficient. For a full breakdown of the cost optimization levers and how rightsizing fits into the broader picture, see Kubernetes Cost Optimization https://cast.ai/blog/kubernetes-cost-optimization/ . Conclusion Rightsizing is not a one-time configuration exercise. Clusters change, teams ship new services, and load patterns shift. The clusters that stay optimized are the ones that measure continuously and apply changes automatically. If you manage multiple clusters or frequently deploy new services, Cast AI’s Workload Autoscaler handles the observe-set-verify-repeat loop automatically, recalculating every 30 minutes, reacting immediately to OOM events, and applying changes without disruption. Connect your cluster at Cast AI https://cast.ai to see current recommendations. What is Kubernetes rightsizing? Kubernetes rightsizing is the process of adjusting pod CPU and memory requests and limits to match actual workload usage. Oversized requests waste money because Kubernetes reserves node capacity based on requests, not real consumption. Undersized requests cause throttling CPU or OOM kills memory . Rightsizing closes that gap safely. How do I avoid OOM kills? Set memory requests at p99 or peak observed usage, and set memory limits equal to or slightly above the request. Monitor with kubectl get events --field-selector reason=OOMKilling . Automated tools like Cast AI detect OOMKill events and add memory overhead automatically before the next recommendation cycle, preventing recurrence. For JVM workloads, set -Xmx to 75% of the container memory limit to leave headroom for metaspace and off-heap buffers. VPA vs automated rightsizing: which is better? VPA in Recreate mode applies recommendations but disrupts pods. It retains only 8 days of history and conflicts with HPA when both run on CPU utilization metrics. Automated platforms like Cast AI update every 30 minutes, support in-place resize no eviction , detect OOM events automatically, and handle LimitRange and ResourceQuota constraints. For production clusters, automated rightsizing reduces ops burden significantly. How much can rightsizing save? The Cast AI 2026 State of Kubernetes Optimization Report found 69% CPU overprovisioning and 79% memory overprovisioning across tens of thousands of production clusters. In practice, savings depend on your starting point. The Bud case study achieved up to 93% improvement in CPU and memory utilization. A reasonable expectation for clusters with no previous optimization is 40-70% reduction in compute costs. What tools are available for Kubernetes rightsizing? The main options are: manual kubectl top + PromQL , Goldilocks VPA-based dashboard, read-only , KRR by Robusta Prometheus-native CLI, longer history windows , VPA automated apply with multiple modes , and Cast AI Workload Autoscaler continuous, OOM-aware, in-place resize, quota-aware . Each option trades off accuracy, ops burden, and disruption risk differently. How do I rightsize without downtime? On Kubernetes 1.33+, use in-place pod resizing with the InPlacePodVerticalScaling feature gate beta in 1.33, GA and default-on in 1.35 . CPU and memory resize up happen without a container restart. On older versions, combine PodDisruptionBudgets with rolling eviction to limit simultaneous pod restarts to a safe count. Cast AI supports both approaches and selects the least disruptive method automatically. What is GPU rightsizing? GPU rightsizing matches GPU allocation to actual model utilization. Average GPU utilization in production Kubernetes clusters is 5% Cast AI 2026 . Sharing strategies include time-slicing temporal sharing, no memory isolation , MIG hardware partitioning on A100/H100, full isolation , and NVIDIA MPS process-level sharing for CUDA . Fractional GPU allocation can cut GPU costs by 70% or more without changing model code.