Is Automated Rightsizing Safe? What Happens to OOMKills and CPU Throttling Automated rightsizing cut OOM kills from an average of 40 to 50 per reporting window, with spikes above 80, to near zero within days of being enabled in one representative cluster, while provisioned CPU fell by roughly half, according to Cast AI's 2026 State of Kubernetes Optimization Report. The report attributes the reduction to bidirectional adjustment — lowering requests for overprovisioned workloads and raising them for starved ones — and notes CPU throttling is a separate failure mode driven by CPU limits and CFS quota rather than request sizing. Cast AI recommends guardrails including a long observation window, headroom above the request, per-workload exclusions for stateful services and hard-SLO workloads, and starting in recommend-only mode. Automated rightsizing reduces crashes rather than causing them, provided it adjusts in both directions. The fear is understandable: cutting a memory request sounds like inviting an OOMKill. But static overprovisioning is a blunt instrument. It adds headroom uniformly, including to workloads that never needed it, while leaving genuinely memory-hungry workloads on limits nobody has reviewed since deployment. An agent that raises requests for workloads approaching their limit catches exactly those. In one representative cluster measured in Cast AI’s 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ , OOM kills averaged 40 to 50 per reporting window with spikes above 80, and dropped to near zero within days of automated rightsizing being enabled, while provisioned CPU fell by about half. Key Takeaways - Automated rightsizing reduces OOMKills rather than causing them. In one representative cluster, OOM kills fell from 40-50 per measurement interval to near zero after enabling automated rightsizing. - Provisioned CPU dropped by approximately 50% at the same time. Fewer crashes and lower cost happened together, not in trade-off. - The mechanism that matters is bidirectional adjustment: down for overprovisioned workloads, and up for starved ones. The upward direction is what eliminates OOMKills. - CPU throttling is a separate failure mode. It is caused by CPU limits and CFS quota, not by request sizing. Rightsizing memory requests does not fix throttling, and rightsizing tools should treat CPU limits differently from memory. - Guardrails make automated rightsizing safe to roll out: an observation window long enough to capture real workload variation, headroom above the request, and per-workload exclusions for stateful services and hard-SLO workloads. - Start with recommend-only mode. Verify the recommendations match expectations, then enable automation and widen scope gradually. The Fear, Stated Fairly The instinct makes sense. Your team set those memory requests with a number that felt safe. It was probably the request from the initial deployment, reviewed once, and left alone. Nobody wants to own a 3am PagerDuty alert because a tool trimmed a workload’s headroom and the kernel killed it. That concern is legitimate, and it deserves a direct answer before anything else. Why Teams Overprovision: Throttling and OOMKills, Both of Which Are Real Two failure modes push teams toward generous resource allocations. The first is CPU throttling. The Linux CFS scheduler enforces CPU limits within a 100ms window using cgroup quota cpu.cfs quota us / cpu.cfs period us . A container that exhausts its quota waits out the remaining window before it can run again, even when the node has idle CPU capacity. Spare capacity on the node does not flow to throttled containers. So teams set limits high, or set requests high enough to push scheduling onto nodes with more headroom. The second failure mode is OOMKills. When a container exceeds its memory limit, the Linux kernel sends SIGKILL exit code 137: 128 + SIGKILL . The container restarts. If the workload was handling live traffic, users noticed. Both failure modes are real. Both have happened to every team that runs production Kubernetes. Overprovisioning is the pragmatic, understandable response to operating without detailed per-workload usage data. What a Bad Rightsizing Implementation Would in Fact Do A tool that only cuts requests would, in fact, cause OOMKills. Specifically: observe a workload during a low-traffic window, see that memory usage is 200Mi, set the request to 220Mi, and then the workload hits Monday morning load and needs 800Mi. The kernel kills it. That failure mode is real. However, it describes a one-directional tool with an inadequate observation window. It does not describe Kubernetes rightsizing as a category. The mechanism matters. If the tool only reduces requests, it is not rightsizing. It is resource rationing, and it will fail in exactly the way the fear describes. The distinction between a tool that adjusts in one direction and one that adjusts in both is the entire answer to the safety question. The Data: OOMKills Before and After Before arguing the mechanism further, here is what happened in a real cluster. This is the data that anchors the rest of the discussion. 40 to 50 OOM Kills Per Reporting Window, Spikes Above 80, to Near Zero In one cluster tracked in the Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ , OOM kills were a persistent operational problem before automated rightsizing was enabled. The cluster averaged 40 to 50 OOM kill events per reporting window per measurement interval; the Cast AI 2026 report collected data in approximately daily snapshots across the fleet . During peak load periods, the count spiked above 80. These were not isolated incidents absorbed quietly. They were a routine operational cost, baked into the on-call burden. After automated rightsizing was enabled, OOM kills dropped to near zero within days. No mass incident followed. No emergency rollback was triggered. The workloads that had been chronically running close to their memory limits got their requests and limits raised to match actual peak usage. The kernel stopped killing them because they stopped exceeding their limits. Provisioned CPU Down About 50% at the Same Time Simultaneously, provisioned CPU fell by approximately 50%. This is the counterintuitive part. Memory limits went up for underprovisioned workloads. CPU allocations came down across the fleet. Both happened at the same time, because overprovisioning was not uniform in its source but uniform in its effect: resources set at deployment and never revisited, accumulating across hundreds of workloads. Fleet-wide, the Cast AI 2026 report found 69% CPU overprovisioning and 79% memory overprovisioning across the measured population. Both figures reflect the same dynamic: initial allocations that nobody went back to adjust as workloads changed. Automated rightsizing is the mechanism that closes that feedback loop. What the Measurement Is and Is Not This is one cluster, measured before and after enabling automated rightsizing. It does not use a controlled experiment with a matched control group, nor does it present the data as one. The fleet-wide overprovisioning figures 69% CPU, 79% memory come from aggregate analysis across the full report population. Both data sets point in the same direction: clusters carrying the most overprovisioning have the most unreviewed limits at risk of being wrong in both directions. Why Kubernetes Rightsizing Works: Adjustment in Both Directions The OOMKill reduction makes no sense unless you understand that the tool adjusts in both directions. This is not a subtle implementation detail. It is the entire mechanism. Down for the Overprovisioned For workloads where observed usage is consistently well below the current request, the tool recommends reducing the request. This recovers capacity the workload was never using. Cost falls and bin-packing on nodes improves. These are the workloads behind the fleet-wide overprovisioning statistics. These are buffers that were reasonable guesses at deployment time and have been sitting unused ever since. The Cast AI PrecisionPack approach to automated rightsizing https://cast.ai/blog/automated-workload-rightsizing-precisionpack/ covers this in more detail. Up for the Starved: The Part That Eliminates OOMKills For workloads where observed usage is approaching or exceeding the current limit, the tool raises the request and the limit. This is the mechanism that eliminates OOMKills. Workloads quietly running at the edge of their memory limit get the headroom they actually need. They stop being killed. The Cast AI Workload Autoscaler specifically detects OOM events per workload and can auto-disable recommendations for that workload while raising its limit, preventing restart loops during the adjustment period. Acting Before the Kernel Does The kernel acts when a container exceeds its memory limit. A good Kubernetes rightsizing tool acts when observed usage trends toward that limit. Cast AI’s default observation window is 7 days, long enough to capture both weekday and weekend traffic patterns for most workloads. This is configurable in the Cast AI console. Recommendations are based on p99 of observed usage, not a single snapshot or a mean. So a workload with a periodic memory spike gets a limit that accommodates that spike, rather than one sized to its quiet-hour baseline. For workloads with heavy garbage collection cycles JVM, Go or batch phases within a microservice, p99 can under-capture genuine peak usage. Consider p99.9 or the observed working-set max for memory limits on GC-heavy workloads. The difference between those numbers is where OOMKills hide. What Static Overprovisioning Actually Protects Understanding what manual overprovisioning actually protects is useful, because it explains precisely where the improvement is possible and where the risk lives. Uniform Headroom Protects the Workloads That Did Not Need It When a team sets a 4Gi memory limit on a workload that uses 800Mi in production, the extra 3.2Gi protects that workload from OOMKill. It also sits unused. Multiplied across hundreds of workloads with similar buffers, this is the 79% memory overprovisioning the report captures. The headroom is real protection, but allocated uniformly regardless of whether each workload needs it. The Workloads That Do Need It Were Set Once and Never Revisited The workload that started at 800Mi two years ago and now routinely hits 2.5Gi under normal load is still running on a 3Gi limit that someone set at deployment. Nobody reviewed it. The traffic pattern changed. The data volume changed. The code changed. The limit did not. Static overprovisioning protects workloads with generous initial buffers and leaves those that grew into their limits quietly at risk. Automated rightsizing addresses both: it recovers waste from the first category and catches risk in the second. | | What it protects | What it misses | What it costs | What happens when the workload changes | |---|---|---|---|---| | Static overprovisioning | Workloads with generous initial buffers; absorbs moderate traffic spikes | Workloads that grew beyond their original assumptions; limits nobody reviewed | 69% CPU waste, 79% memory waste fleet-wide; larger nodes than required | Nothing. The allocation does not follow the workload. Over time, the buffer either grows too small or stays too large. | | Automated rightsizing | All workloads: reduces the underprovisioned before the kernel acts, recovers waste from the overprovisioned | Workloads with memory leaks masks the symptom ; extreme spike profiles not covered by the observation window | Operationally requires an observation period, exclusion policy, and monitoring during rollout | Recommendations update as usage patterns change. The allocation follows the workload. | CPU Throttling Is a Different Failure OOMKills are solved by raising memory limits. CPU throttling requires a different analysis, and conflating the two leads to configurations that fix neither. Limits, CFS Quota, and Why Throttling Is Not Solved by More Requests The Linux CFS scheduler enforces CPU limits within a 100ms period. Each container has a quota cpu.cfs quota us that defines how many microseconds of CPU it can consume per period. When the container exhausts that quota, the CPU throttles it until the next period begins. This happens at the container level, regardless of available CPU on the node. Crucially: CPU requests affect scheduling which node the pod lands on but do not affect throttling. Throttling is determined entirely by the CPU limit. A pod can have a 500m request and a 500m limit. If the process needs 600m for 50ms during a garbage collection cycle, the CPU quota throttles it for the rest of that 100ms period, even when the node has eight mostly idle cores. Raising the request to 1000m does not help. Raising the limit does. For a full breakdown of how requests and limits interact, see Kubernetes Requests and Limits Explained https://cast.ai/blog/kubernetes-requests-and-limits/ . To detect throttling in your cluster: CPU throttle ratio per container. A ratio above 0.25 25% of periods throttled warrants investigation. sum rate container cpu cfs throttled periods total 5m by pod, container / sum rate container cpu cfs periods total 5m by pod, container 0.25 OOMKill detection kube pod container status last terminated reason{reason="OOMKilled"} Memory sizing: use working set, not usage bytes usage bytes includes reclaimable page cache; working set does not. container memory working set bytes How Rightsizing Should Treat CPU Limits Differently From Memory Memory is incompressible. Exceeding the limit triggers a kill. CPU is compressible: exceeding the limit triggers a wait. The failure modes differ, so the limit strategy should differ as well. For latency-sensitive workloads, omitting the CPU limit entirely eliminates CFS throttling. The tradeoff is reduced noisy-neighbor isolation. For batch workloads, a tighter CPU limit is acceptable because a slower batch job is far less damaging than a throttled API. The resources block below shows a practical configuration approach for both cases. resources: requests: CPU request set to p95 of observed usage. This determines which node the pod schedules onto. It does NOT prevent throttling; the limit controls that. cpu: "250m" Memory request set to p99/peak of observed usage. Use container memory working set bytes for sizing, not usage bytes. memory: "512Mi" limits: CPU limit at 3x request for most workloads. For latency-sensitive workloads, omit this field entirely to avoid CFS throttling on short burst activity GC, connection handling . cpu: "750m" Memory limit equal to request = Guaranteed QoS. Guaranteed QoS pods are evicted last under node memory pressure. Kubernetes eviction order: BestEffort first, Burstable second, Guaranteed last. For Burstable QoS with burst headroom, set limit to up to 1.5x request: memory: "768Mi" memory: "512Mi" The Guardrails That Make Kubernetes Rightsizing Safe No tool running automated changes on production workloads is safe without guardrails. The following guardrails are not optional. They are the difference between a smooth rollout and an incident. Observation Window Before Action The tool must observe the workload’s real usage pattern before acting. Cast AI’s default observation window is 7 days, long enough to capture weekday and weekend traffic patterns for most workloads. This is configurable in the Cast AI console. A single day of data misses weekly variation: the Monday morning traffic spike, the end-of-month batch job, the weekly report that runs Sunday at 2am. Do not shorten the window to speed up the initial rollout. Apply Changes in the Right Mode Cast AI Workload Autoscaler applies changes in one of three modes: Immediate restarts the pod at the next Deployment rollout , Deferred waits for the next organic pod restart , or In-Place no restart required, for Kubernetes 1.33+ clusters with InPlacePodVerticalScaling enabled . Deferred is the recommended starting mode for stateless workloads. It eliminates restart risk because the system does not schedule a restart; the new resource values take effect when the pod next restarts for another reason. Check docs.cast.ai https://docs.cast.ai/docs/workload-autoscaling-configuration for mode selection guidance and which managed distributions support In-Place by default. Headroom in the Limit Above the Request Setting the memory limit equal to the request gives you Guaranteed QoS, the highest-priority class in Kubernetes eviction order. However, it also means there is no burst headroom. For memory, the limit should reflect p99 or peak of observed usage, not the mean. For CPU, set the limit to 2-3x the request for most workloads, or omit it for latency-sensitive workloads that cannot tolerate throttling. The YAML block above shows this structure. Headroom must exist in the limit, not just the request. HPA Coexistence: The Most Common Operational Surprise When rightsizing lowers a pod’s CPU request, the CPU utilization ratio rises, potentially triggering HPA to scale out additional replicas to compensate. This is the most common operational surprise in the first weeks of rollout. Teams enable rightsizing, CPU requests fall, HPA sees higher utilization percentages against the new lower baseline, and replica counts climb unexpectedly. Cost savings are partially offset by the extra replicas HPA adds. Mitigation: start rightsizing in recommend-only mode and review HPA scale-out events for two weeks before enabling automation. Workloads where HPA scales on CPU should be treated cautiously. Consider memory-only rightsizing initially. This keeps CPU requests stable while still recovering memory waste and reducing OOMKills. Once you have two weeks of HPA behavior data against the new CPU request values, you can make an informed decision about enabling full automation. Per-Workload Policies, Exclusions, and Pre-Rollout Checks Not every workload should undergo automatic rightsizing. The tool needs to support per-workload exclusions, and teams should apply those exclusions before expanding automation. An annotation-based approach makes exclusions auditable and keeps them with the workload definition: apiVersion: apps/v1 kind: Deployment metadata: name: my-stateful-workload annotations: Exclude this workload from Cast AI automated rightsizing. Apply this annotation to stateful services, hard-SLO workloads, JVM workloads without explicit heap bounds, or anything with a memory leak under investigation. autoscaling.cast.ai/optimization-enabled: "false" spec: template: spec: containers: - name: app Resources remain under manual control when excluded. resources: requests: cpu: "500m" memory: "2Gi" limits: cpu: "2" memory: "2Gi" Before enabling automated rightsizing on any deployment, verify two additional constraints. First, audit PodDisruptionBudgets: a PDB with maxUnavailable: 0 blocks all voluntary disruptions, including rightsizing restarts. The rightsizing operation will stall silently with no error. Run kubectl get pdb -A and confirm no PDB targeting your workload carries that setting. Second, in multi-tenant clusters, verify the namespace ResourceQuota has sufficient ceiling for raised request values. Raising requests across a namespace with a quota ceiling close to consumed will block new pod scheduling after the recommendation is applied. VPA Compatibility If your cluster runs the Kubernetes Vertical Pod Autoscaler VPA , understand the interaction before enabling Cast AI automation. The Cast AI Workload Autoscaler and VPA should not run simultaneously on the same workload. Both tools attempt to set resource values, and the resulting conflicts produce unpredictable behavior. Disable VPA on any workload before enabling Cast AI rightsizing on it. The two tools are complementary at the fleet level as long as each workload has exactly one controller active at a time. Rollback and Evidence If a recommendation causes a problem, rollback should take seconds. First, suspend or disable automation for that workload in the Cast AI console. Without this step, the reconciliation loop will re-apply the recommendation within minutes, overwriting any manual change you make via kubectl. Once automation is suspended, restore prior resource values with kubectl set resources deployment/