# Karpenter Disruption and Drift: How to Consolidate Nodes Safely > Source: > Published: 2026-08-05 10:47:54+00:00 According to CAST AI’s 2026 Kubernetes Efficiency Report, CPU overprovisioning jumped from 40% to 69% year over year. Karpenter’s disruption system exists to recover that waste. However, without the right guardrails, consolidation can evict workloads at exactly the wrong moment. This guide covers how each disruption method works, how drift detection operates, and which controls keep production clusters stable. For a broader introduction to the scheduler itself, see [what is Karpenter](https://cast.ai/blog/what-is-karpenter/). ## Key takeaways - Karpenter disruption covers three mechanisms: consolidation, drift, and expiration. Each has a different trigger condition and a different safety profile. - Drift detection is event-driven: Karpenter re-evaluates nodes after NodePool or EC2NodeClass changes. In practice, this typically happens within approximately 5 minutes, based on the controller’s re-queue behavior — though this is implementation behavior, not an official SLA. - Disruption budgets limit concurrent node replacements. The default, when none is defined, is 10% of the NodePool. - Setting `nodes: "0"` blocks voluntary disruption. It does NOT block expiration.`expireAfter` bypasses disruption budgets entirely: the drain starts gracefully (PDBs are respected), but once the NodePool-level`terminationGracePeriod` elapses, pods are force-deleted regardless of any budget or freeze window. - The `do-not-disrupt` annotation on a Pod or Node prevents voluntary disruption. A`terminationGracePeriod` on the NodePool provides an override after a configurable TTL. - Use `WhenEmptyOrUnderutilized` as the consolidation policy in karpenter.sh/v1 for maximum savings.`Balanced` is a built-in middle ground (no feature gate required).`SpotToSpotConsolidation` is a separate feature gate that enables replacing a running Spot node with a cheaper Spot alternative — standard Spot node deletion (removing an underutilized Spot node entirely) does not require this gate. Only spot-to-spot replacement does. - For stateful workloads, combine PodDisruptionBudgets with `WhenEmpty` policy and AZ-aware scheduling to avoid cross-AZ EBS reattachment failures. ## The three disruption methods | Method | What triggers it | What Karpenter does | |---|---|---| | Consolidation | Node is empty or underutilized relative to bin-packing thresholds | Drains and deletes the node, rescheduling pods onto remaining nodes. Rate-limited by disruption budgets. | | Drift | Node spec diverges from NodePool or EC2NodeClass definition | Replaces the node with a new one matching the current spec. Rate-limited by disruption budgets. | | Expiration | `expireAfter` TTL elapses on the node | Begins graceful drain (cordon + evict), bypassing disruption budgets. PDBs and pod `terminationGracePeriodSeconds` are respected during the drain. Once the NodePool-level `terminationGracePeriod` also elapses, pods are force-deleted. | Consolidation is the cost-savings engine. Karpenter continuously evaluates whether pods fit onto fewer nodes and removes excess capacity when they do. For a deep dive into how consolidation scoring works, see [Karpenter consolidation](https://cast.ai/blog/karpenter-consolidation/). Drift is corrective. When you update a NodePool, Karpenter replaces old nodes that no longer match the new spec. This is how you roll out AMI updates or security group changes without manual intervention. Expiration is designed for node freshness and security hardening. The key distinction from the other two methods: `expireAfter` bypasses disruption budgets. When a node’s TTL elapses, Karpenter begins draining it regardless of budget settings. The drain is graceful: Karpenter respects PDBs and pod `terminationGracePeriodSeconds` . However, once the NodePool-level `terminationGracePeriod` also elapses, pods are force-deleted. This makes `expireAfter` unsuitable as a primary rotation mechanism for workloads with strict disruption windows. Rely on drift-triggered AMI rotation instead, and reserve `expireAfter` for hard compliance TTL requirements. The three consolidation policies give you a sliding scale of aggressiveness. `WhenEmpty` removes only nodes that have no pods running, safest for stateful workloads. `Balanced` considers the same node candidates as WhenEmptyOrUnderutilized but applies a cost-savings-vs-disruption scoring algorithm. Consolidation only proceeds when the savings percentage exceeds the calculated disruption weight for that NodePool. Higher pod priority raises disruption weight, making those nodes less likely candidates. It is a good middle ground for clusters with a mix of latency-sensitive and batch workloads. `WhenEmptyOrUnderutilized` removes any underutilized node and maximizes cost savings. All three are valid `consolidationPolicy` enum values in karpenter.sh/v1, with no feature gate required. ## What drift is and why it matters Drift is one of the more powerful features in Karpenter, and also one of the more frequently misunderstood. Here is the core mechanic: when you define a NodePool or EC2NodeClass, Karpenter computes a hash of the fields that affect node behavior. When that hash changes, Karpenter marks existing nodes as drifted and schedules them for replacement. Drift detection is event-driven: Karpenter re-evaluates nodes after NodePool or EC2NodeClass changes. In practice, this typically happens within approximately 5 minutes, based on the controller’s re-queue behavior, though this is implementation behavior, not an official SLA. Importantly, Karpenter respects disruption budgets during drift-triggered replacements. This makes drift a safer rotation mechanism than expiration for most use cases. ### What triggers drift on NodePool On the NodePool side, drift triggers when you change fields that directly affect the node: `nodeClassRef` , `requirements` , `taints` , `startupTaints` , and `kubelet` configuration. Behavioral fields, specifically `weight` , `limits` , and anything under `disruption.*` , do NOT trigger drift. Changing your disruption budget or consolidation policy does not cause a rolling node replacement. That is intentional behavior, not a gap. ### What triggers drift on EC2NodeClass On the EC2NodeClass side, drift triggers for AMI changes, security group changes, subnet changes, and IAM role changes. This is how you implement automated AMI upgrades: update the `amiSelectorTerms` in your EC2NodeClass and let Karpenter handle the rolling replacement. Pair this with a conservative disruption budget and your nodes rotate safely during off-peak hours. Notably, Karpenter detects actual AMI changes from AWS, not just spec changes. If Amazon releases a new EKS-optimized AMI and your selector matches it, Karpenter detects that the running AMI no longer matches the selector and marks affected nodes as drifted. You do not need to manually update anything after setting up the selector correctly. ## Controlling disruption Karpenter provides two primary controls for when and how disruption happens: disruption budgets and the `do-not-disrupt` annotation. Use both. They address different scenarios and operate at different levels of granularity. Disruption budgets work at the pool level, they limit how many nodes across a NodePool can change at once. The `do-not-disrupt` annotation works at the pod level, it freezes a specific workload without touching the rest of the pool. ### Disruption budgets A disruption budget defines how many nodes Karpenter can disrupt simultaneously within a NodePool. The `nodes` field accepts either a percentage string like `"10%"` or an integer like `5` . The percentage is relative to the total number of nodes in that NodePool at the time of evaluation, not cluster-wide. On a 10-node NodePool, `nodes: "10%"` allows 1 node to be disrupted at a time. On smaller NodePools (fewer than 10 nodes), a 10% budget may round down to 0, blocking all voluntary disruption. For small pools, use an absolute integer value like `nodes: 1` rather than a percentage. On a 100-node NodePool, the same budget allows 10. When you define multiple budgets, Karpenter applies the most restrictive one at any given time. Karpenter applies a default disruption budget of 10% when you do not define one explicitly. For large clusters, that represents significant concurrent disruption. Define an explicit budget for every production NodePool. Budgets also support scheduling via cron expressions in UTC. The `duration` field defines how long the budget applies. This gives you a clean mechanism to freeze disruption during business hours without manual intervention. Additionally, the `reasons` field scopes a budget to specific disruption types: `Underutilized` , `Empty` , or `Drifted` . A budget without a `reasons` field applies to all disruption types. #### Disruption budget YAML ``` apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 10m # 10 minutes is safer than 1m for production budgets: - nodes: "10%" # Allow up to 10% of nodes to be disrupted at once - nodes: "0" # Block all disruption during business hours (UTC) schedule: "0 9 * * 1-5" # Monday-Friday 09:00 UTC duration: 8h reasons: - Underutilized - Drifted ``` Note the `consolidateAfter: 10m` value. Setting this to `1m` is too aggressive for production: pods barely have time to reschedule before Karpenter evaluates again. Ten minutes provides a buffer for workload churn without sacrificing meaningful cost recovery. Also critical: `nodes: "0"` blocks voluntary disruption during the schedule window. It does NOT block expiration. If your `expireAfter` TTL fires during your freeze window, Karpenter begins draining those nodes regardless, budget settings do not apply. For this reason, avoid relying on expiration as your only AMI rotation strategy when you have strict maintenance windows. ### The do-not-disrupt annotation The `karpenter.sh/do-not-disrupt` annotation is a targeted escape hatch for individual workloads. Apply it to a Pod or a Node. When present on a Pod, Karpenter skips voluntary disruption for the node hosting that pod. When applied directly to a Node, the node itself is protected. Setting the value to `"true"` gives permanent protection until you remove the annotation. Alternatively, set a duration string like `"30m"` for time-based protection. This pattern works well for scheduled batch jobs: annotate the pod at launch, and Karpenter leaves the node alone until the job finishes. One important caveat: `terminationGracePeriod` on the NodePool overrides `do-not-disrupt` after a TTL. If a node has been pending replacement longer than the configured period, Karpenter proceeds with disruption regardless of the annotation. This prevents `do-not-disrupt` from permanently blocking replacement in edge cases where a pod gets stuck. #### do-not-disrupt annotation YAML ``` # On a pod apiVersion: v1 kind: Pod metadata: name: payment-processor annotations: karpenter.sh/do-not-disrupt: "true" --- # On a node apiVersion: v1 kind: Node metadata: name: ip-10-0-1-42.ec2.internal annotations: karpenter.sh/do-not-disrupt: "true" ``` ### Verifying disruption state If consolidation appears stalled, check the events first. Karpenter emits `Unconsolidatable` events when a PDB or `do-not-disrupt` annotation is blocking a node removal. The NodeClaim shows drift status directly: a `Drifted` condition means replacement is queued but awaiting a budget window. ``` # Check NodeClaim disruption conditions (drift, underutilized, expired) kubectl get nodeclaims -A -o wide # Watch Karpenter disruption events in real time # Note: reportingComponent requires Kubernetes 1.25 or later. # On older clusters, use --field-selector source=karpenter instead. kubectl get events -A --field-selector reportingComponent=karpenter --sort-by=.lastTimestamp # Check if consolidation is blocked by a PodDisruptionBudget kubectl get events -A --field-selector reason=Unconsolidatable ``` ### Emergency pause If consolidation causes an unexpected incident, you need a way to stop all voluntary disruption immediately. Patch the NodePool with a zero-node disruption budget to halt all new voluntary disruption for that pool immediately: ``` kubectl patch nodepool default --type=merge -p '{"spec":{"disruption":{"budgets":[{"nodes":"0"}]}}}' ``` To resume disruption once the incident is resolved, restore the budget to your normal setting: ``` kubectl patch nodepool default --type=merge -p '{"spec":{"disruption":{"budgets":[{"nodes":"10%"}]}}}' ``` Note: Patching to nodes: “0” prevents Karpenter from *initiating* new voluntary disruption. It does not stop an in-progress drain. It also does not block expireAfter-triggered drains, since expiration bypasses disruption budgets. ## Safe patterns for stateful and critical workloads Stateless workloads are forgiving. A pod that restarts on a new node usually reconnects and resumes within seconds. Stateful workloads are not forgiving. An EBS volume that reattaches to the wrong AZ, a StatefulSet that reschedules mid-transaction, or a cache that warms up cold: these are the scenarios that turn Karpenter disruption from a cost feature into an incident. ### PodDisruptionBudgets PodDisruptionBudgets (PDBs) are the first line of defense. Define `minAvailable` or `maxUnavailable` for every StatefulSet and Deployment that runs critical workloads. Karpenter respects PDBs during voluntary disruption: it will not drain a node if doing so violates the PDB. For example, a PDB with `minAvailable: 2` on a three-replica StatefulSet ensures Karpenter displaces at most one pod at a time. For additional guidance, see [Karpenter best practices](https://cast.ai/blog/karpenter-best-practices/). One common pitfall: setting `minAvailable` to 100% of a deployment’s replicas, or `maxUnavailable: 0` , permanently blocks consolidation for that deployment’s nodes. Karpenter will emit `Unconsolidatable` events but never proceed. If you see nodes that Karpenter never removes despite low utilization, check for zero-tolerance PDBs first. A common fix: set `maxUnavailable: 1` for multi-replica deployments and reserve `minAvailable: 100%` only for single-replica critical services that genuinely cannot tolerate any disruption. ### WhenEmpty policy for StatefulSets For StatefulSets that are difficult to reschedule, consider using `consolidationPolicy: WhenEmpty` . This policy only removes nodes that are completely empty. Karpenter will not bin-pack StatefulSet pods onto other nodes. Instead, it waits for pods to vacate naturally before reclaiming the node. This approach is more conservative than `WhenEmptyOrUnderutilized` , but it eliminates unexpected pod migrations for sticky workloads. ### EBS and availability zone constraints EBS volumes with `ReadWriteOnce` (RWO) access mode bind to a single availability zone. Cross-AZ reattachment fails: a pod cannot bind to the volume if Karpenter moves it to a different AZ. Same-AZ replacement is fast and works correctly. To keep StatefulSet pods in their original zone, combine a topology spread constraint with node selectors that match the AZ. A practical starting point: ``` topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: your-statefulset ``` When this constraint is in place, Karpenter places replacement nodes in the same AZ as the pod. The RWO volume reattaches successfully. Therefore, use AZ-specific NodePools or topology spread constraints to keep StatefulSet pods in their original zone. When enabling consolidation on EBS-backed workloads, validate your AZ affinity rules before turning it on in production. For truly zero-downtime stateful migration, the standard Karpenter disruption path cannot handle the data-movement problem. [Container Live Migration](https://cast.ai/blog/introducing-container-live-migration-zero-downtime-for-stateful-kubernetes-workloads/) addresses this directly by moving the container and its storage together without service interruption. ### A critical warning about expireAfter `expireAfter` bypasses disruption budgets. When a node’s TTL elapses, Karpenter begins draining it regardless of budget settings. The drain is graceful: Karpenter respects PDBs and pod `terminationGracePeriodSeconds` during this initial phase. However, once the NodePool-level `terminationGracePeriod` also elapses, pods are force-deleted. One common misconception: the `karpenter.sh/do-not-disrupt` annotation on a pod does NOT protect it from expireAfter. Once the NodePool-level terminationGracePeriod elapses, pods annotated with do-not-disrupt will still be force-deleted. The annotation only blocks voluntary disruption, not expiration. If you set `expireAfter: 168h` on a production NodePool, nodes will rotate every 7 days regardless of cluster state — and freeze windows do not protect them. For stateful workloads, an expiration-triggered replacement during peak traffic can cause serious service disruption even if the initial drain is graceful. The safer approach: rely on drift-triggered AMI rotation. Update your `amiSelectorTerms` in EC2NodeClass, set a conservative disruption budget with a maintenance window, and let drift handle replacement on your schedule. Reserve `expireAfter` for compliance requirements that demand a hard TTL guarantee, and document the budget-bypass behavior explicitly in your runbook. ## Scaling disruption management across fleets Managing disruption budgets on a single cluster is manageable. Managing them across dozens of clusters, with varying workload types, traffic patterns, and change schedules, is where manual tuning breaks down. Cast AI continuously monitors consolidation efficiency and drift rollout state across your fleet. When consolidation is stalling because a budget window is misaligned with actual low-traffic periods, Cast AI adjusts the window. When AMI drift is queuing up across hundreds of nodes, Cast AI paces the rollout to stay within tolerance. Teams using Cast AI typically see CPU overprovisioning decline from baselines like the 69% industry average — the system continuously corrects for workload churn and demand shifts that would otherwise require weekly manual tuning sessions. To see how Cast AI applies to your Karpenter fleet, [optimize Kubernetes costs with Cast AI](https://cast.ai/karpenter-optimization/). ## Frequently Asked Questions **What is Karpenter disruption?** Karpenter disruption is the collective term for the three mechanisms Karpenter uses to remove or replace nodes: consolidation, drift, and expiration. Consolidation removes underutilized or empty nodes to recover wasted capacity. Drift replaces nodes whose spec no longer matches the current NodePool or EC2NodeClass definition. Expiration begins a graceful drain when a node’s configured TTL elapses, bypassing disruption budgets; once the NodePool-level `terminationGracePeriod` also elapses, pods are force-deleted. Disruption budgets and the `do-not-disrupt` annotation control how aggressively consolidation and drift operate, but neither applies to expiration-triggered rotation. **What is drift in Karpenter?** Drift in Karpenter occurs when a running node no longer matches the spec defined in its NodePool or EC2NodeClass. Karpenter detects this by comparing a hash of the current node configuration against the hash of the desired spec. Drift detection is event-driven: Karpenter re-evaluates nodes after NodePool or EC2NodeClass changes, typically within approximately 5 minutes based on the controller’s re-queue behavior – though this is implementation behavior, not an official SLA. Common triggers include AMI changes, security group changes, updated node requirements, and kubelet configuration changes. Behavioral fields such as weight, limits, and disruption settings do not trigger drift. **How do disruption budgets work?** Disruption budgets limit how many nodes Karpenter can disrupt at the same time within a NodePool. The `nodes` field accepts a percentage string like `10%` or an integer. The percentage is relative to the total number of nodes in that NodePool at the time of evaluation, not cluster-wide. You can also attach a cron schedule and duration to create time-based freeze windows. When multiple budgets exist, Karpenter applies the most restrictive one. If you do not define a disruption budget, Karpenter applies a default value of 10%. Note that disruption budgets apply only to voluntary disruption (consolidation and drift). Expiration bypasses budgets entirely: when a node’s TTL elapses, the drain begins regardless of budget settings. **What is do-not-disrupt?** The `karpenter.sh/do-not-disrupt` annotation prevents Karpenter from voluntarily disrupting a specific pod or node. Set it to `true` for permanent protection, or use a duration string like `30m` for time-based protection. Apply it to individual Pods or directly to Nodes. A `terminationGracePeriod` on the NodePool acts as an override: if a node has been pending replacement longer than the configured TTL, Karpenter proceeds with disruption regardless of the annotation. **How do I protect critical workloads?** Use a layered approach. First, define PodDisruptionBudgets with `minAvailable` or `maxUnavailable` for every critical Deployment and StatefulSet, but avoid `maxUnavailable: 0` , which permanently blocks consolidation. Second, apply the `do-not-disrupt` annotation to pods that cannot be interrupted mid-operation. Third, use `consolidationPolicy: WhenEmpty` for StatefulSets that are difficult to reschedule. Fourth, set a disruption budget with `nodes: 0` during peak traffic windows. Finally, avoid `expireAfter` for stateful workloads unless you have a hard compliance requirement: expiration bypasses disruption budgets, so the graceful drain it initiates can still proceed during a freeze window, and pods are force-deleted once the NodePool `terminationGracePeriod` elapses.