{"slug": "karpenter-disruption-and-drift-how-to-consolidate-nodes-safely", "title": "Karpenter Disruption and Drift: How to Consolidate Nodes Safely", "summary": "CAST AI's 2026 Kubernetes Efficiency Report found CPU overprovisioning jumped from 40% to 69% year over year, and Karpenter's disruption system—consolidation, drift, and expiration—can recover that waste but requires guardrails to avoid evicting workloads at the wrong moment. The guide details how each disruption method works, noting that expiration bypasses disruption budgets and can force-delete pods after the NodePool-level terminationGracePeriod elapses, while consolidation and drift are rate-limited by budgets. It recommends using WhenEmptyOrUnderutilized for maximum savings and combining PodDisruptionBudgets with WhenEmpty policy and AZ-aware scheduling for stateful workloads.", "body_md": "According to CAST AI’s 2026 Kubernetes Efficiency Report, CPU overprovisioning jumped from 40% to 69% year over year. Karpenter’s disruption system exists to recover that waste. However, without the right guardrails, consolidation can evict workloads at exactly the wrong moment. This guide covers how each disruption method works, how drift detection operates, and which controls keep production clusters stable. For a broader introduction to the scheduler itself, see [what is Karpenter](https://cast.ai/blog/what-is-karpenter/).\n\n## Key takeaways\n\n- Karpenter disruption covers three mechanisms: consolidation, drift, and expiration. Each has a different trigger condition and a different safety profile.\n- Drift detection is event-driven: Karpenter re-evaluates nodes after NodePool or EC2NodeClass changes. In practice, this typically happens within approximately 5 minutes, based on the controller’s re-queue behavior — though this is implementation behavior, not an official SLA.\n- Disruption budgets limit concurrent node replacements. The default, when none is defined, is 10% of the NodePool.\n- Setting\n`nodes: \"0\"`\n\nblocks voluntary disruption. It does NOT block expiration.`expireAfter`\n\nbypasses disruption budgets entirely: the drain starts gracefully (PDBs are respected), but once the NodePool-level`terminationGracePeriod`\n\nelapses, pods are force-deleted regardless of any budget or freeze window. - The\n`do-not-disrupt`\n\nannotation on a Pod or Node prevents voluntary disruption. A`terminationGracePeriod`\n\non the NodePool provides an override after a configurable TTL. - Use\n`WhenEmptyOrUnderutilized`\n\nas the consolidation policy in karpenter.sh/v1 for maximum savings.`Balanced`\n\nis a built-in middle ground (no feature gate required).`SpotToSpotConsolidation`\n\nis a separate feature gate that enables replacing a running Spot node with a cheaper Spot alternative — standard Spot node deletion (removing an underutilized Spot node entirely) does not require this gate. Only spot-to-spot replacement does. - For stateful workloads, combine PodDisruptionBudgets with\n`WhenEmpty`\n\npolicy and AZ-aware scheduling to avoid cross-AZ EBS reattachment failures.\n\n## The three disruption methods\n\n| Method | What triggers it | What Karpenter does |\n|---|---|---|\n| Consolidation | Node is empty or underutilized relative to bin-packing thresholds | Drains and deletes the node, rescheduling pods onto remaining nodes. Rate-limited by disruption budgets. |\n| Drift | Node spec diverges from NodePool or EC2NodeClass definition | Replaces the node with a new one matching the current spec. Rate-limited by disruption budgets. |\n| Expiration | `expireAfter` TTL elapses on the node | Begins graceful drain (cordon + evict), bypassing disruption budgets. PDBs and pod `terminationGracePeriodSeconds` are respected during the drain. Once the NodePool-level `terminationGracePeriod` also elapses, pods are force-deleted. |\n\nConsolidation is the cost-savings engine. Karpenter continuously evaluates whether pods fit onto fewer nodes and removes excess capacity when they do. For a deep dive into how consolidation scoring works, see [Karpenter consolidation](https://cast.ai/blog/karpenter-consolidation/).\n\nDrift is corrective. When you update a NodePool, Karpenter replaces old nodes that no longer match the new spec. This is how you roll out AMI updates or security group changes without manual intervention.\n\nExpiration is designed for node freshness and security hardening. The key distinction from the other two methods: `expireAfter`\n\nbypasses disruption budgets. When a node’s TTL elapses, Karpenter begins draining it regardless of budget settings. The drain is graceful: Karpenter respects PDBs and pod `terminationGracePeriodSeconds`\n\n. However, once the NodePool-level `terminationGracePeriod`\n\nalso elapses, pods are force-deleted. This makes `expireAfter`\n\nunsuitable as a primary rotation mechanism for workloads with strict disruption windows. Rely on drift-triggered AMI rotation instead, and reserve `expireAfter`\n\nfor hard compliance TTL requirements.\n\nThe three consolidation policies give you a sliding scale of aggressiveness. `WhenEmpty`\n\nremoves only nodes that have no pods running, safest for stateful workloads. `Balanced`\n\nconsiders the same node candidates as WhenEmptyOrUnderutilized but applies a cost-savings-vs-disruption scoring algorithm. Consolidation only proceeds when the savings percentage exceeds the calculated disruption weight for that NodePool. Higher pod priority raises disruption weight, making those nodes less likely candidates. It is a good middle ground for clusters with a mix of latency-sensitive and batch workloads. `WhenEmptyOrUnderutilized`\n\nremoves any underutilized node and maximizes cost savings. All three are valid `consolidationPolicy`\n\nenum values in karpenter.sh/v1, with no feature gate required.\n\n## What drift is and why it matters\n\nDrift is one of the more powerful features in Karpenter, and also one of the more frequently misunderstood. Here is the core mechanic: when you define a NodePool or EC2NodeClass, Karpenter computes a hash of the fields that affect node behavior. When that hash changes, Karpenter marks existing nodes as drifted and schedules them for replacement.\n\nDrift detection is event-driven: Karpenter re-evaluates nodes after NodePool or EC2NodeClass changes. In practice, this typically happens within approximately 5 minutes, based on the controller’s re-queue behavior, though this is implementation behavior, not an official SLA. Importantly, Karpenter respects disruption budgets during drift-triggered replacements. This makes drift a safer rotation mechanism than expiration for most use cases.\n\n### What triggers drift on NodePool\n\nOn the NodePool side, drift triggers when you change fields that directly affect the node: `nodeClassRef`\n\n, `requirements`\n\n, `taints`\n\n, `startupTaints`\n\n, and `kubelet`\n\nconfiguration. Behavioral fields, specifically `weight`\n\n, `limits`\n\n, and anything under `disruption.*`\n\n, do NOT trigger drift. Changing your disruption budget or consolidation policy does not cause a rolling node replacement. That is intentional behavior, not a gap.\n\n### What triggers drift on EC2NodeClass\n\nOn the EC2NodeClass side, drift triggers for AMI changes, security group changes, subnet changes, and IAM role changes. This is how you implement automated AMI upgrades: update the `amiSelectorTerms`\n\nin your EC2NodeClass and let Karpenter handle the rolling replacement. Pair this with a conservative disruption budget and your nodes rotate safely during off-peak hours.\n\nNotably, Karpenter detects actual AMI changes from AWS, not just spec changes. If Amazon releases a new EKS-optimized AMI and your selector matches it, Karpenter detects that the running AMI no longer matches the selector and marks affected nodes as drifted. You do not need to manually update anything after setting up the selector correctly.\n\n## Controlling disruption\n\nKarpenter provides two primary controls for when and how disruption happens: disruption budgets and the `do-not-disrupt`\n\nannotation. Use both. They address different scenarios and operate at different levels of granularity. Disruption budgets work at the pool level, they limit how many nodes across a NodePool can change at once. The `do-not-disrupt`\n\nannotation works at the pod level, it freezes a specific workload without touching the rest of the pool.\n\n### Disruption budgets\n\nA disruption budget defines how many nodes Karpenter can disrupt simultaneously within a NodePool. The `nodes`\n\nfield accepts either a percentage string like `\"10%\"`\n\nor an integer like `5`\n\n. The percentage is relative to the total number of nodes in that NodePool at the time of evaluation, not cluster-wide. On a 10-node NodePool, `nodes: \"10%\"`\n\nallows 1 node to be disrupted at a time. On smaller NodePools (fewer than 10 nodes), a 10% budget may round down to 0, blocking all voluntary disruption. For small pools, use an absolute integer value like `nodes: 1`\n\nrather than a percentage. On a 100-node NodePool, the same budget allows 10. When you define multiple budgets, Karpenter applies the most restrictive one at any given time.\n\nKarpenter applies a default disruption budget of 10% when you do not define one explicitly. For large clusters, that represents significant concurrent disruption. Define an explicit budget for every production NodePool.\n\nBudgets also support scheduling via cron expressions in UTC. The `duration`\n\nfield defines how long the budget applies. This gives you a clean mechanism to freeze disruption during business hours without manual intervention. Additionally, the `reasons`\n\nfield scopes a budget to specific disruption types: `Underutilized`\n\n, `Empty`\n\n, or `Drifted`\n\n. A budget without a `reasons`\n\nfield applies to all disruption types.\n\n#### Disruption budget YAML\n\n```\napiVersion: karpenter.sh/v1\nkind: NodePool\nmetadata:\n  name: default\nspec:\n  disruption:\n    consolidationPolicy: WhenEmptyOrUnderutilized\n    consolidateAfter: 10m       # 10 minutes is safer than 1m for production\n    budgets:\n    - nodes: \"10%\"              # Allow up to 10% of nodes to be disrupted at once\n    - nodes: \"0\"                # Block all disruption during business hours (UTC)\n      schedule: \"0 9 * * 1-5\"  # Monday-Friday 09:00 UTC\n      duration: 8h\n      reasons:\n      - Underutilized\n      - Drifted\n```\n\nNote the `consolidateAfter: 10m`\n\nvalue. Setting this to `1m`\n\nis too aggressive for production: pods barely have time to reschedule before Karpenter evaluates again. Ten minutes provides a buffer for workload churn without sacrificing meaningful cost recovery.\n\nAlso critical: `nodes: \"0\"`\n\nblocks voluntary disruption during the schedule window. It does NOT block expiration. If your `expireAfter`\n\nTTL fires during your freeze window, Karpenter begins draining those nodes regardless, budget settings do not apply. For this reason, avoid relying on expiration as your only AMI rotation strategy when you have strict maintenance windows.\n\n### The do-not-disrupt annotation\n\nThe `karpenter.sh/do-not-disrupt`\n\nannotation is a targeted escape hatch for individual workloads. Apply it to a Pod or a Node. When present on a Pod, Karpenter skips voluntary disruption for the node hosting that pod. When applied directly to a Node, the node itself is protected.\n\nSetting the value to `\"true\"`\n\ngives permanent protection until you remove the annotation. Alternatively, set a duration string like `\"30m\"`\n\nfor time-based protection. This pattern works well for scheduled batch jobs: annotate the pod at launch, and Karpenter leaves the node alone until the job finishes.\n\nOne important caveat: `terminationGracePeriod`\n\non the NodePool overrides `do-not-disrupt`\n\nafter a TTL. If a node has been pending replacement longer than the configured period, Karpenter proceeds with disruption regardless of the annotation. This prevents `do-not-disrupt`\n\nfrom permanently blocking replacement in edge cases where a pod gets stuck.\n\n#### do-not-disrupt annotation YAML\n\n```\n# On a pod\napiVersion: v1\nkind: Pod\nmetadata:\n  name: payment-processor\n  annotations:\n    karpenter.sh/do-not-disrupt: \"true\"\n---\n# On a node\napiVersion: v1\nkind: Node\nmetadata:\n  name: ip-10-0-1-42.ec2.internal\n  annotations:\n    karpenter.sh/do-not-disrupt: \"true\"\n```\n\n### Verifying disruption state\n\nIf consolidation appears stalled, check the events first. Karpenter emits `Unconsolidatable`\n\nevents when a PDB or `do-not-disrupt`\n\nannotation is blocking a node removal. The NodeClaim shows drift status directly: a `Drifted`\n\ncondition means replacement is queued but awaiting a budget window.\n\n```\n# Check NodeClaim disruption conditions (drift, underutilized, expired)\nkubectl get nodeclaims -A -o wide\n\n# Watch Karpenter disruption events in real time\n# Note: reportingComponent requires Kubernetes 1.25 or later.\n# On older clusters, use --field-selector source=karpenter instead.\nkubectl get events -A --field-selector reportingComponent=karpenter --sort-by=.lastTimestamp\n\n# Check if consolidation is blocked by a PodDisruptionBudget\nkubectl get events -A --field-selector reason=Unconsolidatable\n```\n\n### Emergency pause\n\nIf consolidation causes an unexpected incident, you need a way to stop all voluntary disruption immediately. Patch the NodePool with a zero-node disruption budget to halt all new voluntary disruption for that pool immediately:\n\n```\nkubectl patch nodepool default --type=merge -p '{\"spec\":{\"disruption\":{\"budgets\":[{\"nodes\":\"0\"}]}}}'\n```\n\nTo resume disruption once the incident is resolved, restore the budget to your normal setting:\n\n```\nkubectl patch nodepool default --type=merge -p '{\"spec\":{\"disruption\":{\"budgets\":[{\"nodes\":\"10%\"}]}}}'\n```\n\nNote: Patching to nodes: “0” prevents Karpenter from *initiating* new voluntary disruption. It does not stop an in-progress drain. It also does not block expireAfter-triggered drains, since expiration bypasses disruption budgets.\n\n## Safe patterns for stateful and critical workloads\n\nStateless workloads are forgiving. A pod that restarts on a new node usually reconnects and resumes within seconds. Stateful workloads are not forgiving. An EBS volume that reattaches to the wrong AZ, a StatefulSet that reschedules mid-transaction, or a cache that warms up cold: these are the scenarios that turn Karpenter disruption from a cost feature into an incident.\n\n### PodDisruptionBudgets\n\nPodDisruptionBudgets (PDBs) are the first line of defense. Define `minAvailable`\n\nor `maxUnavailable`\n\nfor every StatefulSet and Deployment that runs critical workloads. Karpenter respects PDBs during voluntary disruption: it will not drain a node if doing so violates the PDB. For example, a PDB with `minAvailable: 2`\n\non a three-replica StatefulSet ensures Karpenter displaces at most one pod at a time. For additional guidance, see [Karpenter best practices](https://cast.ai/blog/karpenter-best-practices/).\n\nOne common pitfall: setting `minAvailable`\n\nto 100% of a deployment’s replicas, or `maxUnavailable: 0`\n\n, permanently blocks consolidation for that deployment’s nodes. Karpenter will emit `Unconsolidatable`\n\nevents but never proceed. If you see nodes that Karpenter never removes despite low utilization, check for zero-tolerance PDBs first. A common fix: set `maxUnavailable: 1`\n\nfor multi-replica deployments and reserve `minAvailable: 100%`\n\nonly for single-replica critical services that genuinely cannot tolerate any disruption.\n\n### WhenEmpty policy for StatefulSets\n\nFor StatefulSets that are difficult to reschedule, consider using `consolidationPolicy: WhenEmpty`\n\n. This policy only removes nodes that are completely empty. Karpenter will not bin-pack StatefulSet pods onto other nodes. Instead, it waits for pods to vacate naturally before reclaiming the node. This approach is more conservative than `WhenEmptyOrUnderutilized`\n\n, but it eliminates unexpected pod migrations for sticky workloads.\n\n### EBS and availability zone constraints\n\nEBS volumes with `ReadWriteOnce`\n\n(RWO) access mode bind to a single availability zone. Cross-AZ reattachment fails: a pod cannot bind to the volume if Karpenter moves it to a different AZ. Same-AZ replacement is fast and works correctly.\n\nTo keep StatefulSet pods in their original zone, combine a topology spread constraint with node selectors that match the AZ. A practical starting point:\n\n```\ntopologySpreadConstraints:\n- maxSkew: 1\n  topologyKey: topology.kubernetes.io/zone\n  whenUnsatisfiable: DoNotSchedule\n  labelSelector:\n    matchLabels:\n      app: your-statefulset\n```\n\nWhen this constraint is in place, Karpenter places replacement nodes in the same AZ as the pod. The RWO volume reattaches successfully.\n\nTherefore, use AZ-specific NodePools or topology spread constraints to keep StatefulSet pods in their original zone. When enabling consolidation on EBS-backed workloads, validate your AZ affinity rules before turning it on in production.\n\nFor truly zero-downtime stateful migration, the standard Karpenter disruption path cannot handle the data-movement problem. [Container Live Migration](https://cast.ai/blog/introducing-container-live-migration-zero-downtime-for-stateful-kubernetes-workloads/) addresses this directly by moving the container and its storage together without service interruption.\n\n### A critical warning about expireAfter\n\n`expireAfter`\n\nbypasses disruption budgets. When a node’s TTL elapses, Karpenter begins draining it regardless of budget settings. The drain is graceful: Karpenter respects PDBs and pod `terminationGracePeriodSeconds`\n\nduring this initial phase. However, once the NodePool-level `terminationGracePeriod`\n\nalso elapses, pods are force-deleted. One common misconception: the `karpenter.sh/do-not-disrupt`\n\nannotation on a pod does NOT protect it from expireAfter. Once the NodePool-level terminationGracePeriod elapses, pods annotated with do-not-disrupt will still be force-deleted. The annotation only blocks voluntary disruption, not expiration. If you set `expireAfter: 168h`\n\non a production NodePool, nodes will rotate every 7 days regardless of cluster state — and freeze windows do not protect them. For stateful workloads, an expiration-triggered replacement during peak traffic can cause serious service disruption even if the initial drain is graceful.\n\nThe safer approach: rely on drift-triggered AMI rotation. Update your `amiSelectorTerms`\n\nin EC2NodeClass, set a conservative disruption budget with a maintenance window, and let drift handle replacement on your schedule. Reserve `expireAfter`\n\nfor compliance requirements that demand a hard TTL guarantee, and document the budget-bypass behavior explicitly in your runbook.\n\n## Scaling disruption management across fleets\n\nManaging disruption budgets on a single cluster is manageable. Managing them across dozens of clusters, with varying workload types, traffic patterns, and change schedules, is where manual tuning breaks down.\n\nCast AI continuously monitors consolidation efficiency and drift rollout state across your fleet. When consolidation is stalling because a budget window is misaligned with actual low-traffic periods, Cast AI adjusts the window. When AMI drift is queuing up across hundreds of nodes, Cast AI paces the rollout to stay within tolerance. Teams using Cast AI typically see CPU overprovisioning decline from baselines like the 69% industry average — the system continuously corrects for workload churn and demand shifts that would otherwise require weekly manual tuning sessions.\n\nTo see how Cast AI applies to your Karpenter fleet, [optimize Kubernetes costs with Cast AI](https://cast.ai/karpenter-optimization/).\n\n## Frequently Asked Questions\n\n**What is Karpenter disruption?** Karpenter disruption is the collective term for the three mechanisms Karpenter uses to remove or replace nodes: consolidation, drift, and expiration. Consolidation removes underutilized or empty nodes to recover wasted capacity. Drift replaces nodes whose spec no longer matches the current NodePool or EC2NodeClass definition. Expiration begins a graceful drain when a node’s configured TTL elapses, bypassing disruption budgets; once the NodePool-level `terminationGracePeriod`\n\nalso elapses, pods are force-deleted. Disruption budgets and the `do-not-disrupt`\n\nannotation control how aggressively consolidation and drift operate, but neither applies to expiration-triggered rotation.\n\n**What is drift in Karpenter?** Drift in Karpenter occurs when a running node no longer matches the spec defined in its NodePool or EC2NodeClass. Karpenter detects this by comparing a hash of the current node configuration against the hash of the desired spec. Drift detection is event-driven: Karpenter re-evaluates nodes after NodePool or EC2NodeClass changes, typically within approximately 5 minutes based on the controller’s re-queue behavior – though this is implementation behavior, not an official SLA. Common triggers include AMI changes, security group changes, updated node requirements, and kubelet configuration changes. Behavioral fields such as weight, limits, and disruption settings do not trigger drift.\n\n**How do disruption budgets work?** Disruption budgets limit how many nodes Karpenter can disrupt at the same time within a NodePool. The `nodes`\n\nfield accepts a percentage string like `10%`\n\nor an integer. The percentage is relative to the total number of nodes in that NodePool at the time of evaluation, not cluster-wide. You can also attach a cron schedule and duration to create time-based freeze windows. When multiple budgets exist, Karpenter applies the most restrictive one. If you do not define a disruption budget, Karpenter applies a default value of 10%. Note that disruption budgets apply only to voluntary disruption (consolidation and drift). Expiration bypasses budgets entirely: when a node’s TTL elapses, the drain begins regardless of budget settings.\n\n**What is do-not-disrupt?** The `karpenter.sh/do-not-disrupt`\n\nannotation prevents Karpenter from voluntarily disrupting a specific pod or node. Set it to `true`\n\nfor permanent protection, or use a duration string like `30m`\n\nfor time-based protection. Apply it to individual Pods or directly to Nodes. A `terminationGracePeriod`\n\non the NodePool acts as an override: if a node has been pending replacement longer than the configured TTL, Karpenter proceeds with disruption regardless of the annotation.\n\n**How do I protect critical workloads?** Use a layered approach. First, define PodDisruptionBudgets with `minAvailable`\n\nor `maxUnavailable`\n\nfor every critical Deployment and StatefulSet, but avoid `maxUnavailable: 0`\n\n, which permanently blocks consolidation. Second, apply the `do-not-disrupt`\n\nannotation to pods that cannot be interrupted mid-operation. Third, use `consolidationPolicy: WhenEmpty`\n\nfor StatefulSets that are difficult to reschedule. Fourth, set a disruption budget with `nodes: 0`\n\nduring peak traffic windows. Finally, avoid `expireAfter`\n\nfor stateful workloads unless you have a hard compliance requirement: expiration bypasses disruption budgets, so the graceful drain it initiates can still proceed during a freeze window, and pods are force-deleted once the NodePool `terminationGracePeriod`\n\nelapses.", "url": "https://wpnews.pro/news/karpenter-disruption-and-drift-how-to-consolidate-nodes-safely", "canonical_source": "https://cast.ai/blog/karpenter-disruption-drift/", "published_at": "2026-08-05 10:47:54+00:00", "updated_at": "2026-08-05 10:53:41.701565+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "robotics", "autonomous-vehicles"], "entities": ["CAST AI", "Karpenter"], "alternates": {"html": "https://wpnews.pro/news/karpenter-disruption-and-drift-how-to-consolidate-nodes-safely", "markdown": "https://wpnews.pro/news/karpenter-disruption-and-drift-how-to-consolidate-nodes-safely.md", "text": "https://wpnews.pro/news/karpenter-disruption-and-drift-how-to-consolidate-nodes-safely.txt", "jsonld": "https://wpnews.pro/news/karpenter-disruption-and-drift-how-to-consolidate-nodes-safely.jsonld"}}