{"slug": "kubernetes-cluster-autoscaler-how-it-works-and-how-to-tune-scale-up-and-scale", "title": "Kubernetes Cluster Autoscaler: How It Works and How to Tune Scale-Up and Scale-Down", "summary": "Kubernetes Cluster Autoscaler reactively adds nodes when pods are pending and removes them after a sustained underutilization window, but its reliance on pod resource requests and default flags often blocks scale-down, contributing to average CPU utilization of just 8% and 69% of clusters being CPU-overprovisioned, according to a Cast AI report. Tuning flags like scale-down-unneeded-time and expander settings can improve cost efficiency, but alternatives like Karpenter offer faster provisioning.", "body_md": "The Kubernetes Cluster Autoscaler adds nodes when pods are pending and removes them after a sustained underutilization window. This guide covers how it works, every tuning flag that affects cost, and when Karpenter or Cast AI delivers better efficiency.\n\n## Key takeaways\n\n- Cluster Autoscaler (CA) is\n**reactive only**: it provisions nodes after pods are already pending. It does not predict demand. - CA measures utilization using\n**pod resource.requests**, not actual CPU or memory consumption. Overprovisioned requests block scale-down even when nodes are nearly idle. - The scale-down default window is\n**10 minutes** of continuous underutilization (`--scale-down-unneeded-time`\n\n). Common flags like`--skip-nodes-with-system-pods`\n\nand`--skip-nodes-with-local-storage`\n\n(both default`true`\n\n) silently block scale-down on many clusters. - The\n**least-waste expander** is the best general-purpose choice for cost. Chain it with`random`\n\nas a tiebreaker:`--expander=least-waste,random`\n\n. - CA is\n**limited to pre-defined node groups**. It cannot choose a different instance type at runtime. - The Cast AI 2026 State of Kubernetes Optimization Report found\n**average CPU utilization at 8%** and**69% of clusters CPU-overprovisioned**. CA responds to inflated requests as genuine demand, making this a structural problem that tuning flags alone cannot fix. - Node provisioning takes\n**3–4 minutes** with CA (cloud provider abstraction layer). Karpenter, which calls EC2 RunInstances directly, provisions in 60–90 seconds under typical conditions.\n\n## How Cluster Autoscaler works\n\nCluster Autoscaler runs as a single pod in your cluster. It polls the Kubernetes API server every 10 seconds (configurable via `--scan-interval`\n\n). On each cycle, CA checks two conditions: are there unschedulable pods, and are there underutilized nodes?\n\n### Scale-up trigger\n\nWhen CA finds pods in `Pending`\n\nstate, it simulates whether adding a node to any available node group would allow those pods to schedule. If a match exists, CA calls the cloud provider API to increase the node group size: an Auto Scaling Group on AWS, a Managed Instance Group on GCP, or a VM Scale Set on Azure.\n\nThat API roundtrip is not instant. Node registration, kubelet startup, and readiness checks bring the end-to-end provisioning time to **3–4 minutes**. Your pods stay pending during that entire window.\n\n### Scale-down trigger\n\nCA continuously scans for underutilized nodes. A node qualifies for removal when the sum of `resource.requests`\n\nacross all its pods falls below the `--scale-down-utilization-threshold`\n\n(default: 50% of node capacity). Additionally, CA must be able to reschedule all those pods onto other existing nodes.\n\nA qualifying node must remain underutilized for the full `--scale-down-unneeded-time`\n\nwindow (default: 10 minutes) before CA attempts removal. CA then cordons the node, evicts pods via the Kubernetes Eviction API (respecting PodDisruptionBudgets), and signals the cloud provider to terminate the instance.\n\nSeveral conditions block scale-down by default. Nodes running kube-system pods without PDBs, pods using local storage (`emptyDir`\n\n, `hostPath`\n\n), and standalone pods not backed by a controller all prevent CA from removing a node. The flags `--skip-nodes-with-system-pods=true`\n\nand `--skip-nodes-with-local-storage=true`\n\nare the most common silent blockers.\n\n### Expanders: which node group gets the new node\n\nWhen multiple node groups could accommodate a pending pod, the **expander** setting determines which one CA selects. The expander is configured via `--expander`\n\n.\n\nAvailable expander types:\n\n**random**(default): picks a qualifying group at random. No cost optimization.** most-pods**: picks the group that would schedule the most pending pods in one event. Useful for batch workloads.** least-waste**: picks the group that leaves the least idle CPU/memory after scheduling. Best general-purpose choice for cost reduction.**price**: picks the lowest-cost group. Supported on GKE and Azure only. Not available natively on AWS EKS.** priority**: uses a user-defined ConfigMap to rank node groups. Useful for preferring Spot ASGs over on-demand.\n\nYou can chain expanders as a tiebreaker sequence. For example: `--expander=least-waste,random`\n\n(CA v1.23+ required). CA applies them in order, falling back to the next only when there is a tie.\n\n### Reactive, not predictive\n\nCA has no awareness of future demand. It does not prefetch nodes before workloads arrive. Every scale-up starts from a pod already stuck in `Pending`\n\n. For batch jobs, ML training runs, or any workload with predictable ramp-up, that reactive model adds latency directly to job start time. This is a design constraint, not a bug you can tune away.\n\n## Tuning scale-up and scale-down\n\nThe following flags have the highest impact on cost. Most are set as deployment args in the CA Deployment manifest.\n\n### Key tuning parameters\n\n| Flag | Default | Effect on cost |\n|---|---|---|\n`--expander` | `random` | Switch to `least-waste` (or chain `least-waste,random` ) to minimize wasted capacity per scale-up event. |\n`--scale-down-utilization-threshold` | `0.5` | Raise to `0.65` –`0.70` to trigger scale-down on nodes that are moderately utilized. Based on `requests` , not actual usage. |\n`--scale-down-unneeded-time` | `10m` | Reduce to `5m` for cost-aggressive configs. Increase for slow-start or stateful workloads to prevent premature evictions. |\n`--scale-down-delay-after-add` | `10m` | How long CA waits after any scale-up before considering scale-down. Reduce to `5m` on fast-settling workloads to reclaim nodes sooner. |\n`--scale-down-delay-after-failure` | `3m` | Controls CA back-off after a failed scale-down attempt. Keep at `3m` by default; increase to `10m` if you see repeated failed scale-downs in CA logs. |\n`--skip-nodes-with-system-pods` | `true` | Set to `false` (with PDBs in place for critical system pods like kube-dns; see example below) to unblock scale-down of nodes that would otherwise be permanently retained. |\n`--skip-nodes-with-local-storage` | `true` | Set to `false` only for fault-tolerant workloads that use `emptyDir` . Reduces blocked node count. |\n`--balance-similar-node-groups` | `false` | Set to `true` to distribute nodes evenly across AZs. Prevents accidental single-AZ concentration and supports HA. |\n`--scan-interval` | `10s` | Lower for faster reaction; raises API server load. Rarely needs changing. Leave at default unless profiling shows CA latency issues. |\n`--max-node-provision-time` | `15m` | Maximum wait for a new node to become Ready. Reduce if your bootstrap is fast; increase for large or complex node setups. |\n\nWhen you set `--skip-nodes-with-system-pods=false`\n\n, you must protect critical system pods with PodDisruptionBudgets. Without PDBs, CA can evict kube-dns or other control-plane components during scale-down, breaking cluster DNS. Apply a PDB for kube-dns before enabling this flag:\n\n```\napiVersion: policy/v1\nkind: PodDisruptionBudget\nmetadata:\n  name: kube-dns-pdb\n  namespace: kube-system\nspec:\n  minAvailable: 1\n  selector:\n    matchLabels:\n      k8s-app: kube-dns\n```\n\n### Cost-optimized CA deployment configuration\n\nThe following YAML shows a cost-tuned CA Deployment args section. Apply it to your existing CA Deployment under `spec.containers[0].command`\n\n.\n\n```\n# cluster-autoscaler deployment args (cost-optimized)\n# Replace YOUR_CLUSTER_NAME and YOUR_REGION with actual values.\n- ./cluster-autoscaler\n- --cloud-provider=aws                          # or gcp, azure\n- --nodes=1:20:YOUR_NODE_GROUP_NAME\n- --expander=least-waste,random                 # minimize idle capacity; random as tiebreaker\n- --scale-down-utilization-threshold=0.65       # more aggressive than default 0.5\n- --scale-down-unneeded-time=5m                 # down from default 10m\n- --scale-down-delay-after-add=5m               # down from default 10m\n- --skip-nodes-with-system-pods=false           # requires PDBs on critical system pods\n- --skip-nodes-with-local-storage=false         # only if emptyDir workloads are fault-tolerant\n- --balance-similar-node-groups=true            # distribute across AZs\n- --max-node-provision-time=10m                 # reduce from 15m if bootstrap is fast\n- --scan-interval=10s                           # leave at default unless profiling shows issues\n- --logtostderr=true\n- --stderrthreshold=info\n- --v=4\n```\n\n⚠️ Note: The `--nodes`\n\nflag applies to self-managed CA installations (EKS, self-managed AKS). On GKE, use the GKE console or `gcloud container node-pools`\n\nto configure min/max node counts. GKE-managed CA does not use deployment args.\n\nBefore applying these settings, audit what is actually blocking scale-down on your cluster. Run `kubectl describe configmap cluster-autoscaler-status -n kube-system`\n\nand look for the `ScaleDown`\n\nsection. CA logs its reasoning for every node it skips. Start there before tuning thresholds.\n\n### Helm chart deployment\n\nMost teams install CA via Helm (`helm install cluster-autoscaler autoscaler/cluster-autoscaler`\n\n). If you deploy CA this way, pass the same flags as `extraArgs`\n\nvalues in your Helm values file:\n\n```\nextraArgs:\n  scale-down-enabled: \"true\"\n  scale-down-utilization-threshold: \"0.65\"\n  scale-down-unneeded-time: \"5m\"\n  skip-nodes-with-local-storage: \"false\"\n  balance-similar-node-groups: \"true\"\n```\n\n### Validating the change\n\nAfter applying new settings, confirm CA loaded them by checking the logs:\n\n```\nkubectl logs -f deployment/cluster-autoscaler -n kube-system | grep \"scale-down-utilization-threshold\"\n```\n\nTo monitor CA behavior in real time, CA exposes a Prometheus metrics endpoint at `:8085/metrics`\n\n. Two metrics are worth tracking immediately. `cluster_autoscaler_last_activity`\n\ndetects a stuck CA that has stopped evaluating the cluster. `cluster_autoscaler_unschedulable_pods_count`\n\nshows pending pods still waiting for a node. A sustained non-zero value here means scale-up is not keeping pace with demand.\n\n## Limitations\n\nTuning helps at the margins. However, CA has structural constraints that no flag combination fully resolves.\n\n### Node group constraint\n\nCA scales within pre-defined node groups only. It cannot select a different instance type or family at runtime. If no group fits a pending pod precisely, CA picks the closest available match and provisions it with leftover capacity. Adding more node groups to improve coverage also adds management overhead: separate ASGs, separate IAM policies, and separate CA configuration entries.\n\n### Bin packing at add-time only\n\nThe `least-waste`\n\nexpander packs pods efficiently when CA adds a node. It does not repack workloads already running on existing nodes. Over time, nodes become fragmented: partly filled with long-running pods that cannot be easily moved. Those nodes stay up indefinitely unless they happen to drain naturally. CA does not consolidate running workloads. Karpenter does, via its `WhenEmptyOrUnderutilized`\n\nconsolidation policy.\n\n### Request-based utilization, not actual usage\n\nThis is the most important limitation for cost. CA compares pod `resource.requests`\n\nto node capacity. It does not measure actual CPU or memory consumption. When requests are inflated, CA perceives the cluster as more utilized than it is.\n\nAs noted in the Key Takeaways above, average CPU utilization across production clusters sits at 8%, and 69% of clusters are CPU-overprovisioned (up from 40% the prior year). Memory overprovisioning affects 79% of clusters.\n\nIn practice, CA sees those inflated requests and treats them as real demand. Therefore, scale-down is blocked on nodes that are nearly idle by actual usage. No tuning flag corrects for this because the root cause is the requests themselves, not the CA configuration.\n\n### Provisioning speed\n\nCA calls the cloud provider abstraction layer: ASG on AWS, MIG on GCP, VMSS on Azure. The full path from pending pod to ready node takes **3–4 minutes**. Workloads that scale up rapidly (burst traffic, job queues) spend several minutes in a degraded state while nodes provision.\n\n### Single replica scalability ceiling\n\nCA runs as a single pod. It does not scale horizontally. Clusters above 500 nodes with high pod churn can put significant CPU and memory pressure on the CA pod itself. This is a documented upstream limitation. For very large clusters, consider CA’s performance characteristics as a factor alongside its functional limitations.\n\n## Cluster Autoscaler vs Karpenter\n\nFor a full comparison, see the dedicated article: [Karpenter vs Cluster Autoscaler](https://cast.ai/blog/karpenter-vs-cluster-autoscaler/). This section covers the key operational differences.\n\n### Where CA wins\n\nCA is battle-tested, stable, and the **only production-grade option on GKE** as of mid-2026 (no official Karpenter provider exists for GKE). It integrates cleanly with existing ASG tooling and works across EKS, GKE, and AKS without rearchitecting your node group strategy.\n\n### Where Karpenter wins\n\nKarpenter selects instances from the entire cloud catalog, not from pre-defined groups. As a result, it provisions nodes in 60–90 seconds under typical conditions via direct EC2 RunInstances API calls versus 3–4 minutes for CA. Karpenter also handles Spot natively (declarative `capacity-type`\n\nin `NodePool`\n\n) and actively consolidates running workloads via its `WhenEmptyOrUnderutilized`\n\npolicy.\n\nAWS EKS Auto Mode uses Karpenter as its engine by default. Azure AKS Node Auto Provisioning (NAP), which also uses Karpenter, reached general availability in 2026. For teams on EKS or AKS who want to evaluate the switch, the full comparison covers migration paths, trade-offs at scale, and Spot handling differences in detail.\n\nIn contrast, teams on GKE, or those with heavily customized node group configurations, should keep CA and focus on the tuning described in this article.\n\nNeither CA nor Karpenter addresses the upstream problem: overprovisioned resource requests. Both autoscalers respond to `requests`\n\nas genuine demand. Fixing requests fixes both autoscalers simultaneously.\n\n## Conclusion\n\nThe Cluster Autoscaler is reliable, well-understood, and sufficient for many production clusters. Apply the tuning flags in this article, audit your scale-down blockers, and switch your expander to `least-waste`\n\n. Those changes recover meaningful capacity without replacing your autoscaling infrastructure.\n\nHowever, as detailed in the Limitations section above, CA’s fundamental constraint is the request-based utilization model. Tuning flags reduce waste at the margins, but CA cannot correct its own input data. The problem compounds when requests are systematically inflated across the cluster.\n\nCast AI corrects that input directly. It measures actual pod consumption over time and adjusts resource requests accordingly, so the next time CA checks utilization, it reads accurate numbers instead of the inflated defaults that blocked scale-down. Cast AI also supports in-place pod resizing (no pod restart required for most workloads; requires Kubernetes 1.27+ with the InPlacePodVerticalScaling feature gate, stable in 1.33), which means resource adjustments take effect without a deployment rollout. For stateful workloads, Cast AI Live Migration handles pod movement without disrupting storage-backed services.\n\nFor teams who have tuned CA and hit its ceiling, the next step is fixing what CA responds to. To take a broader look at all Kubernetes autoscaling layers, see the [guide to Kubernetes autoscaling for cloud cost optimization](https://cast.ai/blog/guide-to-kubernetes-autoscaling-for-cloud-cost-optimization/).\n\n## Frequently Asked Questions\n\n**What is the Kubernetes Cluster Autoscaler?** The Kubernetes Cluster Autoscaler is a component that automatically adjusts the number of nodes in a cluster. It adds nodes when pods are pending due to insufficient resources, and removes nodes when they have been underutilized for a sustained period (default: 10 minutes). CA is reactive: it provisions nodes after pods are already stuck in `Pending`\n\nstate, not before. It operates within pre-defined node groups and measures utilization using pod `resource.requests`\n\n, not actual CPU or memory usage.\n\n**How do I tune Cluster Autoscaler scale-down?** Start by identifying what is blocking scale-down: run `kubectl describe configmap cluster-autoscaler-status -n kube-system`\n\nand inspect the ScaleDown section. Common blockers are `--skip-nodes-with-system-pods=true`\n\nand `--skip-nodes-with-local-storage=true`\n\n(both default on). To make scale-down more aggressive, reduce `--scale-down-unneeded-time`\n\nfrom 10m to 5m, raise `--scale-down-utilization-threshold`\n\nfrom 0.5 to 0.65, and reduce `--scale-down-delay-after-add`\n\nto 5m. Also, ensure your requests are accurately sized. Overprovisioned requests cause CA to perceive nodes as utilized even when actual consumption is low, blocking scale-down silently.\n\n**Cluster Autoscaler vs Karpenter: which should I use?** Use CA if you are on GKE (no production Karpenter provider exists as of mid-2026), or if your team relies on existing ASG-based node group configurations. Consider Karpenter if you are on EKS or AKS and need faster provisioning (60–90 seconds under typical conditions vs. 3–4 minutes), dynamic instance selection from the full cloud catalog, or active workload consolidation. AWS EKS Auto Mode uses Karpenter by default. Azure AKS Node Auto Provisioning (Karpenter-based) reached general availability in 2026. For a full comparison with migration guidance, see [Karpenter vs Cluster Autoscaler](https://cast.ai/blog/karpenter-vs-cluster-autoscaler/).", "url": "https://wpnews.pro/news/kubernetes-cluster-autoscaler-how-it-works-and-how-to-tune-scale-up-and-scale", "canonical_source": "https://cast.ai/blog/kubernetes-cluster-autoscaler/", "published_at": "2026-07-15 09:20:40+00:00", "updated_at": "2026-07-15 09:21:22.119131+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["Kubernetes", "Cluster Autoscaler", "Karpenter", "Cast AI", "AWS", "GCP", "Azure"], "alternates": {"html": "https://wpnews.pro/news/kubernetes-cluster-autoscaler-how-it-works-and-how-to-tune-scale-up-and-scale", "markdown": "https://wpnews.pro/news/kubernetes-cluster-autoscaler-how-it-works-and-how-to-tune-scale-up-and-scale.md", "text": "https://wpnews.pro/news/kubernetes-cluster-autoscaler-how-it-works-and-how-to-tune-scale-up-and-scale.txt", "jsonld": "https://wpnews.pro/news/kubernetes-cluster-autoscaler-how-it-works-and-how-to-tune-scale-up-and-scale.jsonld"}}