{"slug": "kubernetes-node-notready-why-nodes-go-notready-and-how-to-fix-it", "title": "Kubernetes Node NotReady: Why Nodes Go NotReady and How to Fix It", "summary": "Kubernetes nodes enter NotReady status when the kubelet fails to renew its Node Lease within 40 seconds, triggering automatic taints and pod evictions after 5 minutes by default, according to Cast AI. The four primary causes are kubelet failure, network or CNI misconfiguration, resource pressure (memory, disk, or PID), and underlying cloud instance failure. Diagnosis begins with `kubectl describe node` and `journalctl -u kubelet`, and Cast AI's autoscaler can automatically provision replacement capacity to reduce recovery time.", "body_md": "## Key takeaways\n\n- When a node becomes NotReady, Kubernetes applies an automatic taint. Pods without tolerations begin evicting within 5 minutes by default.\n- The kubelet renews its Node Lease every 10 seconds. After 40 seconds without a lease renewal, the node controller marks the node Unknown or NotReady.\n- Four primary causes exist: kubelet failure, network or CNI misconfiguration, resource pressure (memory, disk, or PID), and underlying cloud instance failure.\n`kubectl describe node`\n\nand`journalctl -u kubelet`\n\nare the two most important first diagnostic tools.- Resource pressure conditions have hard numeric thresholds you can monitor before nodes fail.\n- Cast AI autoscaler detects pending pods after a NotReady event and provisions replacement capacity automatically, reducing mean time to recovery.\n\n## What NotReady means\n\nEvery Kubernetes node reports a set of conditions to the API server. The most important is **Ready**. When Ready is True, the kubelet is healthy and the node accepts pods. When Ready is False, the kubelet is running but actively reporting a problem. If Ready is Unknown, the API server has received no heartbeat for 40 seconds, which is the default `node-monitor-grace-period`\n\n.\n\nFour additional conditions indicate specific resource or network states. Each one tells you something precise about root cause.\n\n**MemoryPressure:** True when`memory.available`\n\ndrops below 100Mi (the default hard eviction threshold).**DiskPressure:** True when`nodefs.available`\n\nfalls below 10%,`nodefs.inodesFree`\n\nbelow 5%, or`imagefs.available`\n\nbelow 15%.**PIDPressure:** True when available process IDs fall below the eviction threshold. This condition is Linux-only.**NetworkUnavailable:** True when the CNI plugin has not correctly configured networking on the node.\n\n### The detection timeline\n\nUnderstanding the timeline matters for triage. Every 10 seconds, the kubelet sends a heartbeat to the API server. The node controller monitors these continuously. After 40 seconds without a heartbeat, the node controller transitions all conditions to Unknown. When a node goes NotReady, the node controller automatically applies two taints: `node.kubernetes.io/not-ready:NoExecute`\n\nand `node.kubernetes.io/unreachable:NoExecute`\n\n. Pods tolerate these taints for 300 seconds (5 minutes) by default via `tolerationSeconds`\n\n. After that window, pods without a matching toleration are evicted. The DefaultTolerationSeconds admission controller, enabled by default since Kubernetes 1.8, automatically adds a 300-second toleration for not-ready:NoExecute and unreachable:NoExecute to any pod that does not already define one. As a result, only pods that explicitly set `tolerationSeconds: 0`\n\nare evicted without delay. All other pods receive the full 300-second grace window. This mechanism replaced the deprecated `pod-eviction-timeout`\n\nflag in Kubernetes 1.24.\n\nTherefore, a kubelet restart that completes within 40 seconds produces minimal disruption. However, an outage longer than 5 minutes triggers cascading pod evictions across the cluster. Speed matters: fix the root cause before the eviction window closes.\n\nKubernetes actually uses two heartbeat mechanisms. The primary is the Node Lease object (`coordination.k8s.io/v1`\n\n), which the kubelet renews every 10 seconds. The full NodeStatus update is sent less frequently: approximately every 5 minutes by default. This is why a kubelet that stops suddenly causes Unknown status within 40 seconds: the Lease stops renewing, and the node controller reacts to the lease expiry, not a full status update.\n\n## How to diagnose a NotReady node\n\nBefore diagnosing a NotReady node in production, cordon it first to prevent the scheduler from placing new pods there:\n\n```\nkubectl cordon <node-name>\n```\n\nCordoning protects application availability during investigation. It does not evict running pods. Run `kubectl uncordon <node-name>`\n\nwhen the node is healthy and ready to rejoin the pool.\n\nStart from the control plane and work inward toward the node. This approach gives you the most information with the fewest SSH sessions.\n\n```\n# Step 1: Identify affected nodes\nkubectl get nodes\n\n# Step 2: Get full condition and event details\nkubectl describe node <node-name>\n\n# Step 3: Check kubelet service (run on the node via SSH or kubectl debug)\nsystemctl status kubelet\njournalctl -u kubelet -n 100 --no-pager\n\n# Step 4: Check top memory/CPU consumers\nkubectl top node <node-name>\nkubectl top pods --all-namespaces --sort-by=memory\n\n# Restart kubelet after fixing the root cause\nsystemctl restart kubelet\n```\n\n`kubectl describe node`\n\nis the most information-dense first step. It shows all five conditions, their current status, and the timestamp of the last heartbeat. Scroll to the **Conditions** block first, then check the **Events** section at the bottom for recent kubelet-reported problems.\n\nIf the node is completely unreachable via the API, SSH in directly. When SSH access is unavailable (common on managed Kubernetes nodes), use kubectl debug to access the node’s host filesystem:\n\n```\nkubectl debug node/<node-name> -it --image=busybox -- chroot /host\n```\n\nOnce inside, you can run standard Linux diagnostic commands: check disk usage with `df -h /var/lib/kubelet`\n\n, view kubelet logs with `journalctl -u kubelet -n 100`\n\n, and inspect running containers with `crictl ps`\n\n.\n\n### Condition-to-diagnosis reference table\n\n| Condition | Meaning | First Command | Fix |\n|---|---|---|---|\n| All conditions Unknown | Total kubelet contact loss | `systemctl status kubelet` | Restart kubelet; restart containerd first if PLEG error appears |\n| MemoryPressure | Node running low on memory | `kubectl top pods --sort-by=memory` | Evict or move high-memory pods; set memory limits on unbounded workloads |\n| DiskPressure | Node running low on disk | `df -h /var/lib/kubelet` | Remove unused images with `crictl rmi --prune` ; clean exited containers |\n| PIDPressure | Too many processes on the node | `ps aux | wc -l` | Identify and evict fork-bombing containers |\n| NetworkUnavailable | CNI not configured on this node | `kubectl logs daemonset/aws-node -n kube-system` | Restart CNI DaemonSet pod; fix IAM permissions or subnet IP exhaustion |\n\n## Common causes and fixes\n\n### Kubelet failure\n\nKubelet failure is the most common reason for a node not ready status. Three sub-causes appear most frequently in production clusters.\n\n**PLEG errors.** PLEG stands for Pod Lifecycle Event Generator. It is the kubelet component that tracks container state by polling the container runtime. When the container runtime (containerd or CRI-O) becomes unresponsive, PLEG stalls. The error looks like this:\n\n```\nPLEG is not healthy: pleg was last seen active 4m30s ago; threshold is 3m0s\n```\n\nFirst, restart containerd. Then restart the kubelet:\n\n```\nsystemctl restart containerd\nsystemctl restart kubelet\n```\n\n**Certificate expiry.** Kubelet TLS certificates have a one-year validity period by default. When they expire, the kubelet cannot authenticate to the API server. The log entry reads:\n\n```\nx509: certificate has expired or is not yet valid\n```\n\nFor kubeadm clusters, rotate certificates with `kubeadm certs renew all`\n\n, then restart the kubelet. Additionally, enable automatic certificate rotation in the kubelet configuration (`rotateCertificates: true`\n\n) to prevent recurrence.\n\n**Missing configuration.** After manual node operations, the kubelet configuration file can be absent. The error surfaces as:\n\n```\nFailed to load kubeconfig file: stat /etc/kubernetes/kubelet.conf: no such file or directory\n```\n\nIn this case, restore the kubelet configuration from a backup or rejoin the node to the cluster using your provisioning tooling.\n\n### Network and CNI failure\n\nA NetworkUnavailable condition indicates the CNI plugin failed to configure networking on the node. Pods on the node cannot communicate with other pods or services. However, the node itself may still be reachable via SSH, which makes this condition easier to diagnose remotely than a full kubelet failure.\n\nFor AWS VPC CNI, check the aws-node DaemonSet pods first:\n\n```\n# Check aws-node DaemonSet pod status on the affected node\nkubectl get pods -n kube-system -l k8s-app=aws-node -o wide\n\n# Inspect logs for the specific pod on the failing node\nkubectl logs daemonset/aws-node -n kube-system\n```\n\nThree root causes appear most often with AWS VPC CNI. First, insufficient IAM permissions prevent the node role from calling EC2 network interface APIs. Second, subnet IP exhaustion means the subnet has no free addresses to assign to pods. Third, a version mismatch between aws-vpc-cni and the Kubernetes control plane version causes compatibility failures. Check each in order and fix before restarting the DaemonSet pod.\n\nFor Calico and Flannel, inspect the calico-node or kube-flannel DaemonSet pod on the affected node. A simple restart of the CNI DaemonSet pod on that specific node often resolves transient failures. Furthermore, check for any network policy conflicts that might be blocking CNI communication.\n\n### Resource pressure\n\nResource pressure conditions are predictable, and they are often preventable with correct resource requests and limits. Each condition has a specific threshold, and each has a targeted fix.\n\n**MemoryPressure** becomes True when available memory falls below 100Mi. Find the top memory consumers across the cluster:\n\n```\nkubectl top pods --all-namespaces --sort-by=memory\n```\n\nLook specifically for pods without memory limits. These pods grow unbounded and can saturate any node they land on. Set memory requests and limits that reflect actual usage. Additionally, evict or reschedule the top consumers to reduce immediate pressure on the affected node.\n\n**DiskPressure** has three independent thresholds. Start by checking which one triggered:\n\n```\n# Check disk usage on the kubelet data directory\ndf -h /var/lib/kubelet\n\n# Remove unused container images\ncrictl rmi --prune\n\n# Clean up exited containers\ncrictl rm $(crictl ps -q --state=exited)\n```\n\nIn most cases, accumulated container images and log files are the culprit. Therefore, configure a log rotation policy and image garbage collection thresholds in the kubelet configuration to prevent recurrence.\n\n**PIDPressure** is the least common of the three. However, when it appears, it almost always indicates a fork-bombing container: a process that spawns child processes faster than they exit. Run the following on the node:\n\n```\nps aux | wc -l\n```\n\nIf the process count is several thousand, identify which container is spawning them and evict it. Then investigate the application code for infinite loops or unchecked process spawning.\n\nThe default kubelet eviction threshold is `pid.available < 1000`\n\n(configurable via `evictionHard`\n\nin kubelet configuration). PIDPressure is uncommon in most clusters but appears frequently in environments with containers that spawn many short-lived processes or threads.\n\n### Cloud instance failure\n\nCloud instances fail at the hardware level without warning. In this scenario, the kubelet stops responding without any application-level log entry. The node transitions to Unknown conditions across the board. Each major cloud provider has auto-repair behavior, though coverage varies.\n\nGKE automatically repairs nodes that remain NotReady for 10 or more minutes. AKS: Automatically re-provisions unhealthy nodes. The detection and re-provisioning timeline depends on the health check configuration, but the process typically begins within a few minutes of sustained NotReady status. EKS does not auto-repair by default. Therefore, for EKS clusters, follow this manual workflow:\n\n```\n# Step 1: Cordon the node to stop new pod scheduling\nkubectl cordon <node-name>\n\n# Step 2: Drain existing pods off the node\nkubectl drain <node-name> --ignore-daemonsets --delete-emissary-data\n\n# Step 3: Terminate the EC2 instance (the ASG provisions a replacement)\naws ec2 terminate-instances --instance-ids $INSTANCE_ID\n```\n\nThe Auto Scaling Group detects the terminated instance and launches a replacement automatically. The replacement node joins the cluster, pulls the CNI configuration, and becomes Ready within a few minutes. First cordon, then drain, then terminate: skipping the drain step risks data loss from stateful workloads.\n\nEKS managed node groups running Kubernetes 1.31 and later support automatic node replacement when a node fails EC2 health checks. For earlier versions, or self-managed node groups, use the manual cordon/drain/terminate workflow described above.\n\n### Verify Recovery\n\nAfter applying any fix, confirm the node returned to Ready status:\n\n```\nkubectl get node <node-name> -w\nkubectl describe node <node-name> | grep -A5 Conditions\n```\n\nA healthy node shows Ready True with a recent LastHeartbeatTime. Wait 60-90 seconds for the node controller to update status after a kubelet restart.\n\n## Preventing NotReady events with Cast AI\n\nNode NotReady events are inevitable in production Kubernetes clusters at scale. However, their business impact depends on three factors: how quickly you detect the failure, how fast replacement capacity provisions, and whether workloads had correct resource limits to avoid pressure conditions in the first place.\n\nCast AI addresses both the reactive and the proactive sides of this problem. On the reactive side, Cast AI’s autoscaler continuously monitors pending pods. When a node goes NotReady and its pods become pending, Cast AI detects those pending workloads and provisions replacement nodes sized to the workload within minutes, without manual intervention. For context on how that compares to the built-in scheduler behavior, see the [Kubernetes Cluster Autoscaler](https://cast.ai/blog/kubernetes-cluster-autoscaler/) overview.\n\nOn the proactive side, Cast AI’s workload rightsizing continuously monitors actual CPU and memory consumption and provides right-sizing recommendations. When automatic mode is enabled, it adjusts requests to match observed usage patterns, reducing the over-provisioning that leads to MemoryPressure events on densely scheduled nodes. Furthermore, rightsizing eliminates the guesswork that leads engineers to set limits too high, which wastes capacity, or too low, which triggers OOM kills and pressure conditions.\n\nConnect your cluster to Cast AI to view node health status, resource efficiency metrics, and autoscaling opportunities in a single dashboard. The connection is read-only by default and takes under five minutes to set up.\n\n## Frequently Asked Questions\n\n**What does node NotReady mean in Kubernetes?**\n\nA NotReady node is one where the Ready condition is False or Unknown. False means the kubelet is running but reporting a problem to the API server. Unknown means the node controller has not seen a Node Lease renewal from the kubelet in 40 seconds (the default node-monitor-grace-period). In both cases, the Kubernetes scheduler stops placing new pods on that node, and existing pods may be evicted after 5 minutes via the automatic taint mechanism.\n\n**Why is my Kubernetes node NotReady?** The four most common causes are: a stopped or crashed kubelet process, a CNI plugin failure that sets NetworkUnavailable to True, resource pressure conditions (MemoryPressure, DiskPressure, or PIDPressure), and an underlying cloud instance hardware failure. Run `kubectl describe node <node-name>`\n\nto see which conditions are True or Unknown, then trace the specific cause from there.\n\n**How do I fix a NotReady node?** First, cordon the node to prevent new pod scheduling: `kubectl cordon <node-name>`\n\n. Then identify the failing condition with `kubectl describe node`\n\n. SSH into the node and run `systemctl status kubelet`\n\nand `journalctl -u kubelet -n 100`\n\n. If SSH is unavailable, use `kubectl debug node/<node-name> -it --image=busybox -- chroot /host`\n\nto access the node filesystem. Restart the kubelet if it is stopped. If PLEG errors appear in the logs, restart containerd first, then the kubelet. For resource pressure, free the constrained resource before restarting. For cloud instance failures, cordon and drain the node, then terminate and replace the underlying instance. After any fix, verify recovery with `kubectl get node <node-name> -w`\n\n.\n\n**What is resource pressure in Kubernetes?** Resource pressure refers to three node conditions: MemoryPressure (available memory below 100Mi), DiskPressure (nodefs available below 10% or imagefs available below 15%), and PIDPressure (available process IDs below 1000 by default, configurable via evictionHard). When any of these conditions is True, the kubelet begins evicting pods from the node to free the constrained resource. Persistent pressure usually indicates workloads without proper resource limits or a node that is undersized for its current workload.", "url": "https://wpnews.pro/news/kubernetes-node-notready-why-nodes-go-notready-and-how-to-fix-it", "canonical_source": "https://cast.ai/blog/kubernetes-node-not-ready/", "published_at": "2026-07-23 08:58:23+00:00", "updated_at": "2026-07-23 09:26:35.397406+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools"], "entities": ["Kubernetes", "Cast AI", "kubelet", "CNI"], "alternates": {"html": "https://wpnews.pro/news/kubernetes-node-notready-why-nodes-go-notready-and-how-to-fix-it", "markdown": "https://wpnews.pro/news/kubernetes-node-notready-why-nodes-go-notready-and-how-to-fix-it.md", "text": "https://wpnews.pro/news/kubernetes-node-notready-why-nodes-go-notready-and-how-to-fix-it.txt", "jsonld": "https://wpnews.pro/news/kubernetes-node-notready-why-nodes-go-notready-and-how-to-fix-it.jsonld"}}