Kubernetes Node NotReady: Why Nodes Go NotReady and How to Fix It

Kubernetes nodes enter NotReady status when the kubelet fails to renew its Node Lease within 40 seconds, triggering automatic taints and pod evictions after 5 minutes by default, according to Cast AI. The four primary causes are kubelet failure, network or CNI misconfiguration, resource pressure (memory, disk, or PID), and underlying cloud instance failure. Diagnosis begins with `kubectl describe node` and `journalctl -u kubelet`, and Cast AI's autoscaler can automatically provision replacement capacity to reduce recovery time.

Key takeaways - When a node becomes NotReady, Kubernetes applies an automatic taint. Pods without tolerations begin evicting within 5 minutes by default. - The kubelet renews its Node Lease every 10 seconds. After 40 seconds without a lease renewal, the node controller marks the node Unknown or NotReady. - Four primary causes exist: kubelet failure, network or CNI misconfiguration, resource pressure memory, disk, or PID , and underlying cloud instance failure. kubectl describe node and journalctl -u kubelet are the two most important first diagnostic tools.- Resource pressure conditions have hard numeric thresholds you can monitor before nodes fail. - Cast AI autoscaler detects pending pods after a NotReady event and provisions replacement capacity automatically, reducing mean time to recovery. What NotReady means Every Kubernetes node reports a set of conditions to the API server. The most important is Ready . When Ready is True, the kubelet is healthy and the node accepts pods. When Ready is False, the kubelet is running but actively reporting a problem. If Ready is Unknown, the API server has received no heartbeat for 40 seconds, which is the default node-monitor-grace-period . Four additional conditions indicate specific resource or network states. Each one tells you something precise about root cause. MemoryPressure: True when memory.available drops below 100Mi the default hard eviction threshold . DiskPressure: True when nodefs.available falls below 10%, nodefs.inodesFree below 5%, or imagefs.available below 15%. PIDPressure: True when available process IDs fall below the eviction threshold. This condition is Linux-only. NetworkUnavailable: True when the CNI plugin has not correctly configured networking on the node. The detection timeline Understanding the timeline matters for triage. Every 10 seconds, the kubelet sends a heartbeat to the API server. The node controller monitors these continuously. After 40 seconds without a heartbeat, the node controller transitions all conditions to Unknown. When a node goes NotReady, the node controller automatically applies two taints: node.kubernetes.io/not-ready:NoExecute and node.kubernetes.io/unreachable:NoExecute . Pods tolerate these taints for 300 seconds 5 minutes by default via tolerationSeconds . After that window, pods without a matching toleration are evicted. The DefaultTolerationSeconds admission controller, enabled by default since Kubernetes 1.8, automatically adds a 300-second toleration for not-ready:NoExecute and unreachable:NoExecute to any pod that does not already define one. As a result, only pods that explicitly set tolerationSeconds: 0 are evicted without delay. All other pods receive the full 300-second grace window. This mechanism replaced the deprecated pod-eviction-timeout flag in Kubernetes 1.24. Therefore, a kubelet restart that completes within 40 seconds produces minimal disruption. However, an outage longer than 5 minutes triggers cascading pod evictions across the cluster. Speed matters: fix the root cause before the eviction window closes. Kubernetes actually uses two heartbeat mechanisms. The primary is the Node Lease object coordination.k8s.io/v1 , which the kubelet renews every 10 seconds. The full NodeStatus update is sent less frequently: approximately every 5 minutes by default. This is why a kubelet that stops suddenly causes Unknown status within 40 seconds: the Lease stops renewing, and the node controller reacts to the lease expiry, not a full status update. How to diagnose a NotReady node Before diagnosing a NotReady node in production, cordon it first to prevent the scheduler from placing new pods there: kubectl cordon <node-name Cordoning protects application availability during investigation. It does not evict running pods. Run kubectl uncordon <node-name when the node is healthy and ready to rejoin the pool. Start from the control plane and work inward toward the node. This approach gives you the most information with the fewest SSH sessions. Step 1: Identify affected nodes kubectl get nodes Step 2: Get full condition and event details kubectl describe node <node-name Step 3: Check kubelet service run on the node via SSH or kubectl debug systemctl status kubelet journalctl -u kubelet -n 100 --no-pager Step 4: Check top memory/CPU consumers kubectl top node <node-name kubectl top pods --all-namespaces --sort-by=memory Restart kubelet after fixing the root cause systemctl restart kubelet kubectl describe node is the most information-dense first step. It shows all five conditions, their current status, and the timestamp of the last heartbeat. Scroll to the Conditions block first, then check the Events section at the bottom for recent kubelet-reported problems. If the node is completely unreachable via the API, SSH in directly. When SSH access is unavailable common on managed Kubernetes nodes , use kubectl debug to access the node’s host filesystem: kubectl debug node/<node-name -it --image=busybox -- chroot /host Once inside, you can run standard Linux diagnostic commands: check disk usage with df -h /var/lib/kubelet , view kubelet logs with journalctl -u kubelet -n 100 , and inspect running containers with crictl ps . Condition-to-diagnosis reference table | Condition | Meaning | First Command | Fix | |---|---|---|---| | All conditions Unknown | Total kubelet contact loss | systemctl status kubelet | Restart kubelet; restart containerd first if PLEG error appears | | MemoryPressure | Node running low on memory | kubectl top pods --sort-by=memory | Evict or move high-memory pods; set memory limits on unbounded workloads | | DiskPressure | Node running low on disk | df -h /var/lib/kubelet | Remove unused images with crictl rmi --prune ; clean exited containers | | PIDPressure | Too many processes on the node | ps aux | wc -l | Identify and evict fork-bombing containers | | NetworkUnavailable | CNI not configured on this node | kubectl logs daemonset/aws-node -n kube-system | Restart CNI DaemonSet pod; fix IAM permissions or subnet IP exhaustion | Common causes and fixes Kubelet failure Kubelet failure is the most common reason for a node not ready status. Three sub-causes appear most frequently in production clusters. PLEG errors. PLEG stands for Pod Lifecycle Event Generator. It is the kubelet component that tracks container state by polling the container runtime. When the container runtime containerd or CRI-O becomes unresponsive, PLEG stalls. The error looks like this: PLEG is not healthy: pleg was last seen active 4m30s ago; threshold is 3m0s First, restart containerd. Then restart the kubelet: systemctl restart containerd systemctl restart kubelet Certificate expiry. Kubelet TLS certificates have a one-year validity period by default. When they expire, the kubelet cannot authenticate to the API server. The log entry reads: x509: certificate has expired or is not yet valid For kubeadm clusters, rotate certificates with kubeadm certs renew all , then restart the kubelet. Additionally, enable automatic certificate rotation in the kubelet configuration rotateCertificates: true to prevent recurrence. Missing configuration. After manual node operations, the kubelet configuration file can be absent. The error surfaces as: Failed to load kubeconfig file: stat /etc/kubernetes/kubelet.conf: no such file or directory In this case, restore the kubelet configuration from a backup or rejoin the node to the cluster using your provisioning tooling. Network and CNI failure A NetworkUnavailable condition indicates the CNI plugin failed to configure networking on the node. Pods on the node cannot communicate with other pods or services. However, the node itself may still be reachable via SSH, which makes this condition easier to diagnose remotely than a full kubelet failure. For AWS VPC CNI, check the aws-node DaemonSet pods first: Check aws-node DaemonSet pod status on the affected node kubectl get pods -n kube-system -l k8s-app=aws-node -o wide Inspect logs for the specific pod on the failing node kubectl logs daemonset/aws-node -n kube-system Three root causes appear most often with AWS VPC CNI. First, insufficient IAM permissions prevent the node role from calling EC2 network interface APIs. Second, subnet IP exhaustion means the subnet has no free addresses to assign to pods. Third, a version mismatch between aws-vpc-cni and the Kubernetes control plane version causes compatibility failures. Check each in order and fix before restarting the DaemonSet pod. For Calico and Flannel, inspect the calico-node or kube-flannel DaemonSet pod on the affected node. A simple restart of the CNI DaemonSet pod on that specific node often resolves transient failures. Furthermore, check for any network policy conflicts that might be blocking CNI communication. Resource pressure Resource pressure conditions are predictable, and they are often preventable with correct resource requests and limits. Each condition has a specific threshold, and each has a targeted fix. MemoryPressure becomes True when available memory falls below 100Mi. Find the top memory consumers across the cluster: kubectl top pods --all-namespaces --sort-by=memory Look specifically for pods without memory limits. These pods grow unbounded and can saturate any node they land on. Set memory requests and limits that reflect actual usage. Additionally, evict or reschedule the top consumers to reduce immediate pressure on the affected node. DiskPressure has three independent thresholds. Start by checking which one triggered: Check disk usage on the kubelet data directory df -h /var/lib/kubelet Remove unused container images crictl rmi --prune Clean up exited containers crictl rm $ crictl ps -q --state=exited In most cases, accumulated container images and log files are the culprit. Therefore, configure a log rotation policy and image garbage collection thresholds in the kubelet configuration to prevent recurrence. PIDPressure is the least common of the three. However, when it appears, it almost always indicates a fork-bombing container: a process that spawns child processes faster than they exit. Run the following on the node: ps aux | wc -l If the process count is several thousand, identify which container is spawning them and evict it. Then investigate the application code for infinite loops or unchecked process spawning. The default kubelet eviction threshold is pid.available < 1000 configurable via evictionHard in kubelet configuration . PIDPressure is uncommon in most clusters but appears frequently in environments with containers that spawn many short-lived processes or threads. Cloud instance failure Cloud instances fail at the hardware level without warning. In this scenario, the kubelet stops responding without any application-level log entry. The node transitions to Unknown conditions across the board. Each major cloud provider has auto-repair behavior, though coverage varies. GKE automatically repairs nodes that remain NotReady for 10 or more minutes. AKS: Automatically re-provisions unhealthy nodes. The detection and re-provisioning timeline depends on the health check configuration, but the process typically begins within a few minutes of sustained NotReady status. EKS does not auto-repair by default. Therefore, for EKS clusters, follow this manual workflow: Step 1: Cordon the node to stop new pod scheduling kubectl cordon <node-name Step 2: Drain existing pods off the node kubectl drain <node-name --ignore-daemonsets --delete-emissary-data Step 3: Terminate the EC2 instance the ASG provisions a replacement aws ec2 terminate-instances --instance-ids $INSTANCE ID The Auto Scaling Group detects the terminated instance and launches a replacement automatically. The replacement node joins the cluster, pulls the CNI configuration, and becomes Ready within a few minutes. First cordon, then drain, then terminate: skipping the drain step risks data loss from stateful workloads. EKS managed node groups running Kubernetes 1.31 and later support automatic node replacement when a node fails EC2 health checks. For earlier versions, or self-managed node groups, use the manual cordon/drain/terminate workflow described above. Verify Recovery After applying any fix, confirm the node returned to Ready status: kubectl get node <node-name -w kubectl describe node <node-name | grep -A5 Conditions A healthy node shows Ready True with a recent LastHeartbeatTime. Wait 60-90 seconds for the node controller to update status after a kubelet restart. Preventing NotReady events with Cast AI Node NotReady events are inevitable in production Kubernetes clusters at scale. However, their business impact depends on three factors: how quickly you detect the failure, how fast replacement capacity provisions, and whether workloads had correct resource limits to avoid pressure conditions in the first place. Cast AI addresses both the reactive and the proactive sides of this problem. On the reactive side, Cast AI’s autoscaler continuously monitors pending pods. When a node goes NotReady and its pods become pending, Cast AI detects those pending workloads and provisions replacement nodes sized to the workload within minutes, without manual intervention. For context on how that compares to the built-in scheduler behavior, see the Kubernetes Cluster Autoscaler https://cast.ai/blog/kubernetes-cluster-autoscaler/ overview. On the proactive side, Cast AI’s workload rightsizing continuously monitors actual CPU and memory consumption and provides right-sizing recommendations. When automatic mode is enabled, it adjusts requests to match observed usage patterns, reducing the over-provisioning that leads to MemoryPressure events on densely scheduled nodes. Furthermore, rightsizing eliminates the guesswork that leads engineers to set limits too high, which wastes capacity, or too low, which triggers OOM kills and pressure conditions. Connect your cluster to Cast AI to view node health status, resource efficiency metrics, and autoscaling opportunities in a single dashboard. The connection is read-only by default and takes under five minutes to set up. Frequently Asked Questions What does node NotReady mean in Kubernetes? A NotReady node is one where the Ready condition is False or Unknown. False means the kubelet is running but reporting a problem to the API server. Unknown means the node controller has not seen a Node Lease renewal from the kubelet in 40 seconds the default node-monitor-grace-period . In both cases, the Kubernetes scheduler stops placing new pods on that node, and existing pods may be evicted after 5 minutes via the automatic taint mechanism. Why is my Kubernetes node NotReady? The four most common causes are: a stopped or crashed kubelet process, a CNI plugin failure that sets NetworkUnavailable to True, resource pressure conditions MemoryPressure, DiskPressure, or PIDPressure , and an underlying cloud instance hardware failure. Run kubectl describe node <node-name to see which conditions are True or Unknown, then trace the specific cause from there. How do I fix a NotReady node? First, cordon the node to prevent new pod scheduling: kubectl cordon <node-name . Then identify the failing condition with kubectl describe node . SSH into the node and run systemctl status kubelet and journalctl -u kubelet -n 100 . If SSH is unavailable, use kubectl debug node/<node-name -it --image=busybox -- chroot /host to access the node filesystem. Restart the kubelet if it is stopped. If PLEG errors appear in the logs, restart containerd first, then the kubelet. For resource pressure, free the constrained resource before restarting. For cloud instance failures, cordon and drain the node, then terminate and replace the underlying instance. After any fix, verify recovery with kubectl get node <node-name -w . What is resource pressure in Kubernetes? Resource pressure refers to three node conditions: MemoryPressure available memory below 100Mi , DiskPressure nodefs available below 10% or imagefs available below 15% , and PIDPressure available process IDs below 1000 by default, configurable via evictionHard . When any of these conditions is True, the kubelet begins evicting pods from the node to free the constrained resource. Persistent pressure usually indicates workloads without proper resource limits or a node that is undersized for its current workload.