kubectl get nodes NotReady: Diagnose 6 Causes and Recover in 5 Minutes

kubectl get nodes NotReady? 5-Minute Recovery for 6 kubelet, CNI, and Disk Causes

K8s_Troubleshooting_Guide Part 9

An alarm goes off in the middle of the night, you run kubectl get nodes, and a worker is NotReady. Pods that were running on it all pass through Terminating and vanish, and service latency starts to wobble. This post has one goal: when you see that screen, branch to one of six causes within five minutes, safely evacuate Pods, and bring the node back. Concepts are kept to a minimum. If you see this message, run this command.

1. The first screen to look at after you stop panicking

First, confirm the node is actually NotReady and since when.

kubectl get nodes -o wide
kubectl get node <node> -o jsonpath='{.status.conditions}' | jq

NotReady does not always mean the node is dead. It can be a brief gap in kubelet heartbeats, or kubelet itself flipping Ready to False because the disk filled up. The first real move is reading Conditions to narrow the cause candidates.

2. Conditions cheat sheet — narrow candidates in 5 seconds

kubectl describe node <node> | grep -A8 Conditions

Match the Reason/Message text in the output against the table below and you can branch immediately.

For reference, kubelet by default sets DiskPressure to True when imagefs/nodefs available capacity is below 15% (evictionHard default), and MemoryPressure when memory.available<100Mi. Once the threshold is hit, the node flips to NotReady and Pod eviction starts.

3. Diagnose and recover across 6 cause branches

① kubelet down

systemctl status kubelet
journalctl -u kubelet -f

If it is not active (running), restart immediately.

systemctl restart kubelet && systemctl status kubelet

② CNI plugin not initialized

This is the most common trap. If CNI config or binaries are missing, you get the message above.

journalctl -u kubelet | grep -i cni
ls /etc/cni/net.d    # 비어 있으면 CNI conf 미배포
ls /opt/cni/bin      # bridge, loopback 등 바이너리 존재 확인

If config is empty, redeploy the CNI DaemonSet so conf is written again.

kubectl rollout restart ds <calico-node|cilium|aws-node> -n kube-system

③ Disk / memory pressure

df -h        # nodefs 사용률
df -i        # inode 고갈도 NotReady 유발
free -m

If disk is the cause, start by clearing unused images and logs.

crictl rmi --prune
journalctl --vacuum-size=200M

Once space is free, kubelet automatically clears DiskPressure and returns to Ready.

④ Container runtime down (containerd)

After dockershim removal, containerd is the standard.

systemctl status containerd
crictl ps
crictl info

If it is unresponsive, recover by restarting containerd, then kubelet.

systemctl restart containerd && systemctl restart kubelet

⑤ Certificate / node registration issues

journalctl -u kubelet | grep -i certificate
kubectl get csr

If you see a Pending CSR, approve it.

kubectl certificate approve <csr-name>

⑥ Node ↔ API network disconnect

SSH into the node and check API server reachability.

curl -k https://<API_SERVER>:6443/healthz   # ok 안 나오면 네트워크/SG/방화벽
ss -tnp | grep 6443

Check security groups, routing, and DNS. It is surprisingly common for the node to be fine but unable to reach the control plane.

4. Safe Pod evacuation and node return

Before recovery work, stop new Pods from landing on that node and drain the existing ones.

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
# 복구 작업 수행 후
kubectl uncordon <node>

If drain is blocked by a PDB (PodDisruptionBudget) or emptyDir, temporarily adjust the PDB's minAvailable, or force-delete with --disable-eviction (on environments where the API is not supported). Forced deletion risks data loss, so apply it carefully to stateful workloads.

A word from the field

In practice, most late-night NotReady incidents converge on two causes: disk pressure and CNI not initialized. When the alarm fires, check describe Conditions, then immediately inspect df -h and ls /etc/cni/net.d. Those two lines finish most of the branching; spend the rest of the time draining at a calmer pace. Managed offerings like EKS and GKE will auto-repair and replace NotReady nodes for you, but on a self-managed cluster you still have to run these six steps yourself.

Conclusion: NotReady 5-minute recovery checklist

1. Narrow cause candidates with kubectl describe node Conditions
2. Check kubelet / containerd status → restart if needed
3. Confirm CNI conf and bin exist → DaemonSet rollout restart
4. Check pressure with df -h, df -i, free -m → clean images/logs
5. Check CSR and API healthz
6. cordon → drain → recover → uncordon

References: official docs

The primary sources for the behavior, settings, and errors covered in this post are the following official docs. Check them for version-specific options and exact behavior.

• Kubernetes official documentation

FAQ

Q. It was briefly NotReady, then went back to Ready on its own. Why? A. If kubelet's heartbeat (nodeStatusUpdateFrequency, default 10s) is briefly delayed, the controller marks Ready=Unknown after node-monitor-grace-period (default 40s). If GC, a transient load spike, or network jitter delayed the heartbeat and then it recovered, the node returns to Ready automatically. If it keeps happening, suspect node load or etcd/API latency.

Q. Can I stop Pods from all draining as soon as the node goes NotReady? A. When a node becomes NotReady, the node.kubernetes.io/not-ready (or unreachable) taint is applied, and Pods are evicted after the default tolerationSeconds: 300. For workloads that are not sensitive to brief outages, increase tolerationSeconds on the toleration to delay eviction. If you need fast failover, decrease it.

Q. CNI plugin not initialized, but the DaemonSet is Running? A. The Pod can be Running even if conf never landed in /etc/cni/net.d. Check ls /etc/cni/net.d and ls /opt/cni/bin yourself, and if they are empty, force conf redeployment with rollout restart.

Next installment (Part 10) covers CrashLoopBackOff and OOMKilled — tracing causes when a Pod restarts forever.