# kubectl get nodes NotReady? 5-Minute Recovery for 6 kubelet, CNI, and Disk Causes > K8s_Troubleshooting_Guide Part 9 An alarm goes off in the middle of the night, you run `kubectl get nodes`, and a worker is `NotReady`. Pods that were running on it all pass through `Terminating` and vanish, and service latency starts to wobble. This post has one goal: **when you see that screen, branch to one of six causes within five minutes, safely evacuate Pods, and bring the node back.** Concepts are kept to a minimum. If you see this message, run this command. ## 1. The first screen to look at after you stop panicking First, confirm the node is actually NotReady and since when. ```bash kubectl get nodes -o wide kubectl get node -o jsonpath='{.status.conditions}' | jq ``` `NotReady` does not always mean the node is dead. It can be a brief gap in kubelet heartbeats, or kubelet itself flipping Ready to False because the disk filled up. The first real move is **reading Conditions to narrow the cause candidates**. ## 2. Conditions cheat sheet — narrow candidates in 5 seconds ```bash kubectl describe node | grep -A8 Conditions ``` Match the Reason/Message text in the output against the table below and you can branch immediately. | Conditions / original message | Cause it points to | Next command to run | |---|---|---| | `Ready=Unknown`, `NodeStatusUnknown`, "Kubelet stopped posting node status" | kubelet down or node↔API disconnect | `systemctl status kubelet` / `curl -k https://:6443/healthz` | | `Ready=False`, `KubeletNotReady`, "container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized" | CNI not initialized | `ls /etc/cni/net.d` / `journalctl -u kubelet \| grep -i cni` | | `DiskPressure=True`, "kubelet has disk pressure" | Disk pressure (image/log surge) | `df -h` / `df -i` / `crictl rmi --prune` | | `MemoryPressure=True`, "kubelet has insufficient memory available" | Memory pressure | `free -m` / `kubectl top node` | | `PIDPressure=True`, "kubelet has insufficient PID available" | Process/thread runaway | `ps -eLf \| wc -l` | | `Ready=False`, "failed to get container runtime" / containerd unresponsive | Container runtime down | `systemctl status containerd` / `crictl info` | > For reference, kubelet by default sets `DiskPressure` to True when imagefs/nodefs available capacity is **below 15%** (`evictionHard` default), and `MemoryPressure` when `memory.available<100Mi`. Once the threshold is hit, the node flips to NotReady and Pod eviction starts. ## 3. Diagnose and recover across 6 cause branches ### ① kubelet down ```bash systemctl status kubelet journalctl -u kubelet -f ``` If it is not `active (running)`, restart immediately. ```bash systemctl restart kubelet && systemctl status kubelet ``` ### ② CNI plugin not initialized This is the most common trap. If CNI config or binaries are missing, you get the message above. ```bash journalctl -u kubelet | grep -i cni ls /etc/cni/net.d # 비어 있으면 CNI conf 미배포 ls /opt/cni/bin # bridge, loopback 등 바이너리 존재 확인 ``` If config is empty, redeploy the CNI DaemonSet so conf is written again. ```bash kubectl rollout restart ds -n kube-system ``` ### ③ Disk / memory pressure ```bash df -h # nodefs 사용률 df -i # inode 고갈도 NotReady 유발 free -m ``` If disk is the cause, start by clearing unused images and logs. ```bash crictl rmi --prune journalctl --vacuum-size=200M ``` Once space is free, kubelet automatically clears DiskPressure and returns to Ready. ### ④ Container runtime down (containerd) After dockershim removal, containerd is the standard. ```bash systemctl status containerd crictl ps crictl info ``` If it is unresponsive, recover by restarting containerd, then kubelet. ```bash systemctl restart containerd && systemctl restart kubelet ``` ### ⑤ Certificate / node registration issues ```bash journalctl -u kubelet | grep -i certificate kubectl get csr ``` If you see a `Pending` CSR, approve it. ```bash kubectl certificate approve ``` ### ⑥ Node ↔ API network disconnect SSH into the node and check API server reachability. ```bash curl -k https://:6443/healthz # ok 안 나오면 네트워크/SG/방화벽 ss -tnp | grep 6443 ``` Check security groups, routing, and DNS. It is surprisingly common for the node to be fine but unable to reach the control plane. ## 4. Safe Pod evacuation and node return Before recovery work, stop new Pods from landing on that node and drain the existing ones. ```bash kubectl cordon kubectl drain --ignore-daemonsets --delete-emptydir-data # 복구 작업 수행 후 kubectl uncordon ``` If drain is blocked by a PDB (PodDisruptionBudget) or emptyDir, temporarily adjust the PDB's `minAvailable`, or force-delete with `--disable-eviction` (on environments where the API is not supported). Forced deletion risks data loss, so apply it carefully to stateful workloads. ## A word from the field In practice, most late-night NotReady incidents converge on two causes: **disk pressure** and **CNI not initialized**. When the alarm fires, check describe Conditions, then immediately inspect `df -h` and `ls /etc/cni/net.d`. Those two lines finish most of the branching; spend the rest of the time draining at a calmer pace. Managed offerings like EKS and GKE will auto-repair and replace NotReady nodes for you, but on a self-managed cluster you still have to run these six steps yourself. ## Conclusion: NotReady 5-minute recovery checklist 1. Narrow cause candidates with `kubectl describe node` Conditions 2. Check kubelet / containerd status → restart if needed 3. Confirm CNI conf and bin exist → DaemonSet rollout restart 4. Check pressure with `df -h`, `df -i`, `free -m` → clean images/logs 5. Check CSR and API healthz 6. `cordon` → `drain` → recover → `uncordon` ## References: official docs The primary sources for the behavior, settings, and errors covered in this post are the following official docs. Check them for version-specific options and exact behavior. - [Kubernetes official documentation](https://kubernetes.io/docs/home/) ## FAQ **Q. It was briefly NotReady, then went back to Ready on its own. Why?** A. If kubelet's heartbeat (`nodeStatusUpdateFrequency`, default 10s) is briefly delayed, the controller marks Ready=Unknown after `node-monitor-grace-period` (default 40s). If GC, a transient load spike, or network jitter delayed the heartbeat and then it recovered, the node returns to Ready automatically. If it keeps happening, suspect node load or etcd/API latency. **Q. Can I stop Pods from all draining as soon as the node goes NotReady?** A. When a node becomes NotReady, the `node.kubernetes.io/not-ready` (or `unreachable`) taint is applied, and Pods are evicted after the default `tolerationSeconds: 300`. For workloads that are not sensitive to brief outages, increase `tolerationSeconds` on the toleration to delay eviction. If you need fast failover, decrease it. **Q. CNI plugin not initialized, but the DaemonSet is Running?** A. The Pod can be Running even if conf never landed in `/etc/cni/net.d`. Check `ls /etc/cni/net.d` and `ls /opt/cni/bin` yourself, and if they are empty, force conf redeployment with `rollout restart`. Next installment (Part 10) covers **CrashLoopBackOff and OOMKilled — tracing causes when a Pod restarts forever**.

#kubelet NotReady

kubectl get nodes NotReady: Diagnose 6 Causes and Recover in 5 Minutes