# Pod Evicted Troubleshooting Guide: Diagnose the "node was low on resource" Error in 5 Minutes > K8s Troubleshooting Guide, Part 4 An alert goes off in the middle of the night. Pods that were running just fine suddenly line up in `Evicted` status, and the service is unstable. The most common mistake at this point is assuming "maybe the container has a bug?" and diving into application logs first. Here's the bottom line: **don't blame the container just because it was evicted. The culprit is the node.** `Evicted` does not mean the container died—it means kubelet decided "this node is low on resources, so I'm kicking you out" and forcibly removed it. That's why diagnosis should start with the **node**, not the Pod. In this installment, we'll draw a hard line between `Evicted` and other failures, pinpoint the cause in 5 minutes by reading Conditions from `kubectl describe node`, then cover cause-specific fixes and recurrence prevention in one pass. ## 1. Evicted vs CrashLoopBackOff vs OOMKilled: Who Killed It? These three look similar on the surface, but the **responsible party** is completely different. Mix them up and you'll end up fixing the wrong thing. | Status | Actor (who killed it) | Trigger | Exit code | Where to fix | |---|---|---|---|---| | **Evicted** | **Node kubelet** | Node-wide resource pressure (Disk/Memory/PID Pressure) | None (Pod phase=Failed) | **Node resources / requests·limits** | | **CrashLoopBackOff** | The container itself | Process repeatedly exits abnormally | 1, 2, etc. (varies by app) | Application code/config | | **OOMKilled** | Container cgroup | Container exceeded its own memory **limit** | **137** (128+[SIGKILL](/blog/npm-err-code-elifecycle-해결법-errno-1134sigkill-원인별-진단)) | That container's memory limit | The key point is the **difference between OOMKilled and MemoryPressure eviction**. - **OOMKilled**: *Your container* exceeded its own cgroup limit. Other Pods are fine. Exit code 137. - **MemoryPressure eviction**: *Node-wide* available memory dropped below the threshold. kubelet picks **multiple** lower-priority Pods and kicks them out. There is no exit code—the Pod phase becomes `Failed` with Reason `Evicted`. In short: if a single Pod dies with 137, look at its limit. If multiple Pods get Evicted at once, look at the node. ## 2. Diagnostic Routine: get pods → describe node Conditions If you suspect eviction, follow this sequence as written. ### 2-1. Identify Evicted Pods ```bash # Failed phase(=Evicted 포함) Pod 전체 조회 kubectl get pods -A --field-selector status.phase=Failed ``` ### 2-2. Check the Eviction Reason Describing a specific Pod gives you the smoking gun. ```bash kubectl describe pod -n ``` ```text Status: Failed Reason: Evicted Message: The node was low on resource: ephemeral-storage. Container app was using 4521Mi, request is 0. ``` The `low on resource: ` part of `Message` tells you which resource ran short. You'll see `ephemeral-storage`, `memory`, `pids`, and so on. ### 2-3. Read Node Conditions Now look at the node that Pod was on. ```bash kubectl describe node ``` ```text Conditions: Type Status MemoryPressure False DiskPressure True <-- 범인 PIDPressure False Ready True ``` The Condition that is `True` is the cause. The example above is disk pressure. To extract this quickly as JSON: ```bash kubectl get node -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}' ``` ### 2-4. Understand kubelet Eviction Thresholds kubelet operates with two kinds of thresholds. - **`--eviction-hard`**: Immediate eviction. No grace period. Examples: `memory.available<100Mi`, `nodefs.available<10%`, `imagefs.available<15%`, `pid.available<10%` - **`--eviction-soft`**: When the threshold is reached, wait for `--eviction-soft-grace-period`; if it doesn't recover, then evict. This prevents overreacting to a sudden spike. The flow looks like this: soft threshold reached → grace period countdown → if it drops to the hard threshold in the meantime, immediate eviction; otherwise eviction after grace expires. `nodefs` is the disk used by emptyDir, logs, and container writable layers; `imagefs` is the image/container layer store. ## 3. Cause-Specific Fixes: Before/After ### 3-1. ephemeral-storage Blowup (Most Common) Workloads that dump logs into emptyDir instead of stdout, or that create temp files without bound, fill up the node's `nodefs` and trigger DiskPressure. On top of that, if the `ephemeral-storage` request is `0`, the scheduler never accounts for this load at all. **Before** — no limits: ```yaml resources: requests: cpu: "250m" memory: "256Mi" # ephemeral-storage 미설정 → 스케줄러가 디스크 압박을 모름 ``` **After** — declare ephemeral-storage + emptyDir sizeLimit: ```yaml resources: requests: cpu: "250m" memory: "256Mi" ephemeral-storage: "1Gi" limits: ephemeral-storage: "2Gi" # 초과 시 이 Pod만 Evicted volumes: - name: cache emptyDir: sizeLimit: "1Gi" # 볼륨 자체에 상한 ``` Setting `limits.ephemeral-storage` means **only the offending Pod** gets evicted first—before it takes the whole node down—protecting other Pods. ### 3-2. Tune Image/Container GC Thresholds DiskPressure also fires when `imagefs` fills up with unused images. Adjust kubelet GC settings (KubeletConfiguration). ```yaml # kubelet config imageGCHighThresholdPercent: 80 # 이 % 넘으면 GC 시작 imageGCLowThresholdPercent: 70 # 이 %까지 정리 evictionHard: imagefs.available: "15%" ``` ### 3-3. Clean Node Disk Directly (containerd/CRI-O Standard) In a 2026 environment, dockershim has been gone for a long time, so `crictl`-based cleanup is the standard. ```bash # 어디가 찼는지 점검 du -sh /var/lib/containerd/* | sort -rh | head journalctl --disk-usage # 미사용 이미지 정리 (containerd/CRI-O) crictl rmi --prune # (구형 docker 노드라면) docker system prune -a ``` ### 3-4. Missing requests/limits → Improve Scheduling Without requests, the scheduler mistakenly thinks the node is "empty" and keeps packing Pods onto it, until the node is overloaded and evictions cascade. Lock in a standard resource spec and the scheduler will avoid pressure-hit nodes in the first place. ```yaml resources: requests: { cpu: "200m", memory: "256Mi", ephemeral-storage: "1Gi" } limits: { cpu: "500m", memory: "512Mi", ephemeral-storage: "2Gi" } ``` > **A note from the field**: In large clusters, I've often seen 80% of Evicted spikes start on "one particular node." This gets trickier with Karpenter or Cluster Autoscaler, because the autoscaler's normal behavior of pulling Pods to shrink a node looks mixed in with abnormal evictions caused by resource shortage. That's why I always start with the Message from `describe pod`. If Reason is `Evicted` with `low on resource`, it's a resource problem; if it's just a node shutdown, it's the autoscaler. The two require opposite responses. ## 4. Bulk-Clean Evicted Pods + Prevent Recurrence ### 4-1. Clean Up Accumulated Evicted Pods ```bash # Failed phase 한정으로 일괄 삭제 (Running/Pending 건드리지 않음) kubectl delete pods --field-selector status.phase=Failed -A ``` ⚠️ **Caution**: This is cleanup, not a fix. If the root cause (disk, memory, limits) is still there, Evicted Pods will pile up again in a few minutes. Always remove the cause from section 3 first. ### 4-2. Limit Concurrent Evictions with PodDisruptionBudget ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web-pdb spec: minAvailable: 2 # 최소 2개는 항상 유지 selector: matchLabels: app: web ``` A PDB keeps service availability by preventing everything from disappearing at once during node drain or autoscaler scale-in (but remember that hard eviction can ignore a PDB). ### 4-3. Monitoring + Alerting Proactively watch node resources with node_exporter and kube-state-metrics. ```promql # nodefs 가용량 15% 미만 알림 node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.15 ``` Recommended alert thresholds: - Disk available < **20%** (warning), < **15%** (critical, just before hard eviction) - Memory available < **300Mi** (warning) - Tracking usage and cost for `ephemeral-storage` as well—via OpenCost or similar—lets you catch blowup workloads ahead of time. ## 5. Wrap-up: Recurrence Prevention Checklist - [ ] Standardize `requests`/`limits` (cpu, memory, **ephemeral-storage**) on every workload - [ ] Set `sizeLimit` on emptyDir - [ ] Review kubelet `imageGCHighThresholdPercent`/`evictionHard` - [ ] Run `crictl rmi --prune` regularly, or automate it - [ ] Apply a PDB to critical services - [ ] Set node_exporter disk/memory alert thresholds When you hit `Evicted`, just remember the order. **describe pod to see which resource → describe node Conditions to see which node → fix by cause.** Those three steps and you're done in 5 minutes. Next time we'll cover diagnosing **Pods stuck Pending and not scheduling** (Insufficient cpu/memory, taint/toleration, nodeAffinity). ## FAQ **Q. Why don't Evicted Pods disappear automatically?** A. kubelet keeps the Pod object in `Failed` phase as an eviction record. It's a debugging breadcrumb, not a GC target, so you have to clean it up yourself with `kubectl delete pods --field-selector status.phase=Failed`. **Q. Is setting limits enough?** A. No. `limits` stop an individual Pod from running wild, but the scheduler decides node placement based on `requests`. Without requests, node overpacking continues—you need both. **Q. If an ephemeral-storage limit is exceeded, is it OOMKilled or Evicted?** A. Evicted. Exceeding ephemeral-storage is a disk problem, not memory, so it is handled as kubelet eviction (Reason: Evicted), not exit code 137 (OOMKilled). **Q. What if only one node keeps getting pressured?** A. If the same node keeps getting hit, a heavy workload has likely concentrated there. Spread with PodTopologySpread, or cordon/drain that node and inspect it. In a Karpenter environment, bumping the node type itself (disk capacity) is also an option. **Q. What should I set the soft eviction grace period to?** A. Typically keep memory short (30s–1m30s) and disk a bit longer (around 2m). Too long and you'll drop to the hard threshold and get force-evicted, so we recommend tuning it to match your monitoring alert interval.

#kubelet eviction

Diagnose and Fix Pod Evicted "node was low on resource" in 5 Minutes