Diagnose and Fix Pod Evicted "node was low on resource" in 5 Minutes

Pod Evicted Troubleshooting Guide: Diagnose the "node was low on resource" Error in 5 Minutes

K8s Troubleshooting Guide, Part 4

An alert goes off in the middle of the night. Pods that were running just fine suddenly line up in Evicted status, and the service is unstable. The most common mistake at this point is assuming "maybe the container has a bug?" and diving into application logs first.

Here's the bottom line: don't blame the container just because it was evicted. The culprit is the node. Evicted does not mean the container died—it means kubelet decided "this node is low on resources, so I'm kicking you out" and forcibly removed it. That's why diagnosis should start with the node, not the Pod.

In this installment, we'll draw a hard line between Evicted and other failures, pinpoint the cause in 5 minutes by reading Conditions from kubectl describe node, then cover cause-specific fixes and recurrence prevention in one pass.

1. Evicted vs CrashLoopBackOff vs OOMKilled: Who Killed It?

These three look similar on the surface, but the responsible party is completely different. Mix them up and you'll end up fixing the wrong thing.

The key point is the difference between OOMKilled and MemoryPressure eviction.

• OOMKilled: Your container exceeded its own cgroup limit. Other Pods are fine. Exit code 137.
• MemoryPressure eviction: Node-wide available memory dropped below the threshold. kubelet picks multiple lower-priority Pods and kicks them out. There is no exit code—the Pod phase becomes Failed with Reason Evicted.

In short: if a single Pod dies with 137, look at its limit. If multiple Pods get Evicted at once, look at the node.

2. Diagnostic Routine: get pods → describe node Conditions

If you suspect eviction, follow this sequence as written.

2-1. Identify Evicted Pods

# Failed phase(=Evicted 포함) Pod 전체 조회
kubectl get pods -A --field-selector status.phase=Failed

2-2. Check the Eviction Reason

Describing a specific Pod gives you the smoking gun.

kubectl describe pod <pod-name> -n <namespace>

Status:   Failed
Reason:   Evicted
Message:  The node was low on resource: ephemeral-storage.
          Container app was using 4521Mi, request is 0.

The low on resource: <resource> part of Message tells you which resource ran short. You'll see ephemeral-storage, memory, pids, and so on.

2-3. Read Node Conditions

Now look at the node that Pod was on.

kubectl describe node <node-name>

Conditions:
  Type             Status
  MemoryPressure   False
  DiskPressure     True     <-- 범인
  PIDPressure      False
  Ready            True

The Condition that is True is the cause. The example above is disk pressure. To extract this quickly as JSON:

kubectl get node <node-name> -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'

2-4. Understand kubelet Eviction Thresholds

kubelet operates with two kinds of thresholds.

• --eviction-hard: Immediate eviction. No grace period. Examples: memory.available<100Mi, nodefs.available<10%, imagefs.available<15%, pid.available<10%
• --eviction-soft: When the threshold is reached, wait for --eviction-soft-grace-period; if it doesn't recover, then evict. This prevents overreacting to a sudden spike.

The flow looks like this: soft threshold reached → grace period countdown → if it drops to the hard threshold in the meantime, immediate eviction; otherwise eviction after grace expires. nodefs is the disk used by emptyDir, logs, and container writable layers; imagefs is the image/container layer store.

3. Cause-Specific Fixes: Before/After

3-1. ephemeral-storage Blowup (Most Common)

Workloads that dump logs into emptyDir instead of stdout, or that create temp files without bound, fill up the node's nodefs and trigger DiskPressure. On top of that, if the ephemeral-storage request is 0, the scheduler never accounts for this load at all.

Before — no limits:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  # ephemeral-storage 미설정 → 스케줄러가 디스크 압박을 모름

After — declare ephemeral-storage + emptyDir sizeLimit:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
    ephemeral-storage: "1Gi"
  limits:
    ephemeral-storage: "2Gi"   # 초과 시 이 Pod만 Evicted
volumes:
  - name: cache
    emptyDir:
      sizeLimit: "1Gi"          # 볼륨 자체에 상한

Setting limits.ephemeral-storage means only the offending Pod gets evicted first—before it takes the whole node down—protecting other Pods.

3-2. Tune Image/Container GC Thresholds

DiskPressure also fires when imagefs fills up with unused images. Adjust kubelet GC settings (KubeletConfiguration).

# kubelet config
imageGCHighThresholdPercent: 80   # 이 % 넘으면 GC 시작
imageGCLowThresholdPercent: 70    # 이 %까지 정리
evictionHard:
  imagefs.available: "15%"

3-3. Clean Node Disk Directly (containerd/CRI-O Standard)

In a 2026 environment, dockershim has been gone for a long time, so crictl-based cleanup is the standard.

# 어디가 찼는지 점검
du -sh /var/lib/containerd/* | sort -rh | head
journalctl --disk-usage

# 미사용 이미지 정리 (containerd/CRI-O)
crictl rmi --prune

# (구형 docker 노드라면)
docker system prune -a

3-4. Missing requests/limits → Improve Scheduling

Without requests, the scheduler mistakenly thinks the node is "empty" and keeps packing Pods onto it, until the node is overloaded and evictions cascade. Lock in a standard resource spec and the scheduler will avoid pressure-hit nodes in the first place.

resources:
  requests: { cpu: "200m", memory: "256Mi", ephemeral-storage: "1Gi" }
  limits:   { cpu: "500m", memory: "512Mi", ephemeral-storage: "2Gi" }

A note from the field: In large clusters, I've often seen 80% of Evicted spikes start on "one particular node." This gets trickier with Karpenter or Cluster Autoscaler, because the autoscaler's normal behavior of pulling Pods to shrink a node looks mixed in with abnormal evictions caused by resource shortage. That's why I always start with the Message from describe pod. If Reason is Evicted with low on resource, it's a resource problem; if it's just a node shutdown, it's the autoscaler. The two require opposite responses.

4. Bulk-Clean Evicted Pods + Prevent Recurrence

4-1. Clean Up Accumulated Evicted Pods

# Failed phase 한정으로 일괄 삭제 (Running/Pending 건드리지 않음)
kubectl delete pods --field-selector status.phase=Failed -A

⚠️ Caution: This is cleanup, not a fix. If the root cause (disk, memory, limits) is still there, Evicted Pods will pile up again in a few minutes. Always remove the cause from section 3 first.

4-2. Limit Concurrent Evictions with PodDisruptionBudget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2        # 최소 2개는 항상 유지
  selector:
    matchLabels:
      app: web

A PDB keeps service availability by preventing everything from disappearing at once during node drain or autoscaler scale-in (but remember that hard eviction can ignore a PDB).

4-3. Monitoring + Alerting

Proactively watch node resources with node_exporter and kube-state-metrics.

# nodefs 가용량 15% 미만 알림
node_filesystem_avail_bytes{mountpoint="/"} 
  / node_filesystem_size_bytes{mountpoint="/"} < 0.15

Recommended alert thresholds:

• Disk available < 20% (warning), < 15% (critical, just before hard eviction)
• Memory available < 300Mi (warning)
• Tracking usage and cost for ephemeral-storage as well—via OpenCost or similar—lets you catch blowup workloads ahead of time.

5. Wrap-up: Recurrence Prevention Checklist

• Standardize requests/limits (cpu, memory, ephemeral-storage) on every workload
• Set sizeLimit on emptyDir
• Review kubelet imageGCHighThresholdPercent/evictionHard
• Run crictl rmi --prune regularly, or automate it
• Apply a PDB to critical services
• Set node_exporter disk/memory alert thresholds

When you hit Evicted, just remember the order. describe pod to see which resource → describe node Conditions to see which node → fix by cause. Those three steps and you're done in 5 minutes.

Next time we'll cover diagnosing Pods stuck Pending and not scheduling (Insufficient cpu/memory, taint/toleration, nodeAffinity).

FAQ

Q. Why don't Evicted Pods disappear automatically? A. kubelet keeps the Pod object in Failed phase as an eviction record. It's a debugging breadcrumb, not a GC target, so you have to clean it up yourself with kubectl delete pods --field-selector status.phase=Failed.

Q. Is setting limits enough? A. No. limits stop an individual Pod from running wild, but the scheduler decides node placement based on requests. Without requests, node overpacking continues—you need both.

Q. If an ephemeral-storage limit is exceeded, is it OOMKilled or Evicted? A. Evicted. Exceeding ephemeral-storage is a disk problem, not memory, so it is handled as kubelet eviction (Reason: Evicted), not exit code 137 (OOMKilled).

Q. What if only one node keeps getting pressured? A. If the same node keeps getting hit, a heavy workload has likely concentrated there. Spread with PodTopologySpread, or cordon/drain that node and inspect it. In a Karpenter environment, bumping the node type itself (disk capacity) is also an option.

Q. What should I set the soft eviction grace period to? A. Typically keep memory short (30s–1m30s) and disk a bit longer (around 2m). Too long and you'll drop to the hard threshold and get force-evicted, so we recommend tuning it to match your monitoring alert interval.