Fixing OOMKilled Exit Code 137: A kubectl 30-Second Diagnosis and 5-Minute Recovery Runbook

Fixing OOMKilled Exit Code 137: A kubectl 30-Second Diagnosis and 5-Minute Recovery Runbook

If you landed here after seeing Last State: Terminated, Reason: OOMKilled, Exit Code: 137 in kubectl describe pod output, you're in the right place. We'll skip the theory for now and bring the dead Pod back first.

Exit code 137 is 128 + 9, meaning the process received SIGKILL and was force-killed. The key question is who killed it. There are usually three culprits: the container exceeded its own limit (OOMKilled), the whole node ran out of memory and the Pod was kicked off (Evicted), or a human killed it (SIGKILL). The one thing you must do in 30 seconds is split your 137 into one of these three branches. Get the branch wrong and you'll raise limits while missing node pressure—or you'll poke a perfectly healthy limit.

30-Second Diagnosis Decision Table: Which Branch Is Your 137?

Find your symptoms in the table below, then jump straight to the matching commands in the next section.

The key distinction: OOMKilled is container-scoped; Evicted is Pod-scoped (the node scheduler kicked it out). Even if kubectl get pod doesn't show STATUS as OOMKilled and it looks like a healthy Running, if the restart count is climbing, always check lastState.

Copy-Paste Diagnostic Commands

Once you've identified the branch above, copy and run these as-is. Just replace <pod> with yours.

# A) Last State 한눈에 보기 — Reason과 Exit Code가 여기 나옵니다
kubectl describe pod <pod> | grep -A5 "Last State"

# B) 스크립트/자동화용: reason + exitCode만 정확히 추출
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
# 출력 예: OOMKilled / 137

# C) 죽기 직전 실제 사용량 추세 (metrics-server 필요)
kubectl top pod <pod> --containers

# D) OOMKilling 이벤트를 클러스터 전역에서 수집
kubectl get events --field-selector reason=OOMKilling -A

# E) 노드로 들어가 커널 OOM killer 로그 직접 확인 (가장 확실한 증거)
#   nsenter/디버그 컨테이너로 노드 진입 후:
dmesg -T | grep -i "Out of memory: Killed process"
journalctl -k | grep -i oom

If E's dmesg shows a line like Killed process ... (java), that's airtight evidence that the kernel OOM killer picked that process and killed it. On most 2026 distros where cgroup v2 is the default, OOM is isolated more precisely at the cgroup level, so often only that container dies without affecting sibling containers.

Recovery by Cause + Preventing Recurrence

Recovery 1: Recalculate limits

Take the actual usage from kubectl top plus 30% headroom as the limit. Set requests to typical usage and limits to peak.

# Before — limit이 실사용보다 빠듯해 피크 때 즉사
resources:
  requests: { memory: "256Mi" }
  limits:   { memory: "512Mi" }

# After — top 측정 피크 700Mi 기준 여유 확보
resources:
  requests: { memory: "768Mi" }
  limits:   { memory: "1Gi" }

Recovery 2: Runtime heap vs. container limit mismatch (the most common trap)

On JVM and Node.js, if the runtime heap is set equal to or larger than the container limit, you get OOMKilled immediately at startup or right before GC. Metaspace, stacks, and native buffers outside the heap also consume memory.

# JVM — 절대값 -Xmx 박지 말고 limit 비율로 (limit 인식)
env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=75.0"   # limit의 75%만 힙에, 나머지는 네이티브 여유
resources:
  limits: { memory: "1Gi" }              # 힙 ~768Mi + 여유 256Mi

# Node.js — old space를 limit보다 작게
env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=768"    # limit 1Gi 대비 안전

Setting -Xmx1g with limits.memory: 1Gi (the same value) is the classic instant-death pattern. Always keep heap < limit.

Recovery 3: Eviction from unset limits

Without a limit, the Pod gets BestEffort or Burstable QoS, and when the node runs low on memory these Pods are evicted first. In node autoscaling environments like Karpenter, pressure often appears as the cluster tries to shrink empty nodes, so evictions show up more often. Set requests/limits explicitly to raise priority.

Preventing recurrence: Guaranteed QoS + monitoring

For your most important workloads, set requests.memory == limits.memory to get Guaranteed QoS, which puts you last in the eviction order.

resources:
  requests: { cpu: "500m", memory: "1Gi" }
  limits:   { cpu: "500m", memory: "1Gi" }   # 메모리 동일 → Guaranteed

Leave recommended values to VPA. Since 1.27+, in-place resize has gone beta and the flow of adjusting memory without a restart is becoming established.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: { name: app-vpa }
spec:
  targetRef: { apiVersion: apps/v1, kind: Deployment, name: app }
  updatePolicy: { updateMode: "Off" }   # 우선 추천값만 받아보기

HPA can scale on memory, but memory doesn't shrink (leaks, caches), so it is less stable than CPU-based scaling. Prefer VPA for the right memory size, and HPA (CPU/custom metrics) for traffic scaling.

Finally, get alerted before you die. Memory metrics are trending toward OpenTelemetry standardization, but the rule itself is the same.

# Prometheus alert: limit의 90% 도달 시 사전 경고
- alert: PodMemoryNearLimit
  expr: container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.9
  for: 5m
  labels: { severity: warning }
  annotations:
    summary: "{{ $labels.pod }} 메모리 limit 90% 초과 — OOMKilled 임박"

A note from the field: More than half of 137 incidents in production weren't because the limit was too small, but because of a config miss that pinned the JVM heap equal to the limit. Check MaxRAMPercentage before blindly raising the limit—you'll save cost and hit the root cause.

Conclusion: Don't misdiagnose sibling errors

Confusing 137 with other exit codes sends you digging in the wrong place.

Repeated OOM eventually transitions into CrashLoopBackOff, so when you see repeated restarts, check lastState reason first to tell whether the real cause is OOM.

The next post (K8s_Troubleshooting_Guide part 20) covers exit code 143, SIGTERM, and a runbook for guaranteeing graceful shutdown with the preStop hook.

References: Official docs

The primary sources for the behavior, settings, and errors in this post are the official docs below. Check them for version-specific options and exact behavior.

• Kubernetes official documentation

FAQ

Q. I raised the limit and still get OOMKilled. Why? A. Either a memory leak, or the JVM/Node heap setting didn't follow the new limit. If kubectl top pod --containers shows usage still climbing, it's a leak. If it dies immediately after start, check the heap settings (MaxRAMPercentage, --max-old-space-size).

Q. It's OOMKilled but kubectl get pod shows Running. A. The container may have died and restarted, so it's Running now. Check Last State and the RESTARTS count in kubectl describe pod, plus lastState.terminated.reason via jsonpath—that will surface the past OOM.

Q. Exit code 137 but no OOMKilled at all. A. It's likely an external SIGKILL, not kernel OOM. Suspect kubectl delete --grace-period=0, a deploy tool, node shutdown/drain, or a force kill after a liveness probe failure—then check events and the audit log.