# Fixing OOMKilled Exit Code 137: A kubectl 30-Second Diagnosis and 5-Minute Recovery Runbook If you landed here after seeing `Last State: Terminated, Reason: OOMKilled, Exit Code: 137` in `kubectl describe pod` output, you're in the right place. We'll skip the theory for now and **bring the dead Pod back first.** Exit code 137 is `128 + 9`, meaning the process **received [SIGKILL](/blog/npm-err-code-elifecycle-해결법-errno-1134sigkill-원인별-진단) and was force-killed**. The key question is who killed it. There are usually three culprits: the container exceeded its own limit (OOMKilled), the whole node ran out of memory and the Pod was kicked off (Evicted), or a human killed it (SIGKILL). **The one thing you must do in 30 seconds is split your 137 into one of these three branches.** Get the branch wrong and you'll raise limits while missing node pressure—or you'll poke a perfectly healthy limit. ## 30-Second Diagnosis Decision Table: Which Branch Is Your 137? Find your symptoms in the table below, then jump straight to the matching commands in the next section. | Symptom (phrase in describe / events) | Diagnosis | Next action | |---|---|---| | `Exit Code: 137` + `Reason: OOMKilled` | **Container memory limit exceeded** | Diagnostic commands A·B·C → Recovery 1·2 | | `Reason: Evicted`, `message: The node was low on resource: memory` | **Node-wide memory pressure** | Diagnostic commands D·E → Recovery 3 | | `Exit Code: 137` but no `OOMKilled` | **Manual `kubectl delete --grace-period=0` or external SIGKILL** | Check events and audit log; inspect deploys/scripts | The key distinction: **OOMKilled is container-scoped; Evicted is Pod-scoped (the node scheduler kicked it out)**. Even if `kubectl get pod` doesn't show STATUS as `OOMKilled` and it looks like a healthy Running, if the restart count is climbing, always check lastState. ## Copy-Paste Diagnostic Commands Once you've identified the branch above, copy and run these as-is. Just replace `` with yours. ```bash # A) Last State 한눈에 보기 — Reason과 Exit Code가 여기 나옵니다 kubectl describe pod | grep -A5 "Last State" # B) 스크립트/자동화용: reason + exitCode만 정확히 추출 kubectl get pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}' # 출력 예: OOMKilled / 137 # C) 죽기 직전 실제 사용량 추세 (metrics-server 필요) kubectl top pod --containers # D) OOMKilling 이벤트를 클러스터 전역에서 수집 kubectl get events --field-selector reason=OOMKilling -A # E) 노드로 들어가 커널 OOM killer 로그 직접 확인 (가장 확실한 증거) # nsenter/디버그 컨테이너로 노드 진입 후: dmesg -T | grep -i "Out of memory: Killed process" journalctl -k | grep -i oom ``` If E's `dmesg` shows a line like `Killed process ... (java)`, that's airtight evidence that **the kernel OOM killer picked that process and killed it**. On most 2026 distros where cgroup v2 is the default, OOM is isolated more precisely at the cgroup level, so often only that container dies without affecting sibling containers. ## Recovery by Cause + Preventing Recurrence ### Recovery 1: Recalculate limits Take the actual usage from `kubectl top` plus 30% headroom as the limit. Set requests to typical usage and limits to peak. ```yaml # Before — limit이 실사용보다 빠듯해 피크 때 즉사 resources: requests: { memory: "256Mi" } limits: { memory: "512Mi" } # After — top 측정 피크 700Mi 기준 여유 확보 resources: requests: { memory: "768Mi" } limits: { memory: "1Gi" } ``` ### Recovery 2: Runtime heap vs. container limit mismatch (the most common trap) On JVM and Node.js, **if the runtime heap is set equal to or larger than the container limit, you get OOMKilled immediately at startup or right before GC**. Metaspace, stacks, and native buffers outside the heap also consume memory. ```yaml # JVM — 절대값 -Xmx 박지 말고 limit 비율로 (limit 인식) env: - name: JAVA_TOOL_OPTIONS value: "-XX:MaxRAMPercentage=75.0" # limit의 75%만 힙에, 나머지는 네이티브 여유 resources: limits: { memory: "1Gi" } # 힙 ~768Mi + 여유 256Mi # Node.js — old space를 limit보다 작게 env: - name: NODE_OPTIONS value: "--max-old-space-size=768" # limit 1Gi 대비 안전 ``` Setting `-Xmx1g` with `limits.memory: 1Gi` (the same value) is the classic instant-death pattern. Always keep heap < limit. ### Recovery 3: Eviction from unset limits Without a limit, the Pod gets **BestEffort or Burstable** QoS, and when the node runs low on memory **these Pods are evicted first**. In node autoscaling environments like Karpenter, pressure often appears as the cluster tries to shrink empty nodes, so evictions show up more often. Set requests/limits explicitly to raise priority. ### Preventing recurrence: Guaranteed QoS + monitoring For your most important workloads, set `requests.memory == limits.memory` to get **Guaranteed QoS**, which puts you last in the eviction order. ```yaml resources: requests: { cpu: "500m", memory: "1Gi" } limits: { cpu: "500m", memory: "1Gi" } # 메모리 동일 → Guaranteed ``` Get recommended values from VPA in recommendation mode (`updateMode: "Off"`) first; in this mode VPA only computes recommendations and does not change Pods ([VPA](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler)). In-place Pod resize, which changes container resources without a restart, was alpha in 1.27, beta (on by default) in 1.33, and GA in 1.35 ([resize container resources](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/)). ```yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: { name: app-vpa } spec: targetRef: { apiVersion: apps/v1, kind: Deployment, name: app } updatePolicy: { updateMode: "Off" } # 우선 추천값만 받아보기 ``` > HPA can scale on memory, but memory doesn't shrink (leaks, caches), so it is less stable than CPU-based scaling. Prefer VPA for the right memory size, and HPA (CPU/custom metrics) for traffic scaling. Finally, get alerted before you die. Memory metrics are trending toward OpenTelemetry standardization, but the rule itself is the same. ```yaml # Prometheus alert: limit의 90% 도달 시 사전 경고 - alert: PodMemoryNearLimit expr: container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.9 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.pod }} 메모리 limit 90% 초과 — OOMKilled 임박" ``` **Check this first:** before blindly raising the limit, confirm the runtime heap setting (`MaxRAMPercentage`, `--max-old-space-size`) is comfortably below the limit. A heap pinned equal to the limit is a classic cause you can fix without raising the limit at all. ## Decide from Measurements: Working Set, Sizing Requests and Limits, Spotting Leaks ### What `kubectl top` and eviction look at: the working set `kubectl top` and the kubelet's memory eviction decisions use the **working set**: cgroup memory usage minus inactive file cache. A node's `memory.available` is computed as `capacity - workingSet` ([node-pressure eviction docs](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/)). So when sizing limits, `container_memory_working_set_bytes` matches real behavior better than RSS or total usage (`container_memory_usage_bytes`). The container OOM kill itself is triggered by the kernel when the cgroup limit (`memory.max` on cgroup v2) is hit and nothing can be reclaimed. ### Start requests and limits from observed values Use the distribution over an observation window instead of guesses. With Prometheus, the p95 of the working set over the last 7 days is a starting point for requests, and the maximum over the same window plus headroom is a starting point for the limit. Choose the headroom from the workload's peak pattern (batch jobs, cache warm-up, etc.), then re-measure after the change. ```promql # working set p95 over the last 7 days (per container) quantile_over_time(0.95, container_memory_working_set_bytes{namespace="", pod=~"-.*", container=""}[7d]) # maximum over the same window max_over_time( container_memory_working_set_bytes{namespace="", pod=~"-.*", container=""}[7d]) ``` Setting requests equal to limits gives Guaranteed QoS, which pushes the Pod to the back of the eviction order under node pressure ([Pod QoS docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/)), but memory is incompressible, so a momentary peak that touches the limit is still OOMKilled. When you use Guaranteed for critical workloads, size from the observed maximum. ### For the JVM, count memory outside the heap JVM process memory is the heap (`-Xmx` or `MaxRAMPercentage`) plus non-heap areas: metaspace, thread stacks, code cache, direct buffers. Set the heap equal to the limit and the non-heap areas push you over it. Keep the heap at a fraction of the limit with `MaxRAMPercentage`, as in the example above, and lower the fraction for applications with large non-heap usage (heavy direct-buffer use, many threads). You can see non-heap usage by starting with `-XX:NativeMemoryTracking=summary` and running `jcmd VM.native_memory summary` ([Oracle NMT docs](https://docs.oracle.com/en/java/javase/21/vm/native-memory-tracking.html)). ### Raise the limit, or fix a leak? Decide from the shape of the working-set graph. ``` working set graph ├─ rises and falls (drops after GC / cache release), only peaks touch the limit → limit is tight → adjust the limit └─ keeps climbing and doesn't come down after GC → suspect a leak → heap dump / profiling ``` With the climbing shape, raising the limit only delays the OOM kill. If a higher limit just stretched the restart interval, suspect a leak first. On the JVM, use `jcmd GC.heap_dump /tmp/heap.hprof` or JFR to find the cause. ### Diagnosis matrix | Symptom | Likely cause | Next action | |---|---|---| | 137 + `OOMKilled`, graph rises and falls with peaks near the limit | Limit is tight | Re-size the limit from the observed maximum | | 137 + `OOMKilled`, graph keeps climbing | Memory leak | Heap dump / profiling, fix the code | | 137 on the JVM with heap headroom | Non-heap overflow | Lower `MaxRAMPercentage`, check non-heap with NMT | | `Evicted`, `node was low on resource: memory` | Node memory pressure | Re-size requests, check node capacity and placement (Recovery 3) | | 137 without `OOMKilled` | External SIGKILL (forced delete, grace period exceeded, etc.) | Check events and the audit log | ## Conclusion: Don't misdiagnose sibling errors Confusing 137 with other exit codes sends you digging in the wrong place. | Code/status | Meaning | Distinguishing keywords → where to go | |---|---|---| | **137** | SIGKILL (OOM or force kill) | `OOMKilled` / `dmesg OOM` → this post | | **143** | SIGTERM (graceful shutdown failed) | `preStop` / termination timeout → graceful shutdown post | | **Evicted** | Node memory or disk pressure | `node was low on resource` → Recovery 3 above | | **CrashLoopBackOff** | Restart loop (can follow repeated OOM) | `Back-off restarting` → previous CrashLoopBackOff post | Repeated OOM eventually transitions into CrashLoopBackOff, so **when you see repeated restarts, check lastState reason first** to tell whether the real cause is OOM. The next post (K8s_Troubleshooting_Guide part 20) covers exit code 143, SIGTERM, and a runbook for guaranteeing graceful shutdown with the `preStop` hook. ## References: Official docs The primary sources for the behavior, settings, and errors in this post are the official docs below. Check them for version-specific options and exact behavior. - [Resource management for containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) - [Node-pressure eviction (working set, memory.available)](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/) - [Pod QoS classes](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/) - [Resize container resources in place](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/) - [Vertical Pod Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler) ## FAQ **Q. I raised the limit and still get OOMKilled. Why?** A. Either a memory leak, or the JVM/Node heap setting didn't follow the new limit. If `kubectl top pod --containers` shows usage still climbing, it's a leak. If it dies immediately after start, check the heap settings (`MaxRAMPercentage`, `--max-old-space-size`). **Q. It's OOMKilled but `kubectl get pod` shows Running.** A. The container may have died and restarted, so it's Running now. Check `Last State` and the `RESTARTS` count in `kubectl describe pod`, plus `lastState.terminated.reason` via jsonpath—that will surface the past OOM. **Q. Exit code 137 but no OOMKilled at all.** A. It's likely an external SIGKILL, not kernel OOM. Suspect `kubectl delete --grace-period=0`, a deploy tool, node shutdown/drain, or a force kill after a liveness probe failure—then check events and the audit log.

#kubectl-troubleshooting

OOMKilled exit code 137 해결: kubectl 30초 진단·5분 복구 런북