Fixing OOMKilled Exit Code 137: A kubectl 30-Second Diagnosis and 5-Minute Recovery Runbook

Fixing OOMKilled Exit Code 137: A kubectl 30-Second Diagnosis and 5-Minute Recovery Runbook

If you landed here after seeing Last State: Terminated, Reason: OOMKilled, Exit Code: 137 in kubectl describe pod output, you're in the right place. We'll skip the theory for now and bring the dead Pod back first.

Exit code 137 is 128 + 9, meaning the process received SIGKILL and was force-killed. The key question is who killed it. There are usually three culprits: the container exceeded its own limit (OOMKilled), the whole node ran out of memory and the Pod was kicked off (Evicted), or a human killed it (SIGKILL). The one thing you must do in 30 seconds is split your 137 into one of these three branches. Get the branch wrong and you'll raise limits while missing node pressure—or you'll poke a perfectly healthy limit.

30-Second Diagnosis Decision Table: Which Branch Is Your 137?

Find your symptoms in the table below, then jump straight to the matching commands in the next section.

The key distinction: OOMKilled is container-scoped; Evicted is Pod-scoped (the node scheduler kicked it out). Even if kubectl get pod doesn't show STATUS as OOMKilled and it looks like a healthy Running, if the restart count is climbing, always check lastState.

Copy-Paste Diagnostic Commands

Once you've identified the branch above, copy and run these as-is. Just replace <pod> with yours.

# A) Last State — Reason Exit Code
kubectl describe pod <pod> | grep -A5 "Last State"

# B) /: reason + exitCode
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
# : OOMKilled / 137

# C) (metrics-server )
kubectl top pod <pod> --containers

# D) OOMKilling
kubectl get events --field-selector reason=OOMKilling -A

# E) OOM killer check
# nsenter/ after:
dmesg -T | grep -i "Out of memory: Killed process"
journalctl -k | grep -i oom

If E's dmesg shows a line like Killed process ... (java), that's airtight evidence that the kernel OOM killer picked that process and killed it. On most 2026 distros where cgroup v2 is the default, OOM is isolated more precisely at the cgroup level, so often only that container dies without affecting sibling containers.

Recovery by Cause + Preventing Recurrence

Recovery 1: Recalculate limits

Take the actual usage from kubectl top plus 30% headroom as the limit. Set requests to typical usage and limits to peak.

# Before — limit when
resources:
  requests: { memory: "256Mi" }
  limits:   { memory: "512Mi" }

# After — top measure 700Mi
resources:
  requests: { memory: "768Mi" }
  limits:   { memory: "1Gi" }

Recovery 2: Runtime heap vs. container limit mismatch (the most common trap)

On JVM and Node.js, if the runtime heap is set equal to or larger than the container limit, you get OOMKilled immediately at startup or right before GC. Metaspace, stacks, and native buffers outside the heap also consume memory.

# JVM — -Xmx limit (limit )
env:
  - name: JAVA_TOOL_OPTIONS
 value: "-XX:MaxRAMPercentage=75.0" # limit 75% ,
resources:
 limits: { memory: "1Gi" } # ~768Mi + 256Mi

# Node.js — old space limit
env:
  - name: NODE_OPTIONS
 value: "--max-old-space-size=768" # limit 1Gi

Setting -Xmx1g with limits.memory: 1Gi (the same value) is the classic instant-death pattern. Always keep heap < limit.

Recovery 3: Eviction from unset limits

Without a limit, the Pod gets BestEffort or Burstable QoS, and when the node runs low on memory these Pods are evicted first. In node autoscaling environments like Karpenter, pressure often appears as the cluster tries to shrink empty nodes, so evictions show up more often. Set requests/limits explicitly to raise priority.

Preventing recurrence: Guaranteed QoS + monitoring

For your most important workloads, set requests.memory == limits.memory to get Guaranteed QoS, which puts you last in the eviction order.

resources:
  requests: { cpu: "500m", memory: "1Gi" }
 limits: { cpu: "500m", memory: "1Gi" } # → Guaranteed

Get recommended values from VPA in recommendation mode (updateMode: "Off") first; in this mode VPA only computes recommendations and does not change Pods (VPA). In-place Pod resize, which changes container resources without a restart, was alpha in 1.27, beta (on by default) in 1.33, and GA in 1.35 (resize container resources).

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: { name: app-vpa }
spec:
  targetRef: { apiVersion: apps/v1, kind: Deployment, name: app }
 updatePolicy: { updateMode: "Off" } #

HPA can scale on memory, but memory doesn't shrink (leaks, caches), so it is less stable than CPU-based scaling. Prefer VPA for the right memory size, and HPA (CPU/custom metrics) for traffic scaling.

Finally, get alerted before you die. Memory metrics are trending toward OpenTelemetry standardization, but the rule itself is the same.

# Prometheus alert: limit 90% when warning
- alert: PodMemoryNearLimit
  expr: container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.9
  for: 5m
  labels: { severity: warning }
  annotations:
 summary: "{{ $labels.pod }} limit 90% — OOMKilled "

Check this first: before blindly raising the limit, confirm the runtime heap setting (MaxRAMPercentage, --max-old-space-size) is comfortably below the limit. A heap pinned equal to the limit is a classic cause you can fix without raising the limit at all.

Decide from Measurements: Working Set, Sizing Requests and Limits, Spotting Leaks

What kubectl top and eviction look at: the working set

kubectl top and the kubelet's memory eviction decisions use the working set: cgroup memory usage minus inactive file cache. A node's memory.available is computed as capacity - workingSet (node-pressure eviction docs). So when sizing limits, container_memory_working_set_bytes matches real behavior better than RSS or total usage (container_memory_usage_bytes). The container OOM kill itself is triggered by the kernel when the cgroup limit (memory.max on cgroup v2) is hit and nothing can be reclaimed.

Start requests and limits from observed values

Use the distribution over an observation window instead of guesses. With Prometheus, the p95 of the working set over the last 7 days is a starting point for requests, and the maximum over the same window plus headroom is a starting point for the limit. Choose the headroom from the workload's peak pattern (batch jobs, cache warm-up, etc.), then re-measure after the change.

# working set p95 over the last 7 days (per container)
quantile_over_time(0.95,
  container_memory_working_set_bytes{namespace="<ns>", pod=~"<deploy>-.*", container="<c>"}[7d])

# maximum over the same window
max_over_time(
  container_memory_working_set_bytes{namespace="<ns>", pod=~"<deploy>-.*", container="<c>"}[7d])

Setting requests equal to limits gives Guaranteed QoS, which pushes the Pod to the back of the eviction order under node pressure (Pod QoS docs), but memory is incompressible, so a momentary peak that touches the limit is still OOMKilled. When you use Guaranteed for critical workloads, size from the observed maximum.

For the JVM, count memory outside the heap

JVM process memory is the heap (-Xmx or MaxRAMPercentage) plus non-heap areas: metaspace, thread stacks, code cache, direct buffers. Set the heap equal to the limit and the non-heap areas push you over it. Keep the heap at a fraction of the limit with MaxRAMPercentage, as in the example above, and lower the fraction for applications with large non-heap usage (heavy direct-buffer use, many threads). You can see non-heap usage by starting with -XX:NativeMemoryTracking=summary and running jcmd <pid> VM.native_memory summary (Oracle NMT docs).

Raise the limit, or fix a leak?

Decide from the shape of the working-set graph.

working set graph
  ├─ rises and falls (drops after GC / cache release), only peaks touch the limit → limit is tight → adjust the limit
  └─ keeps climbing and doesn't come down after GC → suspect a leak → heap dump / profiling

With the climbing shape, raising the limit only delays the OOM kill. If a higher limit just stretched the restart interval, suspect a leak first. On the JVM, use jcmd <pid> GC.heap_dump /tmp/heap.hprof or JFR to find the cause.

Diagnosis matrix

Conclusion: Don't misdiagnose sibling errors

Confusing 137 with other exit codes sends you digging in the wrong place.

Repeated OOM eventually transitions into CrashLoopBackOff, so when you see repeated restarts, check lastState reason first to tell whether the real cause is OOM.

The next post (K8s_Troubleshooting_Guide part 20) covers exit code 143, SIGTERM, and a runbook for guaranteeing graceful shutdown with the preStop hook.

References: Official docs

The primary sources for the behavior, settings, and errors in this post are the official docs below. Check them for version-specific options and exact behavior.

• Resource management for containers
• Node-pressure eviction (working set, memory.available)
• Pod QoS classes
• Resize container resources in place
• Vertical Pod Autoscaler

FAQ

Q. I raised the limit and still get OOMKilled. Why? A. Either a memory leak, or the JVM/Node heap setting didn't follow the new limit. If kubectl top pod --containers shows usage still climbing, it's a leak. If it dies immediately after start, check the heap settings (MaxRAMPercentage, --max-old-space-size).

Q. It's OOMKilled but kubectl get pod shows Running. A. The container may have died and restarted, so it's Running now. Check Last State and the RESTARTS count in kubectl describe pod, plus lastState.terminated.reason via jsonpath—that will surface the past OOM.

Q. Exit code 137 but no OOMKilled at all. A. It's likely an external SIGKILL, not kernel OOM. Suspect kubectl delete --grace-period=0, a deploy tool, node shutdown/drain, or a force kill after a liveness probe failure—then check events and the audit log.