Solving OOMKilled Exit Code 137: From Diagnosing Pod Memory Limits to Tuning

Completely Solving OOMKilled and Exit Code 137: From Diagnosing Pod Memory Limits to Tuning

K8s_Troubleshooting_Guide, Part 6

"It Restarted Again?" — The Moment You Hit Exit Code 137

An alert fires in the middle of the night. A Pod's RESTARTS counter has somehow reached double digits. You run kubectl describe and see this:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

This is not an application bug, and it is not a health-check failure. The kernel forcibly killed the container because it exceeded its own memory limit. The point is not to vaguely guess why it died, but to cut the problem off in order: identify it precisely → measure it → tune it → prevent recurrence. This installment is that practical playbook.

5-Minute Diagnosis: Identifying Exit Code 137 vs. Node OOM (Evicted)

exit code 137 = 128 + 9 (SIGKILL)

On Linux, a process terminated by a signal gets exit code 128 + signal number. Signal 9 is SIGKILL, so 128 + 9 = 137. When a container exceeds its cgroup memory.limit (memory.max on cgroup v2), the kernel OOM Killer immediately terminates the process in that cgroup with SIGKILL. There is no graceful shutdown. That is why you see 137.

Copy-Paste Command Set for Identification

# 1) Last State in one line — extract reason and exitCode only
kubectl get pod <pod> -n <ns> \
  -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}{"  "}{.status.containerStatuses[*].lastState.terminated.exitCode}{"\n"}'

# 2) Full status in human-readable form
kubectl describe pod <pod> -n <ns> | grep -A5 "Last State"

# 3) Confirm kernel OOM Killer events directly
kubectl get events -n <ns> --field-selector reason=OOMKilling

# 4) Restarts and status at a glance
kubectl get pod <pod> -n <ns> -o wide

If command 1 prints OOMKilled 137, it is confirmed.

Node OOM (Evicted) vs. Container OOMKilled

A common source of confusion is the difference from Pod Evicted (node was low on resource) covered in Part 5. Both look like "out of memory," but the trigger and the symptoms are completely different. (For a deep dive on Evicted itself, see Part 5.)

In short: "If only your container died and the exit code is 137, it is OOMKilled. If the Pod vanished as Evicted, it is a node problem."

Finding the Real Cause: Measuring Actual Memory Usage

Before you blindly raise the limit, look at how much memory is actually being used.

# metrics-server must be installed for this to work
kubectl top pod <pod> -n <ns> --containers

For more precision, look at cAdvisor metrics in Prometheus. The key metric is container_memory_working_set_bytes.

container_memory_working_set_bytes{pod="<pod>", container="<container>"}

working set ≠ RSS: working set is RSS minus reclaimable page cache—the memory the kernel considers "actually needed." This working set is exactly what the OOM Killer uses, so you must look at this metric when tuning limits.

To capture peaks, take the p95.

quantile_over_time(0.95,
  container_memory_working_set_bytes{pod=~"<deploy>-.*", container="<c>"}[7d])

The requests/limits Sizing Formula and YAML

The sizing rule is simple.

• requests = working set p95 (the scheduler guarantees placement)
• limit = requests × (1 + buffer). Buffer is 25–50% depending on workload variability

# ❌ Bad example: requests and limits are equal, guessed without measurement
resources:
  requests: { memory: "512Mi" }
  limits:   { memory: "512Mi" }   # dies instantly on a slight spike

# ✅ Recommended: p95-based requests + limit with headroom
resources:
  requests: { memory: "640Mi" }   # working set p95
  limits:   { memory: "896Mi" }   # p95 × 1.4

Setting requests and limits equal has the benefit of raising QoS to Guaranteed, but memory is an incompressible resource, so a brief spike will OOMKill you immediately. A little headroom is safer.

The JVM Trap: -Xmx and the Limit Must Not Be Equal

The most common production incident involves the JVM. People set -Xmx2g and match the container limit to 2Gi, then get OOMKilled—constantly. JVM RSS = heap (-Xmx) + metaspace + thread stacks + code cache + off-heap (Direct Buffers). Filling the heap alone is enough for the rest to blow past the limit.

# ❌ heap and limit are the same → non-heap regions exceed the limit
env:
  - { name: JAVA_OPTS, value: "-Xmx2g" }
resources:
  limits: { memory: "2Gi" }

# ✅ heap at 70–75% of the limit
env:
  - { name: JAVA_OPTS, value: "-XX:MaxRAMPercentage=75.0" }
resources:
  limits: { memory: "2Gi" }   # heap ≈ 1.5Gi, remaining 0.5Gi is off-heap headroom

Using the JDK 10+ container-aware option -XX:MaxRAMPercentage means the heap scales automatically when the limit changes, which is easier to maintain.

Raising the Limit vs. Fixing a Leak: Decision Flow

The fork in the decision is the shape of the memory graph.

Look at the working set graph
  ├─ Sawtooth (rise → drop on GC, repeating) → normal; the limit is tight → raise the limit
  └─ Monotonic increase (does not drop after GC) → memory leak → fix the code

The key point: a monotonically increasing graph only postpones the OOMKilled moment if you raise the limit. If the graph trends up and to the right, take a heap dump (jmap, JFR) first—do not just bump the limit number. In production I use this rule of thumb: "If the restart interval scales in proportion to the limit increase, it is 100% a leak." If you go from 4Gi to 8Gi and the time-to-death exactly doubles, that is a debugging target, not a tuning exercise.

Preventing Recurrence: VPA and Prometheus Alerts

If you do not want to keep tuning by hand, automation is the answer.

VPA recommendation mode — it only produces recommended values based on actual usage (no auto-apply), so you can safely get a right-sized number.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata: { name: my-app-vpa }
spec:
  targetRef: { apiVersion: apps/v1, kind: Deployment, name: my-app }
  updatePolicy: { updateMode: "Off" }   # recommendations only; apply manually

Prometheus alert rule — catch even a single OOMKilled immediately.

- alert: PodOOMKilled
  expr: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} > 0
  for: 1m
  labels: { severity: warning }
  annotations:
    summary: "{{ $labels.pod }} was OOMKilled"

It is also worth tracking current trends. With KEP-based in-place Pod resize graduating, you can adjust a memory limit with no restart and no downtime, and VPA is evolving toward in-place updates as well. On cgroup v2, OOM behavior is more predictable because it is based on memory.max, and container memory observability via OpenTelemetry/Prometheus has become the standard.

Diagnosis Matrix: Symptom → Cause → Action

FAQ

Q. I raised the limit and it still dies. A. Almost certainly a memory leak. If the working set graph is monotonically increasing and does not come down after GC, raising the limit is just stalling. Analyze a heap dump and find the cause.

Q. What if I have requests only and no limit? A. You will not get a container-level OOMKilled, but if the node runs out of memory, kubelet will Evict the Pod. QoS is also Burstable, so it can be among the first to be evicted.

Q. Exit code is 137 but I do not see a Reason. A. It is being confused with another SIGKILL case (node graceful shutdown, a manual kill). Cross-check kernel OOM with kubectl get events --field-selector reason=OOMKilling and dmesg | grep -i oom on the node.

Q. How do QoS classes relate to OOM priority? A. Under node memory pressure, kernel OOM priority kills in this order: BestEffort → Burstable → Guaranteed. A Guaranteed Pod (requests=limits) dies last, so put critical workloads in Guaranteed to protect them from node OOM.

Q. What changes on cgroup v2? A. The limit is managed as memory.max, and reclaim pressure hits memory.high (a soft limit) first, so OOM behavior is more predictable. Once you understand the mechanism, the same diagnostic command set still applies.

In the next installment we cover branching through the many causes of CrashLoopBackOff and debugging init containers.