OutOfMemoryError 5-Family 30-Second Diagnosis Runbook: heap space vs Metaspace Recovery Commands

OutOfMemoryError 5-Family 30-Second Diagnosis Runbook: heap space vs Metaspace Recovery Commands

It's 3 a.m., an alert fires, and the logs show java.lang.OutOfMemoryError. Plenty of people immediately bump -Xmx—and depending on the family, that's wasted effort. Not every OutOfMemoryError is the same OOM. If it's a Metaspace leak and you only grow the heap, it will come back. If it's a container cgroup problem and you only tweak JVM flags, the process dies again with OOMKilled.

Here's the key: a single line of the original error message decides the direction of the diagnosis. This post is structured so you can scroll and copy-paste during an incident.

30-Second Decision Table: Classify the 5 Families from the Error Text

Start with the phrase attached to the error message. The text that follows OutOfMemoryError determines the family.

Just distinguishing Java heap space from Metaspace completely changes the response. That's 30 seconds.

Copy-Paste Diagnosis Command Set

Once you've identified the error family, pin down the cause. Copy these in order.

1) Live utilization by region — See at a glance which region is filling up.

# Print GC region utilization (%) every 1 second. M is usage relative to committed Metaspace
jstat -gcutil <pid> 1000

Among S0 S1 E O M CCS YGC YGCT FGC FGCT GCT, if O (Old) sits near 100% and never drops, that's a heap leak; if Metaspace usage (MU in jstat -gc below) keeps growing, that's a classloader leak.

2) Extract top occupying objects — Top 30 of what's eating the heap.

jmap -histo:live <pid> | head -30

3) Heap snapshot

jcmd <pid> GC.heap_info

4) Capture a heap dump — Take it before you restart. If you lose it, you can't find the cause.

jmap -dump:live,format=b,file=heap.hprof <pid>

While a heap dump is being written the JVM is paused (STW), and the :live option runs a Full GC first. The dump file is as large as the heap in use, so check free disk space, and it's safer to take it on an instance drained of traffic. jcmd <pid> GC.heap_dump /tmp/heap.hprof does the same job.

In MAT, go Leak Suspects → Dominator Tree (objects retaining the most) → Path to GC Roots of the suspect object (excluding weak references) to see exactly who is holding that object.

5) Enable GC logging — Verify the growth trend over time. Options differ by JDK version.

# Java 11+ (unified logging)
-Xlog:gc*:file=gc.log:time,uptime,level,tags

# Java 8
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:gc.log

6) Capture the OOM moment automatically — turn these on in advance so you don't miss the dump during a 3 a.m. incident.

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump    # write a heap dump automatically on OOM
-XX:+ExitOnOutOfMemoryError                                # exit immediately on OOM and let the restart policy take over (JDK 8u92+)

ExitOnOutOfMemoryError stops the JVM from serving requests in a half-broken state after an OOM. In containers, point HeapDumpPath at a persistent volume so the dump survives the restart. To reduce GC churn from heap resizing, you can also set -Xms equal to -Xmx (or InitialRAMPercentage equal to MaxRAMPercentage).

The Real Culprit Behind Metaspace Leaks: Classloaders That Never Die

A Metaspace OOM is usually a classloader leak, not simple shortage. Unlike the heap, this is where loaded class metadata accumulates.

The most common trap in practice is Spring Boot DevTools. DevTools creates a new RestartClassLoader every time code changes; if the previous loader is held by a reference and never GC'd, classes get loaded twice. Add CGLIB dynamic proxies or repeated hot deploys and the loaded class count explodes.

The M in jstat -gcutil is utilization relative to the currently committed Metaspace, not to MaxMetaspaceSize, so a healthy JVM often shows it in the 90s. Judge a leak by the trend of absolute usage (MU), not by the percentage.

# MC (committed) and MU (used) are in KB. If MU keeps growing even after Full GC, suspect a leak
jstat -gc <pid> 5000
# Detailed Metaspace usage, commit, and limits (JDK 11+)
jcmd <pid> VM.metaspace

# Typical leak pattern observed with jstat -gc (some columns excerpted, KB)
     MC        MU      FGC
  131072    120310       2
  196608    183551       6
  262144    255208      14    <- Full GCs increase but MU never drops and keeps growing = suspect a leak

If things are healthy, MU levels off or drops as classes unload. If MU keeps growing as Full GCs increase, it's a classloader leak. Count loaded classes like this.

jcmd <pid> VM.classloader_stats   # loaded count / bytes per classloader

Practitioner tip: I once left DevTools enabled only in staging and hit a Metaspace OOM during a long uninterrupted test. Always strip the DevTools dependency from production images, and set an explicit cap like -XX:MaxMetaspaceSize=256m so unbounded growth is caught early. Without a cap it keeps eating native memory until the whole container is OOMKilled.

When the JVM Misreads Memory in a Container

A common incident in container deployments is the JVM seeing the host's full memory instead of the cgroup limit.

On JDK 8u191+ and JDK 10+, -XX:+UseContainerSupport is enabled by default. Prefer sizing the heap as a percentage rather than an absolute (-Xmx).

# 75% of the container limit as max heap. The remaining 25% is for Metaspace/threads/direct buffers
-XX:MaxRAMPercentage=75.0

cgroup v2 migrations have brought recognition issues back, so JDK 17/21 LTS is recommended. cgroup v2 support is solid there.

OOMKilled(137) and JVM OOM Are Different Incidents

Confusing the two means you fix the wrong thing.

The key is that container total usage is not just the heap. When heap + Metaspace + thread stacks + Direct buffer + code cache exceeds the limit, the kernel OOMKills the process. That's why setting -Xmx exactly to the limit always dies from off-heap memory. That's why you leave headroom with MaxRAMPercentage=75.

Pre-deploy checklist

• Does -Xmx/MaxRAMPercentage leave headroom (70–75%) vs. the container limit?
• Is -XX:MaxMetaspaceSize set explicitly?
• Is Spring DevTools removed from the production image?
• Is GC logging (-Xlog:gc*) enabled so you keep a trend?
• Is automatic heap dump on OOM (-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump) configured?
• Have you verified memory recognition on JDK 17/21 LTS + cgroup v2?

References: Official Docs

The primary source for the behavior, settings, and errors in this post is the following official documentation. Check version-specific options and exact behavior there.

• Oracle Java SE 21 Official Documentation

Frequently Asked Questions (FAQ)

Q. I raised -Xmx and a few days later I got another heap space OOM. A. It's likely a heap leak, not simple shortage. If the Old generation doesn't drop after Full GC in jstat -gcutil, it's a leak. Dump with jmap -dump:live and find the culprit in MAT's Dominator Tree. Growing the heap only stretches the incident interval; it will come back.

Q. It's a Metaspace OOM—why isn't the classloader getting GC'd? A. A classloader stays alive as a whole if even one class/instance it loaded is still referenced from a GC root. DevTools reloads, cached CGLIB proxies, and instances stuck in ThreadLocals are common causes. Confirm exploding loader counts with jcmd <pid> VM.classloader_stats.

Q. The Pod dies with Exit Code 137 but there's no OOM in the application logs. A. That's container OOMKilled, not JVM OOM. Total usage including off-heap (Metaspace, threads, Direct buffer) exceeded the limit, so don't raise -Xmx—lower MaxRAMPercentage or raise the limit to leave headroom.