OutOfMemoryError 5-Family 30-Second Diagnosis Runbook: heap space vs Metaspace Recovery Commands

OutOfMemoryError 5-Family 30-Second Diagnosis Runbook: heap space vs Metaspace Recovery Commands

It's 3 a.m., an alert fires, and the logs show java.lang.OutOfMemoryError. Plenty of people immediately bump -Xmx—and half the time that's wasted effort. Not every OutOfMemoryError is the same OOM. If it's a Metaspace leak and you only grow the heap, it will come back. If it's a container cgroup problem and you only tweak JVM flags, the process dies again with OOMKilled.

Here's the key: a single line of the original error message decides 90% of the diagnosis. This post is structured so you can scroll and copy-paste during an incident.

30-Second Decision Table: Classify the 5 Families from the Error Text

Start with the phrase attached to the error message. The text that follows OutOfMemoryError determines the family.

Just distinguishing Java heap space from Metaspace completely changes the response. That's 30 seconds.

Copy-Paste Diagnosis Command Set

Once you've identified the error family, pin down the cause. Copy these in order.

1) Live utilization by region — See at a glance which region is filling up.

# Print GC region utilization (%) every 1 second. M column is Metaspace
jstat -gcutil <pid> 1000

Among S0 S1 E O M CCS YGC YGCT FGC FGCT GCT, if O (Old) sits near 100% and never drops, that's a heap leak; if M (Metaspace) keeps climbing, that's a classloader leak.

2) Extract top occupying objects — Top 30 of what's eating the heap.

jmap -histo:live <pid> | head -30

3) Heap snapshot

jcmd <pid> GC.heap_info

4) Capture a heap dump — Take it before you restart. If you lose it, you can't find the cause.

jmap -dump:live,format=b,file=heap.hprof <pid>

Open the resulting heap.hprof in Eclipse MAT and look at the Leak Suspects report and Dominator Tree—the leaking objects show up immediately.

5) Enable GC logging — Verify the growth trend over time. Options differ by JDK version.

# Java 11+ (unified logging)
-Xlog:gc*:file=gc.log:time,uptime,level,tags

# Java 8
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:gc.log

The Real Culprit Behind Metaspace Leaks: Classloaders That Never Die

A Metaspace OOM is almost always a classloader leak, not simple shortage. Unlike the heap, this is where loaded class metadata accumulates.

The most common trap in practice is Spring Boot DevTools. DevTools creates a new RestartClassLoader every time code changes; if the previous loader is held by a reference and never GC'd, classes get loaded twice. Add CGLIB dynamic proxies or repeated hot deploys and the loaded class count explodes.

# Typical leak pattern observed with jstat -gcutil (M column excerpted)
  M      CCS    YGC   FGC
 71.20  68.11    12     0    <- healthy
 84.55  79.03    18     2
 93.87  88.40    25     6
 99.12  95.71    31    14    <- FGC exploding but M doesn't drop = leak confirmed

If things were healthy, M should rise and fall as classes unload. If FGC increases but M never drops, it's a classloader leak. Count loaded classes like this.

jcmd <pid> VM.classloader_stats   # loaded count / bytes per classloader

Practitioner tip: I once left DevTools enabled only in staging and hit a Metaspace OOM during a long uninterrupted test. Always strip the DevTools dependency from production images, and set an explicit cap like -XX:MaxMetaspaceSize=256m so unbounded growth is caught early. Without a cap it keeps eating native memory until the whole container is OOMKilled.

When the JVM Misreads Memory in a Container

A common incident in container deployments is the JVM seeing the host's full memory instead of the cgroup limit.

On JDK 8u191+ and JDK 10+, -XX:+UseContainerSupport is enabled by default. Prefer sizing the heap as a percentage rather than an absolute (-Xmx).

# 75% of the container limit as max heap. The remaining 25% is for Metaspace/threads/direct buffers
-XX:MaxRAMPercentage=75.0

cgroup v2 migrations have brought recognition issues back, so JDK 17/21 LTS is recommended. cgroup v2 support is solid there.

OOMKilled(137) and JVM OOM Are Different Incidents

Confusing the two means you fix the wrong thing.

The key is that container total usage is not just the heap. When heap + Metaspace + thread stacks + Direct buffer + code cache exceeds the limit, the kernel OOMKills the process. That's why setting -Xmx exactly to the limit always dies from off-heap memory. That's why you leave headroom with MaxRAMPercentage=75.

Pre-deploy checklist

• Does -Xmx/MaxRAMPercentage leave headroom (70–75%) vs. the container limit?
• Is -XX:MaxMetaspaceSize set explicitly?
• Is Spring DevTools removed from the production image?
• Is GC logging (-Xlog:gc*) enabled so you keep a trend?
• Is automatic heap dump on OOM (-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump) configured?
• Have you verified memory recognition on JDK 17/21 LTS + cgroup v2?

References: Official Docs

The primary source for the behavior, settings, and errors in this post is the following official documentation. Check version-specific options and exact behavior there.

• Oracle Java SE 21 Official Documentation

Frequently Asked Questions (FAQ)

Q. I raised -Xmx and a few days later I got another heap space OOM. A. It's likely a heap leak, not simple shortage. If the Old generation doesn't drop after Full GC in jstat -gcutil, it's a leak. Dump with jmap -dump:live and find the culprit in MAT's Dominator Tree. Growing the heap only stretches the incident interval; it will come back.

Q. It's a Metaspace OOM—why isn't the classloader getting GC'd? A. A classloader stays alive as a whole if even one class/instance it loaded is still referenced from a GC root. DevTools reloads, cached CGLIB proxies, and instances stuck in ThreadLocals are common causes. Confirm exploding loader counts with jcmd <pid> VM.classloader_stats.

Q. The Pod dies with Exit Code 137 but there's no OOM in the application logs. A. That's container OOMKilled, not JVM OOM. Total usage including off-heap (Metaspace, threads, Direct buffer) exceeded the limit, so don't raise -Xmx—lower MaxRAMPercentage or raise the limit to leave headroom.