# OutOfMemoryError 5-Family 30-Second Diagnosis Runbook: heap space vs Metaspace Recovery Commands It's 3 a.m., an alert fires, and the logs show `java.lang.OutOfMemoryError`. Plenty of people immediately bump `-Xmx`—and depending on the family, that's wasted effort. **Not every OutOfMemoryError is the same OOM.** If it's a Metaspace leak and you only grow the heap, it will come back. If it's a container cgroup problem and you only tweak JVM flags, the process dies again with OOMKilled. Here's the key: **a single line of the original error message decides the direction of the diagnosis.** This post is structured so you can scroll and copy-paste during an incident. ## 30-Second Decision Table: Classify the 5 Families from the Error Text Start with the phrase attached to the error message. The text that follows `OutOfMemoryError` determines the family. | Error text | Region | Typical cause | 30-second first action | Next-step command | |---|---|---|---|---| | `Java heap space` | Heap (Eden/Old) | Object leak or simple heap shortage | Capture a heap dump, then restart | `jmap -dump:live` + MAT analysis | | `Metaspace` | Metaspace | Classloader leak; explosion of dynamic proxies/hot deploys | Check loaded class count | Trend of MU (used) in `jstat -gc`, `jcmd VM.metaspace` | | `GC overhead limit exceeded` | Heap (GC time) | Heap nearly full; GC reclaiming only tiny amounts | Effectively heap shortage/leak | Confirm FGC explosion in `jstat -gcutil` | | `unable to create new native thread` | Native (thread stacks) | Thread leak; excessive ulimit/stack size | Count threads | `jstack ` thread dump | | `Direct buffer memory` | Off-heap direct memory | Unreleased NIO/Netty ByteBuffers | Check MaxDirectMemorySize | `jcmd VM.native_memory summary` (only for a JVM started with `-XX:NativeMemoryTracking=summary`) | Just distinguishing `Java heap space` from `Metaspace` completely changes the response. That's 30 seconds. ## Copy-Paste Diagnosis Command Set Once you've identified the error family, pin down the cause. Copy these in order. **1) Live utilization by region** — See at a glance which region is filling up. ```bash # Print GC region utilization (%) every 1 second. M is usage relative to committed Metaspace jstat -gcutil 1000 ``` Among `S0 S1 E O M CCS YGC YGCT FGC FGCT GCT`, **if O (Old) sits near 100% and never drops, that's a heap leak**; **if Metaspace usage (MU in `jstat -gc` below) keeps growing, that's a classloader leak**. **2) Extract top occupying objects** — Top 30 of what's eating the heap. ```bash jmap -histo:live | head -30 ``` **3) Heap snapshot** ```bash jcmd GC.heap_info ``` **4) Capture a heap dump** — Take it before you restart. If you lose it, you can't find the cause. ```bash jmap -dump:live,format=b,file=heap.hprof ``` While a heap dump is being written the JVM is paused (STW), and the `:live` option runs a Full GC first. The dump file is as large as the heap in use, so check free disk space, and it's safer to take it on an instance drained of traffic. `jcmd GC.heap_dump /tmp/heap.hprof` does the same job. In MAT, go **Leak Suspects → Dominator Tree (objects retaining the most) → Path to GC Roots of the suspect object (excluding weak references)** to see exactly who is holding that object. **5) Enable GC logging** — Verify the growth trend over time. Options differ by JDK version. ```bash # Java 11+ (unified logging) -Xlog:gc*:file=gc.log:time,uptime,level,tags # Java 8 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:gc.log ``` **6) Capture the OOM moment automatically** — turn these on in advance so you don't miss the dump during a 3 a.m. incident. ```bash -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump # write a heap dump automatically on OOM -XX:+ExitOnOutOfMemoryError # exit immediately on OOM and let the restart policy take over (JDK 8u92+) ``` `ExitOnOutOfMemoryError` stops the JVM from serving requests in a half-broken state after an OOM. In containers, point `HeapDumpPath` at a persistent volume so the dump survives the restart. To reduce GC churn from heap resizing, you can also set `-Xms` equal to `-Xmx` (or `InitialRAMPercentage` equal to `MaxRAMPercentage`). ## The Real Culprit Behind Metaspace Leaks: Classloaders That Never Die A `Metaspace` OOM is usually **a classloader leak, not simple shortage**. Unlike the heap, this is where loaded class metadata accumulates. The most common trap in practice is **Spring Boot DevTools**. DevTools creates a new `RestartClassLoader` every time code changes; if the previous loader is held by a reference and never GC'd, classes get loaded twice. Add CGLIB dynamic proxies or repeated hot deploys and the loaded class count explodes. The `M` in `jstat -gcutil` is utilization relative to **the currently committed Metaspace**, not to `MaxMetaspaceSize`, so a healthy JVM often shows it in the 90s. Judge a leak by **the trend of absolute usage (MU)**, not by the percentage. ```bash # MC (committed) and MU (used) are in KB. If MU keeps growing even after Full GC, suspect a leak jstat -gc 5000 # Detailed Metaspace usage, commit, and limits (JDK 11+) jcmd VM.metaspace ``` ``` # Typical leak pattern observed with jstat -gc (some columns excerpted, KB) MC MU FGC 131072 120310 2 196608 183551 6 262144 255208 14 <- Full GCs increase but MU never drops and keeps growing = suspect a leak ``` If things are healthy, MU levels off or drops as classes unload. **If MU keeps growing as Full GCs increase**, it's a classloader leak. Count loaded classes like this. ```bash jcmd VM.classloader_stats # loaded count / bytes per classloader ``` > Practitioner tip: I once left DevTools enabled only in staging and hit a Metaspace OOM during a long uninterrupted test. **Always strip the DevTools dependency from production images**, and set an explicit cap like `-XX:MaxMetaspaceSize=256m` so unbounded growth is caught early. Without a cap it keeps eating native memory until the whole container is OOMKilled. ## When the JVM Misreads Memory in a Container A common incident in container deployments is **the JVM seeing the host's full memory instead of the cgroup limit**. | | UseContainerSupport not applied | Applied (ON by default in JDK 10+) | |---|---|---| | Recognized memory | Entire host (e.g. 64Gi) | cgroup limit (512Mi) | | Default heap sizing | Over-allocated from the host | Ratio allocated from the limit | | Result | Heap exceeds limit → OOMKilled | Heap cap decided safely | On JDK 8u191+ and JDK 10+, `-XX:+UseContainerSupport` is enabled by default. Prefer sizing the heap as a percentage rather than an absolute (`-Xmx`). ```bash # 75% of the container limit as max heap. The remaining 25% is for Metaspace/threads/direct buffers -XX:MaxRAMPercentage=75.0 ``` cgroup v2 migrations have brought recognition issues back, so **JDK 17/21 LTS is recommended**. cgroup v2 support is solid there. ## OOMKilled(137) and JVM OOM Are Different Incidents Confusing the two means you fix the wrong thing. | | OOMKilled | JVM OutOfMemoryError | |---|---|---| | Where you find it | `kubectl describe pod` | Application logs | | What you see | `Reason: OOMKilled`, `Exit Code: 137` | `java.lang.OutOfMemoryError` stack trace | | Who killed it | Kernel via [SIGKILL](/blog/npm-err-code-elifecycle-해결법-errno-1134sigkill-원인별-진단)(9) | JVM throwing an exception itself | | Cause | Container total memory > limit | A specific JVM region exceeded its limit | The key is that **container total usage is not just the heap**. When `heap + Metaspace + thread stacks + Direct buffer + code cache` exceeds the limit, the kernel OOMKills the process. That's why setting `-Xmx` exactly to the limit always dies from off-heap memory. That's why you leave headroom with `MaxRAMPercentage=75`. ### Pre-deploy checklist - [ ] Does `-Xmx`/`MaxRAMPercentage` leave headroom (70–75%) vs. the container limit? - [ ] Is `-XX:MaxMetaspaceSize` set explicitly? - [ ] Is Spring DevTools removed from the production image? - [ ] Is GC logging (`-Xlog:gc*`) enabled so you keep a trend? - [ ] Is automatic heap dump on OOM (`-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump`) configured? - [ ] Have you verified memory recognition on JDK 17/21 LTS + cgroup v2? ## References: Official Docs The primary source for the behavior, settings, and errors in this post is the following official documentation. Check version-specific options and exact behavior there. - [Oracle Java SE 21 Official Documentation](https://docs.oracle.com/en/java/javase/21/) ## Frequently Asked Questions (FAQ) **Q. I raised `-Xmx` and a few days later I got another heap space OOM.** A. It's likely a heap leak, not simple shortage. If the Old generation doesn't drop after Full GC in `jstat -gcutil`, it's a leak. Dump with `jmap -dump:live` and find the culprit in MAT's Dominator Tree. Growing the heap only stretches the incident interval; it will come back. **Q. It's a Metaspace OOM—why isn't the classloader getting GC'd?** A. A classloader stays alive as a whole if even one class/instance it loaded is still referenced from a GC root. DevTools reloads, cached CGLIB proxies, and instances stuck in ThreadLocals are common causes. Confirm exploding loader counts with `jcmd VM.classloader_stats`. **Q. The Pod dies with Exit Code 137 but there's no OOM in the application logs.** A. That's container OOMKilled, not JVM OOM. Total usage including off-heap (Metaspace, threads, Direct buffer) exceeded the limit, so don't raise `-Xmx`—lower `MaxRAMPercentage` or raise the limit to leave headroom.

#heap dump 분석

OutOfMemoryError 5계열 30초 진단 런북: heap space vs Metaspace 복구 명령