fork: retry: Resource temporarily unavailable — 30-Second Diagnostic Runbook

fork: retry: Resource temporarily unavailable — 30-Second Diagnostic Runbook

If you landed here by copy-pasting one of the three messages below from a terminal or a log, here's the punchline first. It's almost always one of two things — nproc (process/thread creation) limit exhaustion, or memory shortage.

• bash: fork: retry: Resource temporarily unavailable (shell/script)
• pthread_create failed (EAGAIN) (C/native)
• java.lang.OutOfMemoryError: unable to create new native thread (JVM)

The names and wording differ, but all three are the kernel returning EAGAIN (= Resource temporarily unavailable) when it tries to create a new execution context. In other words, different messages, same root. Run the commands below in order and you'll know in 30 seconds whether it's a limit or memory.

30-second first branch: hit the nproc limit first 🟢

Explanations later. Diagnose first (all read-only, safe):

# 🟢 Current shell's soft / hard nproc limit
ulimit -u
ulimit -Hu

# 🟢 Total "threads (LWPs)" currently used by a given user
ps -eLf | grep '^appuser ' | wc -l
#   or total LWP count
ps -eLf --no-headers | wc -l

# 🟢 Thread count of one problem process
ps -o nlwp= -p <pid>
cat /proc/<pid>/status | grep Threads

# 🟢 System-wide ceiling (separate from per-user limits)
cat /proc/sys/kernel/threads-max
cat /proc/sys/kernel/pid_max

The verdict rule is simple. If the value from ulimit -u and ps -eLf | grep <user> | wc -l are nearly equal (e.g. limit 4096, usage 4090) → nproc limit exhaustion confirmed. The key point: Linux RLIMIT_NPROC is not really "number of processes" — it counts the per-user total of threads (LWPs). So when a multithreaded runtime (JVM, Go, Node workers) explodes thread count, you can hit the limit even with only a handful of processes.

Second branch: the limit is plenty but it still blows up → memory shortage 🟡

ulimit -u is in the tens of thousands and fork still fails? Then the second suspect is memory.

# 🟡 Available memory / swap — suspect if available is on the floor
free -h

# 🟡 Check for traces that the OOM Killer visited
dmesg -T | grep -i -E 'oom|out of memory|killed process'

# 🟡 Check stack size (virtual memory consumed per thread)
ulimit -s
cat /proc/sys/vm/max_map_count
cat /proc/sys/vm/overcommit_memory

pthread_create reserves a stack per thread (default ulimit -s, typically 8MB) as virtual memory. 8MB × thousands of threads = tens of GB of virtual memory, and that demand hitting the overcommit policy or max_map_count is what produces EAGAIN. On Java, thread stacks are set with -Xss, and that memory comes from the native region outside the heap (-Xmx). So unable to create new native thread can fire even when the heap is fine — bumping -Xmx can actually make it worse by shrinking native headroom.

Field tip (from experience): When this error hits a JVM, a lot of people reflexively raise -Xmx first — that's almost always the wrong move. Count threads first (jstack <pid> | grep -c 'java.lang.Thread.State'), check whether a connection pool or thread pool is leaking, and if the count is actually healthy, drop -Xss to 512k to cut per-thread memory. That's been the faster emergency fix.

Label: nproc is fine + tight memory / OOM traces = native memory / stack exhaustion.

Three errors, one cause — mapping

Permanent fix: the limits.conf vs systemd TasksMax trap 🔴

Once you've pinned the cause, raise the limit. But where it fired completely changes which file you edit. Confirm the scope before applying anything.

(a) The most common trap: limits.conf does not apply to systemd services

# /etc/security/limits.conf or limits.d/*.conf  🔴 (re-login required)
appuser  soft  nproc  65536
appuser  hard  nproc  65536

This applies only to PAM login sessions (SSH, su, etc.). A service started with systemctl start never goes through PAM, so this setting is ignored. 99% of "I already fixed limits.conf — why doesn't it work?" is this case.

# 🟢 Check the values actually applied to the service — the truth is here
systemctl show myapp.service -p TasksMax -p LimitNPROC

DefaultTasksMax is typically set to 15% of the system's nproc, so it often becomes the thread ceiling on RHEL 8/9 and Ubuntu 22.04+.

# 🔴 /etc/systemd/system/myapp.service.d/override.conf  (drop-in recommended)
[Service]
TasksMax=infinity
LimitNPROC=65536

# 🔴 Apply (daemon-reload + restart required; rollback: delete the drop-in then same procedure)
sudo systemctl daemon-reload
sudo systemctl restart myapp.service
systemctl show myapp.service -p TasksMax -p LimitNPROC  # verify

(b) Container difference box 📦

Host ulimit -u is plenty, but fork fails only inside the pod/container? The culprit isn't ulimit — it's cgroup pids.max.

# cgroup v2 (default on recent distros)  🟢
cat /sys/fs/cgroup/<slice-path>/pids.max
# cgroup v1  🟢
cat /sys/fs/cgroup/pids/<path>/pids.max

• Docker: docker run --pids-limit=4096 ... (default is unlimited, but you still hit the cgroup ceiling)
• Kubernetes: kubelet --pod-max-pids or node config for per-pod PID limits

Wrap-up: finish it with one decision tree

fork/pthread/native thread error
        │
ulimit -u ≈ ps -eLf usage?
   ├─ YES → nproc limit exhaustion → raise limits.conf (login) / TasksMax (systemd) / pids.max (container)
   └─ NO  → OOM in free·dmesg?
              ├─ YES → memory shortage → check thread leak / shrink -Xss·ulimit -s / add memory
              └─ NO  → check global ceilings (threads-max, pid_max)

This error is just one sibling in the resource-exhaustion cluster. Only the exhausted resource changes; the diagnostic method is the same — Too many open files (file descriptors/EMFILE), No space left on device (inodes/disk), OutOfMemoryError: Java heap space (heap), connection refused (backlog/ports). Swap in "what ran out" and run the same runbook.

FAQ

Q. I raised nproc in limits.conf but the service still fails to fork. Why? A. Services started by systemd never go through PAM, so limits.conf does not apply. Check the actual values with systemctl show <svc> -p TasksMax -p LimitNPROC, put TasksMax= and LimitNPROC= in a unit drop-in, then daemon-reload + restart.

Q. Host ulimit is plenty, but it only blows up inside the container. A. The real ceiling for a container is not the host ulimit — it's cgroup pids.max. Check /sys/fs/cgroup/.../pids.max, then adjust Docker --pids-limit or the kubelet PID limit on Kubernetes.

Q. I raised -Xmx on Java and the native thread error happens more often. A. That's expected. Thread stacks come from native memory outside the heap, so growing the heap shrinks native headroom. If the thread count is abnormal, fix the leak; if it's a healthy load, lower -Xss to cut per-thread memory.