Too many open files (EMFILE errno 24): 30-second diagnosis and recovery runbook

3 a.m.: "Too many open files" just hit the logs

Skip the theory. Right now the service cannot open sockets, and Too many open files (EMFILE, errno 24) keeps stacking in the logs. This runbook is only error-string matching → status-check commands → copy-paste recovery by layer. Paste from top to bottom.

Remember one thing. This error means file descriptors (FDs) have hit a limit, and the problem is always in one of four layers: session / process (service) / worker / kernel. Identify the layer in 30 seconds and recovery is one or two commands.

Scope: Linux (RHEL/CentOS 7–9, Ubuntu 18.04–24.04), systemd-managed services, Nginx 1.1x, Node.js/Java applications, Docker/containerd and Kubernetes container environments.

30-second diagnosis — error-string matching table

Find the exact string from the logs in the table below and you immediately know which layer is failing.

One-line decision rule:

• If you cannot even run commands in the shell → session layer. Raise ulimit -n and you are done.
• If it only appears in service logs and the shell is fine → process/service layer. Check systemd and nginx config.
• If file-nr is approaching file-max → kernel layer (system-wide). Rare, but fatal.

Current-state check command set

Before recovery, confirm how many FDs are open and what the limit actually is. Trust /proc/PID/limits over ulimit -n. Daemons often start with a different limit than your login shell.

1) How many FDs the process actually has open

# 앱 프로세스가 현재 열고 있는 FD 개수
lsof -p $(pgrep -f myapp) | wc -l

Expected output:

4832

2) The limit actually applied to that process (most trustworthy)

cat /proc/$(pgrep -f myapp)/limits | grep "open files"

Expected output:

Max open files            1024                 4096                 files
#                         ^soft (currently applied)      ^hard (max you can raise to)

In this example, soft is 1024 while 4,832 FDs are open → the limit is clearly exceeded.

3) Session ulimit (for checking the shell layer)

ulimit -Sn   # soft limit
ulimit -Hn   # hard limit

Expected output:

1024
1048576

If soft is a low 1024, any app started from that shell is capped at 1024. If hard is large, raising soft alone fixes it immediately.

4) System-wide (kernel layer)

cat /proc/sys/fs/file-nr
# 출력: 사용중  미사용(할당됐다 반납)  최대치

Expected output:

9856    0    2097152
# allocated  unused  file-max

If the first number exceeds 80% of the third (file-max), the kernel ceiling itself is too low. Most incidents never get this far and stop at the process layer.

Decision summary: if lsof count / soft limit is ≥ 0.8, you are in danger; if it is near 1.0, errors are already firing.

Recovery by layer — copy-paste from top to bottom

Run only the block that matches the failing layer. See the branch note at the end of each step for how high to raise the limit.

① Temporary session raise (right now; gone after reboot)

ulimit -n 65535        # 현재 셸에만 적용
ulimit -n              # 확인

Applies only to processes you restart from this shell. Restart the app here and recovery is immediate, but a reboot reverts it → you must also apply the permanent settings below.

Branch: a regular user cannot raise above the hard limit (ulimit: value exceeds hard limit). In that case raise hard as root, or go to ②.

② Permanent setting — /etc/security/limits.conf

sudo tee -a /etc/security/limits.conf <<'EOF'
*        soft    nofile    65535
*        hard    nofile    65535
root     soft    nofile    65535
root     hard    nofile    65535
EOF

pam_limits must be enabled for this file to take effect. Verify:

grep pam_limits /etc/pam.d/common-session /etc/pam.d/login 2>/dev/null
# 출력에 session required pam_limits.so 가 있어야 함

Caveat: limits.conf applies only to login sessions. It does not apply to daemons started by systemd. That is 80% of "I fixed limits.conf, why didn't it work?" Go to ③ in that case.

③ systemd service — LimitNOFILE (this is the answer for daemons)

For Nginx, application servers, and anything systemd manages, the unit's LimitNOFILE wins. Add it safely with a drop-in:

sudo systemctl edit myapp.service

In the editor that opens, enter:

[Service]
LimitNOFILE=65535

Apply:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

# 검증 — 실제 적용됐는지 반드시 확인
cat /proc/$(pgrep -f myapp)/limits | grep "open files"

Expected healthy result:

Max open files            65535                65535                files

Branch: if it is still 1024 → you skipped daemon-reload, or the drop-in path (/etc/systemd/system/myapp.service.d/override.conf) points at a different unit. Confirm the final value with systemctl show myapp -p LimitNOFILE.

④ Nginx — worker_rlimit_nofile ↔ worker_connections

Nginx needs its own directives in addition to systemd LimitNOFILE. If worker_connections is larger than the FDs it can actually open, you get accept() failed (24).

# /etc/nginx/nginx.conf 최상단(main 컨텍스트)
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;   # worker_rlimit_nofile 이하로
}

Rough formula: FDs needed ≈ worker_connections × 2 (client + upstream) + headroom. So give worker_rlimit_nofile at least 2× worker_connections.

sudo nginx -t          # 문법 검사
sudo systemctl reload nginx
cat /proc/$(pgrep -o nginx)/limits | grep "open files"   # 65535 확인

⑤ Kernel ceiling — fs.file-max (only when the whole system is short)

Only in the extreme case where file-nr is approaching file-max:

# 즉시 적용
sudo sysctl -w fs.file-max=2097152

# 영구화
echo 'fs.file-max = 2097152' | sudo tee /etc/sysctl.d/99-nofile.conf
sudo sysctl --system

Most servers already have a default file-max in the millions, so you almost never reach this step. If you did, you should almost certainly suspect an FD leak. See below.

FD leak vs. limit too low — the distinction that stops recurrence

If you only raise the limit and walk away, it will blow up again in a few days at a higher number. You must tell these two apart.

Watch the trend

# 5초마다 FD 개수 추이 관찰
watch -n5 'ls /proc/$(pgrep -f myapp)/fd | wc -l'

• Keeps climbing from the moment of restart → FD leak. The code is not close()-ing sockets, files, or connections.
• Stays low in normal traffic and only approaches the limit at peak → limit too low. Raising the layer above is enough.

Aggregate FD types when you suspect a leak

lsof -p $(pgrep -f myapp) | awk '{print $5}' | sort | uniq -c | sort -rn

Expected output:

  38210 IPv4     # sockets dominate → missing connection close / too much keep-alive
    412 REG      # regular files
     88 pipe

If IPv4/sock is abnormally high → check HTTP client connection pools not being returned, DB connections not being returned, and keep-alive settings. If REG keeps growing → the code is not closing file handles.

The container / Kubernetes trap

Raising ulimit -n on the host does not change processes inside a container. Docker/containerd default nofile often differs from the host, which causes constant confusion.

# 컨테이너 실행 시 명시
docker run --ulimit nofile=65535:65535 myimage

# 실행 중 컨테이너 내부 실제 한계 확인
docker exec <cid> sh -c 'cat /proc/1/limits | grep "open files"'

On Kubernetes, check both the node's containerd defaults and the Pod's securityContext. If you are asking "the host is 65535, why is the container 1024?", this is why.

Healthy vs. unhealthy criteria

Use this table to decide whether to raise the limit now or look at the code.

Resource-limit troubleshooting — related runbooks

This post is the "FD limit" installment. Most resource-limit incidents follow the same four-step flow: check the limit → temporary raise → permanent raise → leak vs. limit. Other resources use the same pattern:

• fork: Resource temporarily unavailable — process/thread limits (nproc, pids.max)
• PostgreSQL too many clients already — DB connection limits (max_connections, connection pools)

In all three, the key is distinguishing "limit too low" from "leak."

FAQ

Q. I set limits.conf to 65535. Why didn't the service pick it up? A. /etc/security/limits.conf applies only to login sessions (PAM). Daemons started by systemd honor the unit's LimitNOFILE first, so add a drop-in with systemctl edit and run daemon-reload && restart. Confirm with cat /proc/PID/limits.

Q. I raised ulimit -n, but after reboot it went back. A. ulimit is a temporary setting for the current shell only. To persist it, set limits.conf for sessions, systemd LimitNOFILE for daemons, and Nginx worker_rlimit_nofile respectively.

Q. I raised the limit and it blew up again a few days later. A. Likely an FD leak. Watch the trend with watch -n5 'ls /proc/PID/fd | wc -l'. If it climbs from the moment of restart, inspect the code for missing socket/file/connection close(). Raising the limit is only a stopgap.