Fixing Too Many Open Files in Practice: ulimit, limits.conf, and systemd LimitNOFILE

A Complete Fix for Too Many Open Files: Practical ulimit, limits.conf, and systemd LimitNOFILE

It's 3 a.m. and the service stopped accepting connections

You didn't touch the deploy, but health checks suddenly fail and the logs fill with Too many open files. A server that looked healthy stops accepting new connections. The real cause is file descriptor (FD) exhaustion. Linux treats sockets, files, and pipes as FDs; the moment a process exceeds its allowed FD count (nofile), new open()/accept() calls are rejected with EMFILE (errno 24).

This post is a single playbook that takes you from identifying the error → diagnosing current limits → separating real load from leaks → copy-paste settings at session, permanent, container, and kernel levels → preventing recurrence. If you're in a hurry, jump straight to the diagnostic commands.

1. Identify the symptom precisely (four stack-specific error logs)

If you've seen any of the messages below, you're looking at FD exhaustion. errno 24 = EMFILE is the common signal.

# Node.js
Error: EMFILE: too many open files, open '/app/uploads/x.tmp'

# Java
java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

# Nginx
2026/06/13 03:11:02 [crit] 12345#0: accept4() failed (24: Too many open files)

# MySQL
[ERROR] Out of resources when opening file './db/t.MYD' (errno: 24 - Too many open files)

Identification tip: If you see errno 24, EMFILE, or Too many open files, this is an FD limit problem—not disk space or permissions. For Nginx, accept() failed alongside it means the process cannot accept new connections at all.

2. Diagnose current limits: how many can you open, and how many are in use

Start by comparing the limit against actual usage.

# 현재 셸의 soft / hard limit
ulimit -n      # soft limit
ulimit -Hn     # hard limit

# 실제 동작 중인 프로세스의 진짜 한계 (가장 신뢰)
cat /proc/$(pgrep -f java)/limits | grep "open files"

# 실사용 FD 개수 — ls /proc 방식이 가장 정확하고 빠름
ls /proc/<pid>/fd | wc -l
lsof -p <pid> | wc -l

If ls /proc/<pid>/fd | wc -l is sitting right up against the limit, you've found the culprit. Dig one step further and count what's consuming FDs by type.

lsof -p <pid> | awk '{print $5}' | sort | uniq -c | sort -rn
# 예) 48000 IPv4  ← 소켓이 대부분이면 커넥션/소켓 누수 의심
#       1200 REG   ← 일반 파일이 비정상적으로 많으면 파일 미close

3. Separate the cause: real load vs. FD leak

Before you raise the limit, split the cause. If it's a leak, raising the limit only buys time until it blows up again.

If you suspect a leak, check for accumulating CLOSE_WAIT sockets. A steadily growing count is strong evidence the application is not closing sockets.

ss -tan state close-wait | wc -l

At the code level, check the following:

• Java: not using try-with-resources, missing close in finally, not returning JDBC/HTTP connections to the pool
• Node: not handling stream close events, keep-alive agent sockets never released
• Common: HTTP keep-alive grown too large, oversized connection pools between microservices

War story: A payment-integration service kept hitting Too many open files. The cause was creating a new HttpClient on every outbound API call and never closing it. Raising ulimit to 65535 put out the fire, but it came back two days later. Only after we reused the client as a singleton did the FD graph flatten. Raising the limit is first aid; fixing the leak is the cure.

4. Copy-paste settings, step by step

(1) Session-only (for testing / immediate check)

ulimit -n 65535   # 현재 셸과 자식 프로세스에만 적용, 재로그인 시 사라짐

(2) Permanent — /etc/security/limits.conf

# /etc/security/limits.conf
*       soft    nofile  65535
*       hard    nofile  65535
root    soft    nofile  65535
root    hard    nofile  65535

Note: * does not apply to root, so add a separate root line. Also, the login modules under /etc/pam.d/ must include session required pam_limits.so. You need to re-login or restart the service for this to take effect.

(3) systemd services

limits.conf does not apply to daemons started by systemd. You have to set it in the unit file itself.

# /etc/systemd/system/myapp.service
[Service]
LimitNOFILE=65535

systemctl daemon-reload
systemctl restart myapp

Default systemd LimitNOFILE values differ by distro (small on older ones, larger on recent ones), so always set it explicitly.

(4) Docker / Compose

What applies is the container runtime setting, not the host ulimit.

docker run --ulimit nofile=65535:65535 myimage

# docker-compose.yml
services:
  app:
    ulimits:
      nofile:
        soft: 65535
        hard: 65535

On Kubernetes, it follows the node runtime (containerd/CRI-O) or the Pod's securityContext/runtime settings. Raising the host limit may have no effect inside the container—verify with cat /proc/1/limits from inside the container.

(5) Kernel-wide limit — fs.file-max

No matter how high you raise the per-process limit, you cannot exceed the system-wide cap.

# /etc/sysctl.conf
fs.file-max = 2097152

sysctl -p
cat /proc/sys/fs/file-nr   # 현재 사용/할당/최대 확인

5. Verify after applying — the most common trap

Editing limits.conf without restarting the service will never take effect. Even if ulimit -n looks different in your shell, already-running processes still hold the old limit. Always check against the live process.

# 서비스 재시작 후, 동작 중인 프로세스의 진짜 한계 확인
cat /proc/$(pgrep -f myapp)/limits | grep "open files"
# Max open files   65535   65535   files   ← 이렇게 나와야 성공

Recurrence-prevention checklist

• Confirm FD exhaustion via errno 24 in the error logs
• Monitor live FD usage with ls /proc/<pid>/fd | wc -l (an upward-sloping graph means a leak)
• Alert on ss -tan state close-wait | wc -l (notify when it exceeds a threshold)
• Alert at 80% of the limit (Prometheus process_open_fds / process_max_fds)
• Code: guarantee close of every socket, file, and connection pool (try-with-resources/finally)
• After changing settings, verify the live process with cat /proc/<pid>/limits

FAQ

Q. I set limits.conf to 65535 and still get Too many open files. A. Almost always you didn't restart the service, or it's a systemd-managed daemon so limits.conf is ignored. Put LimitNOFILE=65535 in the systemd unit, daemon-reload, restart, then confirm with cat /proc/<pid>/limits.

Q. I raised ulimit and it still blows up a few days later. Why? A. It's likely an FD leak, not load. If ls /proc/<pid>/fd | wc -l keeps climbing over time or CLOSE_WAIT sockets pile up, the application is not closing sockets/files. You have to find the missing close in the code for a real fix.

Q. ulimit is not applied inside a Docker container. A. Containers follow the runtime setting, not the host ulimit. Set docker run --ulimit nofile=65535:65535 or compose ulimits:, then check from inside the container with cat /proc/1/limits. On Kubernetes, inspect the node runtime settings.