Fixing Too Many Open Files in Practice: ulimit, limits.conf, and systemd LimitNOFILE

A Complete Fix for Too Many Open Files: Practical ulimit, limits.conf, and systemd LimitNOFILE

It's 3 a.m. and the service stopped accepting connections

You didn't touch the deploy, but health checks suddenly fail and the logs fill with Too many open files. A server that looked healthy stops accepting new connections. The real cause is file descriptor (FD) exhaustion. Linux treats sockets, files, and pipes as FDs; the moment a process exceeds its allowed FD count (nofile), new open()/accept() calls are rejected with EMFILE (errno 24).

This post is a single playbook that takes you from identifying the error → diagnosing current limits → separating real load from leaks → copy-paste settings at session, permanent, container, and kernel levels → preventing recurrence. If you're in a hurry, jump straight to the diagnostic commands.

1. Identify the symptom precisely (four stack-specific error logs)

If you've seen any of the messages below, you're looking at FD exhaustion. errno 24 = EMFILE is the common signal.

# Node.js
Error: EMFILE: too many open files, open '/app/uploads/x.tmp'

# Java
java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

# Nginx
2026/06/13 03:11:02 [crit] 12345#0: accept4() failed (24: Too many open files)

# MySQL
[ERROR] Out of resources when opening file './db/t.MYD' (errno: 24 - Too many open files)

Identification tip: If you see errno 24, EMFILE, or Too many open files, this is an FD limit problem—not disk space or permissions. For Nginx, accept() failed alongside it means the process cannot accept new connections at all.

The log text also tells you which layer hit the limit.

If you can't even run commands in the shell, it's the session layer; if it only shows up in the service log while the shell works fine, it's the service (process) layer.

2. Diagnose current limits: how many can you open, and how many are in use

Start by comparing the limit against actual usage.

# soft / hard limit of the current shell
ulimit -n      # soft limit
ulimit -Hn     # hard limit

# The real limit of the running process (most reliable)
cat /proc/$(pgrep -f java)/limits | grep "open files"

# FDs actually in use — reading /proc is the most accurate and fastest
ls /proc/<pid>/fd | wc -l
lsof -p <pid> | wc -l   # note: also counts the header line and cwd/txt/mem (memory-mapped) entries, so it reads higher than the real FD count

If ls /proc/<pid>/fd | wc -l is sitting right up against the limit, you've found the culprit. Dig one step further and count what's consuming FDs by type.

lsof -p <pid> | awk '{print $5}' | sort | uniq -c | sort -rn
# ) 48000 IPv4 ← socket /socket
# 1200 REG ← file file close

3. Separate the cause: real load vs. FD leak

Before you raise the limit, split the cause. If it's a leak, raising the limit only buys time until it blows up again.

If you suspect a leak, check for accumulating CLOSE_WAIT sockets. A steadily growing count is strong evidence the application is not closing sockets.

ss -tan state close-wait | wc -l

At the code level, check the following:

• Java: not using try-with-resources, missing close in finally, not returning JDBC/HTTP connections to the pool
• Node: not handling stream close events, keep-alive agent sockets never released
• Common: HTTP keep-alive grown too large, oversized connection pools between microservices

War story: A payment-integration service kept hitting Too many open files. The cause was creating a new HttpClient on every outbound API call and never closing it. Raising ulimit to 65535 put out the fire, but it came back two days later. Only after we reused the client as a singleton did the FD graph flatten. Raising the limit is first aid; fixing the leak is the cure.

4. Copy-paste settings, step by step

(1) Session-only (for testing / immediate check)

ulimit -n 65535 # , when

(2) Permanent — /etc/security/limits.conf

# /etc/security/limits.conf
*       soft    nofile  65535
*       hard    nofile  65535
root    soft    nofile  65535
root    hard    nofile  65535

Note: * does not apply to root, so add a separate root line. Also, the login modules under /etc/pam.d/ must include session required pam_limits.so. You need to re-login or restart the service for this to take effect.

(3) systemd services

limits.conf does not apply to daemons started by systemd. You have to set it in the unit file itself.

# /etc/systemd/system/myapp.service
[Service]
LimitNOFILE=65535

systemctl daemon-reload
systemctl restart myapp

Instead of editing the unit file directly, add a drop-in so the setting survives package updates.

sudo systemctl edit myapp.service       # enter [Service] / LimitNOFILE=65535 in the editor
sudo systemctl daemon-reload
sudo systemctl restart myapp.service
systemctl show myapp.service -p LimitNOFILE -p LimitNOFILESoft   # the final values systemd applied

Since systemd 240 the default is soft 1024 and hard 524288 (DefaultLimitNOFILE=1024:524288). Even with a large hard limit, the soft limit is 1024, so unless the program raises its own soft limit, it gets stuck at 1024. That is why setting LimitNOFILE explicitly for services is recommended. If you still see 1024, you forgot daemon-reload or the drop-in went into a different unit.

(3-1) Nginx — worker_rlimit_nofile and worker_connections

Nginx workers can raise their own FD limit with worker_rlimit_nofile. If worker_connections is larger than the FDs a worker can actually open, you get accept4() failed (24: Too many open files).

# Top of /etc/nginx/nginx.conf (main context)
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
}

worker_connections is the per-worker number of simultaneous connections, and it already includes upstream (proxied) connections as well as client connections. Each proxied request uses two connections, client and upstream, so the number of clients you can actually serve is lower. Set worker_rlimit_nofile higher than worker_connections to leave room for other FDs such as log and cache files. nginx docs

sudo nginx -t && sudo systemctl reload nginx
cat /proc/$(pgrep -o nginx)/limits | grep "open files"   # checks the master process

(4) Docker / Compose

What applies is the container runtime setting, not the host ulimit.

docker run --ulimit nofile=65535:65535 myimage

# docker-compose.yml
services:
  app:
    ulimits:
      nofile:
        soft: 65535
        hard: 65535

Kubernetes has no field in the Pod spec for nofile (not in securityContext either); it follows the node's container runtime (containerd/CRI-O) and that service's limit settings. Raising the host limit may have no effect inside the container—verify with cat /proc/1/limits from inside the container.

(5) Kernel-wide limit — fs.file-max

No matter how high you raise the per-process limit, you cannot exceed the system-wide cap.

# /etc/sysctl.conf
fs.file-max = 2097152

sysctl -p
cat /proc/sys/fs/file-nr # // check

5. Verify after applying — the most common trap

Editing limits.conf without restarting the service will never take effect. Even if ulimit -n looks different in your shell, already-running processes still hold the old limit. Always check against the live process.

# when after, check
cat /proc/$(pgrep -f myapp)/limits | grep "open files"
# Max open files 65535 65535 files ←

Recurrence-prevention checklist

• Confirm FD exhaustion via errno 24 in the error logs
• Monitor live FD usage with ls /proc/<pid>/fd | wc -l (an upward-sloping graph means a leak)
• Alert on ss -tan state close-wait | wc -l (notify when it exceeds a threshold)
• Alert at 80% of the limit (Prometheus process_open_fds / process_max_fds)
• Code: guarantee close of every socket, file, and connection pool (try-with-resources/finally)
• After changing settings, verify the live process with cat /proc/<pid>/limits

FAQ

Q. I set limits.conf to 65535 and still get Too many open files. A. Almost always you didn't restart the service, or it's a systemd-managed daemon so limits.conf is ignored. Put LimitNOFILE=65535 in the systemd unit, daemon-reload, restart, then confirm with cat /proc/<pid>/limits.

Q. I raised ulimit and it still blows up a few days later. Why? A. It's likely an FD leak, not load. If ls /proc/<pid>/fd | wc -l keeps climbing over time or CLOSE_WAIT sockets pile up, the application is not closing sockets/files. You have to find the missing close in the code for a real fix.

Q. ulimit is not applied inside a Docker container. A. Containers follow the runtime setting, not the host ulimit. Set docker run --ulimit nofile=65535:65535 or compose ulimits:, then check from inside the container with cat /proc/1/limits. On Kubernetes, inspect the node runtime settings.