No space left on device: a 30-second triage runbook — when df shows space but writes still fail

3 a.m.: df -h shows free space, but the deploy is failing

The deploy pipeline has stalled, and the application log just keeps repeating a single line.

write /var/log/app/app.log: no space left on device

no space left on device is the kernel-level error that surfaces as ENOSPC (errno 28). The name sounds like "disk is full," but in incident response this is exactly where people get confused.

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   50G   31G   17G  65% /

There's 17G free — so why can't we write? Panic here and you'll burn 30 minutes. In reality, ENOSPC has at least five causes, not just "partition is full." This post is a runbook that branches those five causes with five commands in 30 seconds, then recovers with copy-paste commands per cause and stops it from happening again. It's built around command blocks you can paste during an incident.

Note: The commands below assume a typical Linux (systemd-based) environment. Where interpretation depends on distro, filesystem, or container runtime, each section is marked "Varies by environment."

30-second triage tree — narrow the cause with five commands

When an incident hits, don't think — run these five in order from top to bottom. Most of the time the cause is identified by the third command.

# 1. ?
df -h

# 2. inode ? ( 100% )
df -i

# 3. file ?
sudo lsof +L1 2>/dev/null | head -20

# 4. ?
sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head

# 5. Docker //volume ?
docker system df

Map the results to the table below and you'll have the cause.

The key is to always look at df -h and df -i together. If you only check df -h, see "there's space," and move on, you'll miss inode exhaustion.

Two traps that df alone gets wrong

ENOSPC is confusing because of two cases where df -h doesn't match reality. With inode exhaustion, df -h looks like it has free space; with deleted-but-open files, df is full but du can't find the culprit.

First check which filesystem the failing path actually lives on. Even if / has space, /var/log, /tmp (tmpfs), or a container volume may be a separate mount that is full.

findmnt -T /var/log/app       # the mount and device this path belongs to
df -h /var/log/app; df -i /var/log/app

Trap A: inode exhaustion — 60% used, but you can't create files

A filesystem uses inodes — metadata slots — separately from data blocks. Each file needs one inode. When millions of tiny files pile up (session files, caches, mail queues), space looks fine but inodes run out first.

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1 50G 30G 20G 60% / # 60%

$ df -i /
Filesystem       Inodes  IUsed IFree IUse% Mounted on
/dev/nvme0n1p1 3276800 3276800 0 100% / # inode 100%!

If IUse% is 100%, that's definitive. Now find where the inodes went.

# file
for d in /tmp /var/tmp /var/lib/php/sessions /var/spool; do
  echo -n "$d: "; find "$d" -xdev -type f 2>/dev/null | wc -l
done

Varies by environment: On ext4, the inode count is fixed at format time (you can't grow it later). xfs allocates inodes dynamically, so IFree in df -i can look different depending on the situation and exhaustion is generally less common. This trap shows up especially often on ext4.

Trap B: deleted files still held open by a process

You deleted a huge log with rm -f app.log, but the space never came back. du doesn't catch it, yet df is still full.

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   50G   50G     0  100% /

$ sudo du -xh --max-depth=1 / | sort -rh | head -3
12G   /var
8.0G  /usr
3.0G  /home
# 50G → du

$ sudo lsof +L1
COMMAND   PID  USER   FD   TYPE ...  SIZE/OFF NLINK    NODE NAME
java     2314  app    5w   REG  ...  23622320128  0  1835012 /var/log/app/app.log (deleted)

That's the answer. Even after rm, if a process still holds the file handle (fd), the inode is not released and the space is not returned. NLINK 0 plus (deleted) is the smoking gun. Note the PID before NAME (2314) and the FD number (5).

Copy-paste recovery per cause

Cause identified — now fix it. Dangerous commands have warnings; don't skip them.

1) systemd journal / log cleanup

Logs piling up unbounded in /var/log/journal is common.

# check
journalctl --disk-usage

# 200M (example — )
sudo journalctl --vacuum-size=200M

# when
sudo journalctl --vacuum-time=3d

# after check (: Archived and active journals take under 200.0M)
journalctl --disk-usage

After the one-off cleanup, set a permanent cap.

sudo mkdir -p /etc/systemd/journald.conf.d
printf '[Journal]\nSystemMaxUse=500M\n' | sudo tee /etc/systemd/journald.conf.d/size.conf
sudo systemctl restart systemd-journald

Varies by environment: journalctl only exists on systemd-based distros. If you don't have systemd, or logs go straight to /var/log/*.log files, this command isn't available. In that case, jump to large-file hunting below.

2) Hunting large files / core dumps

# /var 100M file ( : -xdev)
sudo find /var -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null

#
sudo find / -xdev -name 'core.*' -type f 2>/dev/null
sudo find / -xdev -name 'core' -type f 2>/dev/null

# check after (path when !)
# sudo rm -f /var/dump/core.12345

-xdev keeps you from crossing into other mounts so you stay on the partition that actually ran out.

3) Docker — cleaning up /var/lib/docker buildup

A classic ENOSPC cause on container/CI runner hosts. Overlay layers, stopped containers, dangling images, and build cache keep accumulating.

docker system df
# TYPE            TOTAL  ACTIVE  SIZE     RECLAIMABLE
# Images          48     6       22.3GB   18.1GB (81%)
# Containers      12     3       1.2GB    900MB
# Build Cache     210    0       9.4GB    9.4GB

You can also see which part is large at the directory level.

sudo du -sh /var/lib/docker/* 2>/dev/null | sort -rh | head
docker image prune -a          # remove only images not used by any container

If RECLAIMABLE is large, that's your cleanup target. Narrow it down in a safe order.

# ·dangling · · when
docker system prune

# to (run )
docker system prune -a

# when
docker builder prune

⚠️ Danger — the --volumes flag

BashCopy

docker system prune -a --volumes # ←

> `--volumes` **permanently deletes data in volumes not referenced by any container**. On Docker 23 and later it removes only **anonymous volumes** by default and keeps named ones (`docker volume prune -a` removes named volumes too); older versions remove named volumes as well. If you ran an image that declares `VOLUME`, like the official DB images, without `-v`, its data lives in an anonymous volume, so removing the container (`docker compose down`, etc.) and then pruning **can wipe the DB data in one shot.** Volumes of a container that is merely stopped are still referenced and are not removed. Always list volumes with `docker volume ls` and delete them individually.

> **Varies by environment:** The `/var/lib/docker` path may be customized via `data-root` in `daemon.json`. Layer storage also differs by storage driver (overlay2, etc.). Check the actual location first with `docker info | grep -e "Docker Root Dir" -e "Storage Driver"`.

### 4) Deleted but still-open files — truncate without downtime

**The proper fix is to restart the process holding the file.** Restart closes the fd and space is returned immediately.

```bash
# check PID(2314) when (: systemd )
sudo systemctl restart app.service

If restart isn't possible (you need zero downtime), truncate the open file via /proc/<PID>/fd/<N>. Even a deleted file is reachable through its fd path.

# lsof check PID=2314, FD=5
# (when lsof +L1 when after!)
sudo truncate -s 0 /proc/2314/fd/5
# or
sudo sh -c ': > /proc/2314/fd/5'

The process keeps writing to that fd, but the file contents become 0 bytes so space is returned. This is a temporary measure. The process is still writing to a "file that doesn't exist," so as soon as you have breathing room, restart to normalize the fd and fix log rotation.

5) Inode exhaustion — deleting masses of small files

The problem is count, not size, so you delete clumps of small files, not large ones. Count first so you know you have the right target.

# (: /tmp file )
find /tmp -xdev -type f 2>/dev/null | wc -l

# check after ( whenfile -mtime )
sudo find /tmp -xdev -type f -mtime +3 -delete

# file
# sudo find /var/lib/php/sessions -xdev -type f -mtime +1 -delete

rm -rf * fails with argument list too long when there are too many arguments, so for bulk deletes find ... -delete is safer.

6) ENOSPC while both df and df -i look fine — the inotify watch limit

If tail -f, a dev server's file watcher (webpack, Vite, etc.), or a log shipper reports No space left on device while disk space and inodes both have room, you've hit the inotify watch limit, not the disk. inotify_add_watch returns the same ENOSPC.

cat /proc/sys/fs/inotify/max_user_watches
sudo sysctl fs.inotify.max_user_watches=524288                                   # apply now
echo 'fs.inotify.max_user_watches=524288' | sudo tee /etc/sysctl.d/90-inotify.conf  # persist

A higher value lets watches use more kernel memory, so also consider watching fewer directories (e.g. excluding node_modules).

Preventing recurrence — put these in place within 30 minutes of recovery

You put the fire out. Now make sure it doesn't start again.

logrotate config

Unrotated logs are an ENOSPC regular. Especially easy to miss on ephemeral nodes or containers that don't live long.

# /etc/logrotate.d/app
/var/log/app/*.log {
 size 100M # 100M (example — )
 rotate 5 # 5
 compress # gzip
    delaycompress
    missingok
    notifempty
 copytruncate # when without (fd )
}

copytruncate is especially useful for preventing Trap B (deleted-but-open files). It empties the file instead of deleting it, so the fd stays valid.

# settings run
sudo logrotate -d /etc/logrotate.d/app   # dry-run
sudo logrotate -f /etc/logrotate.d/app # run

Monitoring thresholds — watch space and inodes separately

The most important lesson: if you only monitor space (df -h), you'll miss inode exhaustion again. Inode usage must be a separate metric.

The 80%/90% figures above are examples. Lower them if logs can explode quickly; raise them for stable workloads. Tune per workload.

If you use Prometheus, the key is two separate alerts: node_exporter's node_filesystem_files_free (inodes) and node_filesystem_avail_bytes (space).

Triage tree cheat sheet

When an incident hits, remember this order.

1. df -h → 100% → du -xh file
2. df -i → 100% inode → file find -delete
3. lsof +L1 → (deleted) file → when or truncate
4. du -xh / →
5. docker system df → prune (--volumes !)

Always treat df -h and df -i as a pair, and never casually run --volumes. Keep those two rules and 3 a.m. ENOSPC becomes a 30-second problem.

FAQ

Q. df -h still shows free space — why am I getting no space left on device? A. Three typical reasons. (1) Inode exhaustion — too many small files, so IUse% in df -i is 100%. (2) The failing path is on a different mount (tmpfs, a separate partition, a container volume) — check with findmnt -T <path>. (3) The inotify watch limit — it returns the same error regardless of disk. Note that a deleted-but-open file is the opposite case: df is full while the du total is small, and you find it with lsof +L1.

Q. Is docker system prune -a --volumes safe? A. Not necessarily. --volumes permanently deletes data in volumes not referenced by any container (only anonymous volumes on Docker 23+, named ones too on older versions). Right after removing a DB container that kept its data in an anonymous volume, you can lose that data. List volumes with docker volume ls first, and prefer docker system prune -a without --volumes.

Q. df -i is at 100%. Can I just grow the disk? A. It depends on the filesystem. On ext4 the inode count is fixed at format time, so enlarging the disk does not add inodes (you'd need a reformat or different mkfs options). xfs allocates inodes dynamically, so you generally have more headroom. The real fix is to stop creating masses of small files (sessions, caches, temp files) and schedule regular cleanup.