Fixing "No space left on device": 5-Minute Diagnosis with df, inodes, and Docker

Diagnose and Fix "No space left on device" Disk-Full Errors in 5 Minutes

An alert goes off in the middle of the night. The database has stopped, and that familiar line is sitting in the application logs.

write error: No space left on device

In a panic you run df -h and the root partition is only 50%. "There's free disk space — why can't I write?" That's one of the most confusing moments in production ops. The short answer: a disk-full error can be a capacity (block) problem or an inode exhaustion problem. The fix is completely different for each. This post is structured so you can follow diagnose → remediate → prevent recurrence command-first when you're in a panic.

5-minute diagnosis flow (summary)

1. Check for a capacity-full mount with df -h (Use% 100%?)
2. Check for inode-full with df -i (IUse% 100%?)
3. Trace the offending directory (du)
4. Check the "I deleted it but space didn't come back" trap (lsof +L1)
5. Immediate fix by situation → restore the service

Step 1: Distinguish capacity vs. inodes

The first job is to figure out which side is at 100%. Run both commands side by side.

Check capacity (blocks):

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   50G   50G     0  100% /
/dev/nvme1n1     20G  3.2G   16G   17% /data

Check inodes:

df -i

Filesystem      Inodes  IUsed  IFree IUse% Mounted on
/dev/nvme0n1p1  3.2M     3.2M      0  100% /
/dev/nvme1n1    1.3M     8.4K   1.3M    1% /data

How to read it is simple.

• Use% is 100% → Capacity (large files) is full. → Go to steps 2–3 for capacity tracing
• IUse% is 100% → You've hit the file count limit (millions of small files). Capacity can look fine and you still can't create new files. → Go to the inode fix

Both look fine but you're still getting the error? Suspect the trap where a process is holding a deleted file (later in step 2).

Step 2: Trace what's eating the capacity

If it's a capacity-full situation, start at the root and narrow down step by step.

du -sh /* 2>/dev/null | sort -rh | head

32G   /var
9.1G  /usr
4.2G  /home
...

/var is the culprit. Go one level deeper.

du -sh /var/* 2>/dev/null | sort -rh | head
du -sh /var/log/* 2>/dev/null | sort -rh | head
du -sh /var/lib/docker/* 2>/dev/null | sort -rh | head

Most incidents converge on three usual suspects.

The "I deleted it but space didn't free up" trap

If you rm'd a large log and df -h didn't change, a process is still holding the deleted file's handle (fd). Linux does not reclaim the blocks until every open file is closed.

lsof +L1

COMMAND  PID  USER  FD  TYPE  ...  NLINK  NAME
nginx    812  root  4w  REG   ...      0   /var/log/nginx/access.log (deleted)

NLINK 0 + (deleted) is the culprit. Two fixes:

# 방법 1: 해당 프로세스 재시작(가장 안전)
systemctl restart nginx

# 방법 2: 재시작이 곤란하면 fd를 비워 즉시 회수 (PID=812, FD=4)
: > /proc/812/fd/4

Step 3: Immediate fixes by situation

Log explosion

# journald 저널을 200MB로 축소
journalctl --vacuum-size=200M

# 특정 대용량 로그를 0으로 비우기 (rm 대신 truncate 권장)
truncate -s 0 /var/log/nginx/access.log

⚠️ truncate sets the file contents to zero immediately. If you need to keep the logs, back them up or compress them (gzip) first, then empty the file.

Docker buildup

As container deploys increase, images, volumes, and build cache piling up on local disk have become the #1 cause of new incidents. Clean up in stages.

# 1) 사용 안 하는(dangling) 이미지만
docker image prune

# 2) 중지된 컨테이너·네트워크·캐시 정리(이미지는 dangling만)
docker system prune

# 3) 빌드 캐시 정리
docker builder prune

# 최후의 수단: 미사용 이미지 전부 + 볼륨까지 삭제
docker system prune -a --volumes

⚠️ prune -a --volumes deletes every unused image and volume (including DB data!). A production volume can get wiped by accident, so check what you need to keep with docker volume ls first.

Inode exhaustion

If you still have capacity but IUse% is 100%, it's usually millions of small files. Find where they are.

# 디렉터리별 파일 개수 상위 추적
for d in /var/* /tmp /home/*; do echo "$(find "$d" -xdev | wc -l) $d"; done | sort -rn | head

Common causes are session files, mail queues, __pycache__, and cache fragments that never expired. Bulk deletes with rm -rf * hit the argument-list limit, so:

find /var/cache/myapp -type f -name '*.tmp' -delete

Full-mount identification table

A note from the field

In my experience, 80% of disk-full incidents blow up on servers that "used to run fine." Traffic grew, access logs piled up at several GB a day, and logrotate was missing — or a CI/CD runner kept stacking build cache forever. Once I spent 30 minutes stuck because df -h showed 70% but writes were blocked; the culprit was a deploy script that only rm'd old logs and never restarted the app, so the fd stayed open. Since then, when a disk incident hits I always run df -i and lsof +L1 first. Those two lines cut average recovery time in half.

Preventing recurrence

Once you've put out the fire, stop it from happening again with these three.

1) Auto-rotate logs with logrotate

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    copytruncate
}

2) Permanent journald size limit

# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=500M
SystemMaxFileSize=50M

systemctl restart systemd-journald

3) Alert at the 80% threshold

Cloud platforms offer EBS auto-expand, but cost and limits mean it's safer to alert at 80% usage and let a human decide. CloudWatch, Prometheus node_filesystem_avail_bytes, or even a simple cron + Slack webhook is enough.

References: official docs

The primary sources for the behavior, settings, and errors in this post are the official docs below. Check version-specific options and exact behavior there.

• Docker official documentation

FAQ

Q. df -h shows free space — why am I still getting "No space left on device"? A. Two possibilities. First, inode exhaustion (IUse% 100% on df -i) — you still have capacity but can't create more files. Second, a process is holding a deleted file (lsof +L1) — the blocks are not reclaimed.

Q. I rm'd a log file but space didn't increase. A. A process still has that fd open. Find the PID with lsof +L1 and restart the process, or in an emergency empty the fd with : > /proc/PID/fd/N to reclaim space immediately. Getting in the habit of emptying with truncate -s 0 instead of rm also helps.

Q. Is docker system prune -a safe? A. -a --volumes deletes all unused images and volumes, so production DB data can disappear. Start with narrower commands like docker image prune and docker builder prune, and always check what you need to keep with docker volume ls before deleting volumes.

Related troubleshooting worth reading as well — port conflicts (EADDRINUSE), PostgreSQL too many clients, MySQL 1045 authentication errors, diagnosing systemd service failures.