No space left on device: a 30-second triage runbook — when df shows space but writes still fail

3 a.m.: df -h shows free space, but the deploy is failing

The deploy pipeline has stalled, and the application log just keeps repeating a single line.

write /var/log/app/app.log: no space left on device

no space left on device is the kernel-level error that surfaces as ENOSPC (errno 28). The name sounds like "disk is full," but in incident response this is exactly where people get confused.

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   50G   31G   17G  65% /

There's 17G free — so why can't we write? Panic here and you'll burn 30 minutes. In reality, ENOSPC has at least five causes, not just "partition is full." This post is a runbook that branches those five causes with five commands in 30 seconds, then recovers with copy-paste commands per cause and stops it from happening again. It's built around command blocks you can paste during an incident.

Note: The commands below assume a typical Linux (systemd-based) environment. Where interpretation depends on distro, filesystem, or container runtime, each section is marked "Varies by environment."

30-second triage tree — narrow the cause with five commands

When an incident hits, don't think — run these five in order from top to bottom. Most of the time the cause is identified by the third command.

# 1. 파티션 용량이 실제로 꽉 찼나?
df -h

# 2. inode가 고갈됐나? (용량은 남았는데 여기서 100%면 그거다)
df -i

# 3. 삭제됐지만 프로세스가 붙잡고 있는 파일이 있나?
sudo lsof +L1 2>/dev/null | head -20

# 4. 어느 디렉터리가 용량을 먹고 있나? (마운트 경계 안에서)
sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head

# 5. Docker 오버레이/이미지/볼륨이 범인인가?
docker system df

Map the results to the table below and you'll have the cause.

The key is to always look at df -h and df -i together. If you only check df -h, see "there's space," and move on, you'll walk straight into two traps.

Two traps: "there's space, but writes still fail"

ENOSPC drives people crazy because of two cases where df -h looks perfectly fine.

Trap A: inode exhaustion — 60% used, but you can't create files

A filesystem uses inodes — metadata slots — separately from data blocks. Each file needs one inode. When millions of tiny files pile up (session files, caches, mail queues), space looks fine but inodes run out first.

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   50G   30G   20G  60% /      # 용량은 60%

$ df -i /
Filesystem       Inodes  IUsed IFree IUse% Mounted on
/dev/nvme0n1p1  3276800 3276800     0  100% /   # inode는 100%!

If IUse% is 100%, that's definitive. Now find where the inodes went.

# 하위 디렉터리별 파일 개수 카운트 (범인 찾기)
for d in /tmp /var/tmp /var/lib/php/sessions /var/spool; do
  echo -n "$d: "; find "$d" -xdev -type f 2>/dev/null | wc -l
done

Varies by environment: On ext4, the inode count is fixed at format time (you can't grow it later). xfs allocates inodes dynamically, so IFree in df -i can look different depending on the situation and exhaustion is generally less common. This trap shows up especially often on ext4.

Trap B: deleted files still held open by a process

You deleted a huge log with rm -f app.log, but the space never came back. du doesn't catch it, yet df is still full.

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   50G   50G     0  100% /

$ sudo du -xh --max-depth=1 / | sort -rh | head -3
12G   /var
8.0G  /usr
3.0G  /home
# 다 더해도 50G가 안 됨 → du로 안 잡히는 용량이 있다

$ sudo lsof +L1
COMMAND   PID  USER   FD   TYPE ...  SIZE/OFF NLINK    NODE NAME
java     2314  app    5w   REG  ...  23622320128  0  1835012 /var/log/app/app.log (deleted)

That's the answer. Even after rm, if a process still holds the file handle (fd), the inode is not released and the space is not returned. NLINK 0 plus (deleted) is the smoking gun. Note the PID before NAME (2314) and the FD number (5).

Copy-paste recovery per cause

Cause identified — now fix it. Dangerous commands have warnings; don't skip them.

1) systemd journal / log cleanup

Logs piling up unbounded in /var/log/journal is common.

# 현재 저널 용량 확인
journalctl --disk-usage

# 200M 남기고 정리 (예시 수치 — 워크로드별 조정)
sudo journalctl --vacuum-size=200M

# 시간 기준으로도 가능
sudo journalctl --vacuum-time=3d

# 정리 후 재확인 (예상: Archived and active journals take under 200.0M)
journalctl --disk-usage

Varies by environment: journalctl only exists on systemd-based distros. If you don't have systemd, or logs go straight to /var/log/*.log files, this command isn't available. In that case, jump to large-file hunting below.

2) Hunting large files / core dumps

# /var 안에서 100M 넘는 파일 찾기 (마운트 경계 유지: -xdev)
sudo find /var -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null

# 코어덤프 잔해 찾기
sudo find / -xdev -name 'core.*' -type f 2>/dev/null
sudo find / -xdev -name 'core' -type f 2>/dev/null

# 확인 후 삭제 (경로 반드시 눈으로 검증하고!)
# sudo rm -f /var/dump/core.12345

-xdev keeps you from crossing into other mounts so you stay on the partition that actually ran out.

3) Docker — cleaning up /var/lib/docker buildup

A classic ENOSPC cause on container/CI runner hosts. Overlay layers, stopped containers, dangling images, and build cache keep accumulating.

docker system df
# TYPE            TOTAL  ACTIVE  SIZE     RECLAIMABLE
# Images          48     6       22.3GB   18.1GB (81%)
# Containers      12     3       1.2GB    900MB
# Build Cache     210    0       9.4GB    9.4GB

If RECLAIMABLE is large, that's your cleanup target. Narrow it down in a safe order.

# (가장 안전) 중지된 컨테이너·dangling 이미지·미사용 네트워크·빌드 캐시만 정리
docker system prune

# 미사용 이미지까지 전부 (실행 중이 아닌 이미지 제거)
docker system prune -a

# 빌드 캐시만 따로
docker builder prune

⚠️ Danger — the --volumes flag

Bash복사

docker system prune -a --volumes   # ← 프로덕션에서 함부로 치지 말 것

--volumes permanently deletes data in volumes not attached to a container. If a DB container is briefly down, or a named volume is temporarily detached, you can wipe production DB data in one shot. Always list volumes with docker volume ls and delete them individually.

Varies by environment: The /var/lib/docker path may be customized via data-root in daemon.json. Layer storage also differs by storage driver (overlay2, etc.). Check the actual location first with docker info | grep -e "Docker Root Dir" -e "Storage Driver".

4) Deleted but still-open files — truncate without downtime

The proper fix is to restart the process holding the file. Restart closes the fd and space is returned immediately.

# 확인했던 PID(2314)의 프로세스를 재시작 (예: systemd 서비스)
sudo systemctl restart app.service

If restart isn't possible (you need zero downtime), truncate the open file via /proc/<PID>/fd/<N>. Even a deleted file is reachable through its fd path.

# lsof에서 확인한 PID=2314, FD=5 였다면
# (반드시 lsof +L1로 대상이 맞는지 다시 검증 후!)
sudo truncate -s 0 /proc/2314/fd/5
# 또는
sudo sh -c ': > /proc/2314/fd/5'

The process keeps writing to that fd, but the file contents become 0 bytes so space is returned. This is a temporary measure. The process is still writing to a "file that doesn't exist," so as soon as you have breathing room, restart to normalize the fd and fix log rotation.

5) Inode exhaustion — deleting masses of small files

The problem is count, not size, so you delete clumps of small files, not large ones. Count first so you know you have the right target.

# 먼저 카운트 (예: /tmp에 파일이 몇 개인가)
find /tmp -xdev -type f 2>/dev/null | wc -l

# 확인 후 삭제 (오래된 임시파일만 지우고 싶으면 -mtime 조합)
sudo find /tmp -xdev -type f -mtime +3 -delete

# 세션 파일 등 특정 디렉터리 대량 삭제
# sudo find /var/lib/php/sessions -xdev -type f -mtime +1 -delete

rm -rf * fails with argument list too long when there are too many arguments, so for bulk deletes find ... -delete is safer.

Preventing recurrence — put these in place within 30 minutes of recovery

You put the fire out. Now make sure it doesn't start again.

logrotate config

Unrotated logs are an ENOSPC regular. Especially easy to miss on ephemeral nodes or containers that don't live long.

# /etc/logrotate.d/app
/var/log/app/*.log {
    size 100M          # 100M 넘으면 로테이션 (예시 — 조정 필요)
    rotate 5           # 5개 보관
    compress           # gzip 압축
    delaycompress
    missingok
    notifempty
    copytruncate       # 앱 재시작 없이 원본을 비움 (fd 유지형 로그에 유용)
}

copytruncate is especially useful for preventing Trap B (deleted-but-open files). It empties the file instead of deleting it, so the fd stays valid.

# 설정 문법 검증 및 강제 실행 테스트
sudo logrotate -d /etc/logrotate.d/app   # dry-run
sudo logrotate -f /etc/logrotate.d/app   # 강제 실행

Monitoring thresholds — watch space and inodes separately

The most important lesson: if you only monitor space (df -h), you'll hit both traps again. Inode usage must be a separate metric.

The 80%/90% figures above are examples. Lower them if logs can explode quickly; raise them for stable workloads. Tune per workload.

If you use Prometheus, the key is two separate alerts: node_exporter's node_filesystem_files_free (inodes) and node_filesystem_avail_bytes (space).

Triage tree cheat sheet

When an incident hits, remember this order.

1. df -h        → 100%면 용량 초과 → du -xh로 큰 파일 추적
2. df -i        → 100%면 inode 고갈 → 대량 소형 파일 find -delete
3. lsof +L1     → (deleted)면 열린 파일 → 프로세스 재시작 or truncate
4. du -xh /     → 범인 디렉터리 특정
5. docker system df → 오버레이 누적이면 prune (--volumes 주의!)

Always treat df -h and df -i as a pair, and never casually run --volumes. Keep those two rules and 3 a.m. ENOSPC becomes a 30-second problem.

FAQ

Q. df -h still shows free space — why am I getting no space left on device? A. Two typical reasons. (1) Inode exhaustion — too many small files, so IUse% in df -i is 100%. (2) A deleted file still held open via an fd — it shows up as (deleted) in lsof +L1. Don't stop at df -h; check df -i and lsof +L1 together.

Q. Is docker system prune -a --volumes safe? A. Not necessarily. --volumes permanently deletes data in volumes not attached to a running container. If a DB container is briefly down, you can lose production data. List volumes with docker volume ls first, and prefer docker system prune -a without --volumes.

Q. df -i is at 100%. Can I just grow the disk? A. It depends on the filesystem. On ext4 the inode count is fixed at format time, so enlarging the disk does not add inodes (you'd need a reformat or different mkfs options). xfs allocates inodes dynamically, so you generally have more headroom. The real fix is to stop creating masses of small files (sessions, caches, temp files) and schedule regular cleanup.