DB-유

Disk Usage on Linux: Finding What Fills Your Disk

· linux, disk, troubleshooting

The alert says root is at 95 percent, the database is refusing writes, and I have a few minutes. It's the emergency I get most often, and the tools are all standard: df, du, find, lsof. What helps is doing it in the same order every time. Here's my order, plus the two cases that fool me: deleted files still held open, and files hidden under a mount point.

Step 1: Which filesystem is full?

Start with df, and check inodes as well as blocks. A filesystem full of tiny files (sessions, mail queue, a runaway cache) can run out of inodes with plenty of bytes left.

df -hT
df -i

-T shows the filesystem type, which matters later; -i shows inode usage. If IUse% is near 100 while Use% is low, you are hunting for millions of small files rather than a few huge ones.

Step 2: Biggest directories

du over the whole filesystem is slow but definitive. Always add -x to stay on one filesystem and out of /proc, /sys and NFS mounts:

sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head -20

Then drill into the largest entry and repeat:

sudo du -xh --max-depth=1 /var 2>/dev/null | sort -rh | head -20

On CentOS 7 the usual suspects are /var/log, /var/lib/mysql, /var/cache/yum and /var/lib/docker. On Ubuntu add /var/cache/apt and /var/lib/snapd.

If I'm allowed to install a package, I use ncdu (EPEL on CentOS, main repo on Ubuntu). Same idea, far easier to navigate:

sudo ncdu -x /

Step 3: Biggest files

Sometimes it's one file. find with a size threshold, on one filesystem:

sudo find / -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head

Common finds: a /var/log/messages or syslog that logrotate stopped rotating, a MySQL general log someone turned on for debugging and forgot, a core dump in an application directory, or an old .tar.gz backup left in /root.

Step 4: Deleted but still open

This is what makes du and df disagree. If a process has a file open and you delete it, the name goes away but the blocks aren't freed until the process closes it. du can't see it; df still counts it. The classic is rm-ing a huge log while the daemon is still writing to it, which I have done.

sudo lsof +L1
sudo lsof -nP | grep '(deleted)'

The output shows the process, PID and file descriptor. You don't have to restart anything; truncate the file through /proc:

sudo : > /proc/1234/fd/5

Use the PID and FD number from the lsof output. From now on I truncate logs (: > file) instead of deleting them, or use logrotate with copytruncate.

Step 5: Hidden under a mount point

If a mount failed at boot and a service wrote into the empty mount point anyway, those files sit on the root filesystem, hidden once the real mount is back. du can't see through it. Bind-mount root somewhere else and look:

sudo mkdir /mnt/rootfs
sudo mount --bind / /mnt/rootfs
sudo du -sh /mnt/rootfs/data /mnt/rootfs/mnt/*
sudo umount /mnt/rootfs

Step 6: Getting space back

These are always safe:

sudo journalctl --vacuum-size=200M
sudo yum clean all                    # CentOS
sudo apt-get clean                    # Ubuntu
sudo find /var/log -name "*.gz" -mtime +30 -delete
sudo docker system prune              # if Docker is installed

For MySQL, check SHOW VARIABLES LIKE 'general_log%' and the binary log retention (expire_logs_days) before deleting anything under /var/lib/mysql; never remove InnoDB files by hand.

Did the number move?

Run df -h again. If it didn't move after deleting big files, go back to Step 4; something still holds them. Run sync first if you're impatient, since ext4 and XFS free blocks asynchronously.

Oddities

  • du: cannot read directory '/proc/…': Permission denied or huge numbers for /proc: you forgot -x, or you aren't root.
  • df says 100% but du adds up to far less: deleted-but-open files (Step 4) or files under a mount (Step 5). Occasionally it's reserved blocks; ext4 reserves 5 percent for root by default (tune2fs -l /dev/sda1 | grep Reserved).
  • No space left on device while df -h shows free space: inodes. df -i, then find the directory with the most files: sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head.
  • XFS can't be shrunk, and ext4 needs the filesystem unmounted for resize2fs to shrink it. Growing is the easy direction, so size accordingly.

Once the fire is out, fix the cause: a logrotate rule or retention setting. And set the monitoring threshold to warn at 80 percent, not 95.