DB-유

Using rsync for Backups and Mirroring

· linux, rsync, backup

rsync has been my default tool for moving directories between servers for as long as I have been doing this. It only transfers what changed, it preserves permissions and timestamps, it resumes cleanly after a dropped connection, and it is installed everywhere. The flip side is that the option set is large and a couple of the options (--delete above all) can destroy data if you point them the wrong way.

Below are the invocations I actually use: a plain copy, a nightly mirror, a mirror with excludes for a web root, and hard-link snapshots for daily history. You need rsync on both ends (3.1.2 in CentOS 7 and Ubuntu 18.04 is fine) and SSH key auth if it's going to run unattended.

The trailing-slash rule

This is the source of most rsync surprises. With a trailing slash on the source, the contents of the directory are copied; without it, the directory itself is copied into the destination.

rsync -av /var/www/example.com/ backup:/srv/backup/example.com/
rsync -av /var/www/example.com  backup:/srv/backup/

Both of these produce /srv/backup/example.com/html/... on the destination. Mixing them up produces /srv/backup/example.com/example.com/html/.... The trailing slash on the destination does not matter. I always run a new command with -n (dry run) first and look at the paths it prints.

A basic copy

rsync -avhP --stats /var/lib/mysql-dumps/ backup.example.internal:/srv/dumps/db01/
  • -a archive: recursive, preserves symlinks, permissions, times, group, owner (when root), and device files. It is -rlptgoD spelled out.
  • -v verbose, -h human-readable sizes.
  • -P is --partial --progress: keep partially transferred files so a retry resumes, and show progress.
  • --stats prints a summary at the end, which is what I log from cron.

On a slow WAN link add -z to compress in transit, and --bwlimit=20000 (KB/s) so the backup does not starve replication traffic on the same link.

Mirroring with --delete

A backup that should be an exact copy needs --delete, which removes files on the destination that no longer exist on the source. I treat it like rm -rf. Dry run first:

rsync -avn --delete /var/www/example.com/ backup:/srv/backup/example.com/

Read the deleting ... lines in the dry-run output. If they look right, run without -n. Two guards I use with --delete on anything important:

rsync -av --delete --delete-after --max-delete=100 \
  --exclude 'cache/' --exclude '*.log' --exclude '.git/' \
  /var/www/example.com/ backup:/srv/backup/example.com/

--delete-after waits until the transfer succeeds before removing anything, and --max-delete=100 aborts if more than 100 deletions would happen, which catches the case where the source directory is empty because a disk failed to mount. The exit code in that case is 25.

For longer exclude lists keep them in a file and use --exclude-from=/etc/rsync-excludes.txt, one pattern per line.

The hard-link trick gives you a full directory tree for each day while only storing changed files once. Each new snapshot is compared with the previous one; unchanged files become hard links to the existing copy.

#!/bin/bash
set -euo pipefail
SRC=/var/www/example.com/
DEST=/srv/snapshots/example.com
TODAY=$(date +%Y-%m-%d)

mkdir -p "$DEST"
rsync -a --delete \
  --link-dest="$DEST/latest" \
  "$SRC" "$DEST/$TODAY/"

rm -f "$DEST/latest"
ln -s "$DEST/$TODAY" "$DEST/latest"

# keep 30 days
find "$DEST" -maxdepth 1 -type d -name '20??-??-??' -mtime +30 -exec rm -rf {} +

Save it as /usr/local/sbin/snapshot-www.sh, chmod 700, and schedule it:

30 2 * * * /usr/local/sbin/snapshot-www.sh >> /var/log/snapshot-www.log 2>&1

du -sh on a single day's directory will show the full size, but du -sh /srv/snapshots/example.com shows the real total, since hard links are counted once. This only works with a destination on a local disk or a mounted filesystem, not over an rsync-to-remote path.

Non-root on the remote side

If the destination is a shared backup server, I don't give every source root there. Create a backup user, and for backups that must preserve ownership, grant it passwordless sudo for rsync only, then use --rsync-path='sudo rsync':

rsync -a --rsync-path='sudo rsync' /etc/ backup@backup.example.internal:/srv/backup/db01/etc/

The sudoers line on the backup host: backup ALL=(root) NOPASSWD: /usr/bin/rsync.

Checking the copy

Compare the two trees after the run. -c forces checksums instead of trusting size and mtime, so it is slow but definitive:

rsync -avnc --delete /var/www/example.com/ backup:/srv/backup/example.com/ | head

An output that lists only the summary lines and no file names means the copies are identical. Also check the exit code in your cron log: 0 is success, 23 means some files could not be transferred (permissions, usually), 24 means files vanished during the transfer, which is normal for busy directories.

Common failures

  • rsync: failed to set times on "/srv/backup/...": Operation not permitted (1): the remote user does not own the directory. Either fix ownership or drop -t/-o/-g from the flags (use -rlptD instead of -a when the destination is a CIFS mount).
  • rsync error: some files/attrs were not transferred (see previous errors) (code 23): scroll up; usually a Permission denied on a root-owned file when running as a normal user.
  • rsync: connection unexpectedly closed right after connecting: rsync is not installed on the remote host, or the remote shell prints something on login (a motd echo in .bashrc), which corrupts the protocol.
  • Everything transfers every time even though nothing changed: timestamps are being lost, typically because the destination is FAT or a CIFS share with coarse mtime resolution. Add --modify-window=2.

rsync alone isn't a backup system: no catalogue, no encryption at rest, no restore testing. Test your restores.