DB-유

Automated Offsite Backups with rsync over SSH

· linux, backup, rsync, ssh

A backup on the same machine as the data is a convenience, not a backup. I've watched a disk failure take out both. The offsite setup I trust most needs nothing fancy: rsync over SSH to a second machine, run from cron, with a restricted key, so that someone who breaks into the source server can't wipe the backups too.

Here's how I wire a web server (the source) to a backup host, using --link-dest to keep daily snapshots without full copies of unchanged files. rsync and OpenSSH are on both hosts by default on CentOS 7 and Ubuntu 16.04/18.04. On the backup host I use a dedicated backup account, with data under /srv/backup.

1. A dedicated key on the source

Run the backup as root on the source so it can read everything, but with its own key rather than your personal one:

sudo ssh-keygen -t ed25519 -N '' -f /root/.ssh/backup_ed25519 -C "backup@web01"
sudo cat /root/.ssh/backup_ed25519.pub

-N '' gives an empty passphrase, which is unavoidable for an unattended job. That's why the next step restricts what the key can do.

2. Restrict the key on the backup host

On the backup host, add the public key to /home/backup/.ssh/authorized_keys with a forced command and no other capabilities:

command="rsync --server -logDtprze.iLsfxC --link-dest=/srv/backup/web01/latest . /srv/backup/web01/incoming/",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAA...rest-of-key backup@web01

The command= forces the key to run only an rsync server for that path. The exact argument string must match what the client sends; the easy way to get it right is to run the client with -e 'ssh -v' once and copy the Sending command: line from the output, or use the rrsync script shipped in rsync's documentation directory (/usr/share/doc/rsync/scripts/rrsync on Ubuntu) which restricts by directory instead:

command="/usr/local/bin/rrsync /srv/backup/web01",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAA... backup@web01

I stick to rrsync because it survives small changes to client flags; the forced-command string broke on me the first time I added an option. Copy it into place and make it executable.

Create the directory layout:

sudo mkdir -p /srv/backup/web01
sudo chown -R backup:backup /srv/backup/web01

3. The backup script

sudo nano /usr/local/sbin/offsite-backup.sh

#!/bin/bash
set -euo pipefail

DEST_HOST="backup@backup.example.com"
DEST_BASE="/srv/backup/web01"
KEY="/root/.ssh/backup_ed25519"
DATE=$(date +%Y-%m-%d)
LOG="/var/log/offsite-backup.log"

# Dump databases first so we back up a consistent copy, not live files.
mysqldump --single-transaction --all-databases | gzip > /var/backups/mysql-all.sql.gz

rsync -az --delete \
  --exclude='/var/lib/mysql' \
  --exclude='/var/cache' \
  --exclude='*.tmp' \
  --link-dest="${DEST_BASE}/latest" \
  -e "ssh -i ${KEY} -o BatchMode=yes -o StrictHostKeyChecking=yes" \
  /etc /var/www /var/backups /home \
  "${DEST_HOST}:${DEST_BASE}/${DATE}/" >> "$LOG" 2>&1

echo "$(date -Is) backup ${DATE} completed" >> "$LOG"

sudo chmod 700 /usr/local/sbin/offsite-backup.sh

-a preserves permissions, ownership and timestamps, -z compresses in transit, --delete removes files on the destination that no longer exist at the source (within today's snapshot only), and --link-dest hard-links unchanged files to the previous snapshot so each daily directory looks complete but only costs the space of the changed files. BatchMode=yes makes ssh fail instead of prompting when something is wrong, which is what you want from cron.

4. Rotation and pruning, on the backup host

With rrsync the client cannot run arbitrary commands, so maintain the latest symlink and prune old snapshots from the backup host's own cron:

crontab -e   # as the backup user

30 3 * * * cd /srv/backup/web01 && ln -sfn "$(ls -d 20* | tail -n 1)" latest && ls -d 20* | head -n -14 | xargs -r rm -rf

That keeps fourteen daily snapshots. Because of the hard links, deleting an old directory frees only the blocks no newer snapshot references.

5. Cron on the source

sudo crontab -e

0 2 * * * /usr/local/sbin/offsite-backup.sh

Run it once by hand first so the host key gets accepted into /root/.ssh/known_hosts; with StrictHostKeyChecking=yes a first run from cron would fail.

Check it, then restore something

On the backup host:

ls -la /srv/backup/web01/
du -sh /srv/backup/web01/*
stat -c '%h %n' /srv/backup/web01/latest/etc/hostname

The link count printed by stat should equal the number of snapshots that contain an unchanged copy of that file, which confirms --link-dest is working. Then do the only test that matters: restore something.

rsync -av backup.example.com:/srv/backup/web01/latest/etc/nginx/ /tmp/nginx-restore/

Problems you'll probably meet

  • rsync: connection unexpectedly closed or Permission denied (publickey) from cron but not by hand. The cron environment has no agent and a different HOME. The -i and BatchMode options in the script avoid that; check also that the key is owned by root with mode 600.
  • --link-dest arg does not exist: /srv/backup/web01/latest. Harmless on the very first run; rsync warns and does a full copy. The latest link exists after the first rotation.
  • rsync error: some files/attrs were not transferred (code 23). Usually a file changed while being read or an unreadable path. Look at the log for the specific file; --exclude it if it is a socket or a lock file.
  • Backup host fills up faster than expected. --link-dest only helps for files whose content, permissions and timestamps are all unchanged. Database dumps and rotated logs are new every day; keep those under a shorter retention or compress them before rsync.

Test that restore every few months. A backup nobody has restored from is a hypothesis.