Bash Scripting for Sysadmins: Essentials

Most of my production scripts began as three commands I got tired of retyping. The gap between a script that runs from cron for years and one that quietly eats a backup is a handful of habits: fail on errors, quote variables, check inputs, log what happened. None are hard. They're just easy to skip when the script "works on my terminal", and I've skipped them.

This is my checklist for anything that runs unattended, with a small maintenance script as the example. It sticks to Bash 4.x (CentOS 7 has 4.2, Ubuntu 16.04/18.04 have 4.3/4.4). Install shellcheck too: sudo apt install shellcheck, or sudo yum install ShellCheck from EPEL.

Strict header

Every script starts with this:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

-e exits on the first failing command, -u treats an unset variable as an error rather than an empty string, and -o pipefail makes a pipeline fail if any stage fails, not just the last one. Without pipefail, mysqldump db | gzip > backup.gz succeeds even when mysqldump dies, because gzip exits zero. The restricted IFS stops word splitting on spaces, which prevents filenames with spaces from being torn into pieces.

Quote everything, use "$@"

target_dir="/var/backups/app data"
rm -rf "$target_dir"/*.tmp      # correct
rm -rf $target_dir/*.tmp        # deletes /var/backups/app and data/*.tmp

The unquoted version is the classic way to lose a directory. The same applies to arguments: pass them through as "$@", never $* or $@ without quotes. shellcheck flags every instance of this, which is reason enough to run it.

Validate inputs, fail loudly

Check argument count, check that paths exist, and write errors to stderr so they are not mixed into output that another program parses:

usage() {
    echo "Usage: $(basename "$0") <database> <retention_days>" >&2
    exit 64
}

[[ $# -eq 2 ]] || usage
db_name="$1"
retention="$2"
[[ "$retention" =~ ^[0-9]+$ ]] || { echo "retention must be an integer" >&2; exit 64; }
command -v mysqldump >/dev/null || { echo "mysqldump not found" >&2; exit 69; }

Exit codes 64 through 78 come from sysexits.h and give the caller more to go on than a bare 1.

Clean up with a trap

Temporary files and lock files should disappear even when the script is killed or fails halfway:

tmp_file=$(mktemp /tmp/backup.XXXXXX)
lock_file="/var/lock/backup-${db_name}.lock"

cleanup() {
    rm -f "$tmp_file" "$lock_file"
}
trap cleanup EXIT

if ! ( set -o noclobber; echo "$$" > "$lock_file" ) 2>/dev/null; then
    echo "another instance is running (pid $(cat "$lock_file"))" >&2
    exit 75
fi

The noclobber trick creates the lock file atomically, so two cron runs cannot both proceed. These days I just reach for flock -n "$lock_file" command from util-linux, which does the same thing in one line.

Log with timestamps

log() {
    printf '%s %s\n' "$(date '+%F %T')" "$*" >&2
}

log "starting backup of ${db_name}"
mysqldump --single-transaction "$db_name" | gzip > "$tmp_file"
mv "$tmp_file" "/var/backups/${db_name}-$(date +%F).sql.gz"
log "removing dumps older than ${retention} days"
find /var/backups -name "${db_name}-*.sql.gz" -mtime +"$retention" -delete
log "done"

Writing to a temp file and then mv into place means a half-written dump never has the final name. With cron you can redirect both streams to a log file: 0 2 * * * /usr/local/sbin/db-backup.sh appdb 14 >> /var/log/db-backup.log 2>&1.

Lint and test

shellcheck /usr/local/sbin/db-backup.sh
bash -n /usr/local/sbin/db-backup.sh

-n checks syntax without running. Then run it once by hand with a throwaway database and inspect the log and exit code (echo $?). A good run leaves one new file with today's date, three timestamped log lines, and no leftover /tmp/backup.* or lock file. Start a second copy while the first is still running; it should exit 75 with the "another instance" message.

Errors you'll see

• line 12: db_name: unbound variable. The effect of set -u: a variable was referenced before assignment, often a typo. Use ${VAR:-default} when a variable is legitimately optional.
• Script exits with no message halfway through. set -e stopped on a command you expected to fail harmlessly, such as grep finding nothing. Append || true to commands whose non-zero exit is acceptable.
• Works interactively, fails from cron with command not found. Cron's PATH is minimal. Set PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin at the top of the script or use absolute paths.
• [[: not found or syntax error near unexpected token. The script is being run by sh (dash on Ubuntu) rather than bash: it was invoked as sh script.sh, or the shebang is missing. Run it as ./script.sh with the shebang in place.

When I inherit someone else's script, the header is the first thing I look at. No set -e usually tells me what the rest looks like.