DB-유

Sending Server Alerts by Email with mailx

· linux, email, monitoring, bash

My backup script failed for a week once and nobody knew, because the alert mail from the server's local Postfix went straight to spam. Many servers don't need a full monitoring stack, just a message when a backup fails or a disk hits 90 percent. A local MTA relaying to the world usually ends up in spam or rejected, especially on a VPS with no reverse DNS. So I have mailx talk to an authenticated SMTP server directly: the company mail server, Gmail, or a transactional provider.

This covers mailx on CentOS 7 (Heirloom) and Ubuntu 18.04 (s-nail, its successor), with authentication and TLS, and then how I call it from cron scripts. You need an SMTP account (host, port 587 with STARTTLS or 465 with implicit TLS, username, password). No local MTA is required, and a running Postfix doesn't conflict.

Install

sudo yum -y install mailx                 # CentOS 7
sudo apt-get install -y s-nail            # Ubuntu 18.04 (provides the mailx command)

On Ubuntu 16.04 the package is heirloom-mailx; the config below works for both, with one difference noted in the options.

SMTP account

The system-wide config is /etc/mail.rc on CentOS and /etc/s-nail.rc on Ubuntu. I put settings in ~/.mailrc for the root account that runs cron jobs, so the password isn't world-readable:

# /root/.mailrc
set smtp-use-starttls
set ssl-verify=ignore
set smtp=smtp://smtp.example.com:587
set smtp-auth=login
set smtp-auth-user=alerts@example.com
set smtp-auth-password=the-smtp-password
set from="alerts@example.com (Server Alerts)"
set nss-config-dir=/etc/pki/nssdb

chmod 600 /root/.mailrc

Notes on the options:

  • smtp-use-starttls is for port 587. For port 465 use set smtp=smtps://smtp.example.com:465 and drop the starttls line.
  • nss-config-dir is required only by the CentOS Heirloom build, which uses NSS for TLS. Omit it on Ubuntu.
  • ssl-verify=ignore is a shortcut for testing. Once mail flows, change it to strict and, on CentOS, make sure the CA bundle is there (yum install ca-certificates is normally enough). I've left ignore in place too long before; don't.

On Ubuntu with s-nail the option names are the same, and you can also use the newer set mta=smtp://... form.

Test message

echo "Test from $(hostname -f) at $(date)" | mailx -s "mailx test" you@example.com

Add -v to see the SMTP conversation. It's the fastest way to debug authentication:

echo test | mailx -v -s "verbose test" you@example.com

From a script

My pattern for backup and check scripts: collect output into a variable, decide, and mail only on failure. A disk-space check from cron:

#!/bin/bash
# /usr/local/bin/check-disk.sh
THRESHOLD=90
TO="ops@example.com"
HOST=$(hostname -f)

REPORT=$(df -hP -x tmpfs -x devtmpfs | awk -v t="$THRESHOLD" 'NR>1 { gsub("%","",$5); if ($5+0 >= t) print $0 }')

if [ -n "$REPORT" ]; then
    printf 'Filesystems over %s%% on %s:\n\n%s\n' "$THRESHOLD" "$HOST" "$REPORT" \
        | mailx -s "[DISK] $HOST over ${THRESHOLD}%" "$TO"
fi

To attach a file (a log, for example), mailx -a /var/log/backup.log works on both the Heirloom and s-nail versions.

Schedule it:

echo '*/30 * * * * root /usr/local/bin/check-disk.sh' | sudo tee /etc/cron.d/check-disk

Since the script prints nothing unless something is wrong, cron has nothing to mail on its own, and the alert goes through the authenticated relay.

Credentials

Never pass the password with -S smtp-auth-password=... on the command line. It shows up in ps and shell history. A ~/.mailrc with mode 600 is the place. If several users need to send, put the account in /etc/mail.rc and make it 640 with a dedicated group.

Forcing a test

Send the test message and check the receiving mailbox, spam folder included. To force the disk alert, run the check with a low threshold:

sudo THRESHOLD=1 /usr/local/bin/check-disk.sh

(Make the script accept the environment override, or edit the value temporarily.) Confirm the From: header shows your configured address; some providers rewrite it to the authenticated account, which is fine.

What goes wrong

  • smtp-server: 535 5.7.8 Authentication failed: wrong password, or the provider wants an app-specific password when two-factor auth is on. Gmail also blocks "less secure apps" by default.
  • Error in certificate: Peer's certificate issuer is not recognized. on CentOS: NSS can't find the CA. Set nss-config-dir=/etc/pki/nssdb and check that certutil -d /etc/pki/nssdb -L works, or use ssl-verify=ignore temporarily.
  • Could not connect to smtp.example.com: Connection timed out: outbound 587 or 465 is blocked. Many clouds block 25 but allow 587; try nc -zv smtp.example.com 587.
  • Sent, but the body is empty or says null: you piped nothing into mailx. Check the variable has content before the pipe.
  • Works interactively but not from cron: HOME=/root so ~/.mailrc is found, but hostname -f or other commands may not be in PATH. Set PATH at the top of the cron file.

It isn't real alerting with escalation and dedup. For a dozen servers and a handful of scripts, though, I like that there's no daemon to maintain and the mail reaches the inbox.