Managing Services with systemd

2017-12-11 · linux, systemd, centos, ubuntu

Both CentOS 7 and Ubuntu 16.04 boot with systemd, so the service and chkconfig commands I had in my fingers for years are now compatibility shims. They still mostly work, but they hide the useful parts: dependency ordering, automatic restarts, resource limits, and a journal that captures stdout from every daemon. This is the cheat sheet I wish I'd had when the first CentOS 7 box landed on my desk, ending with a unit file for a small custom service.

Everyday commands

The verbs are the same for every unit:

systemctl status mariadb
systemctl start mariadb
systemctl stop mariadb
systemctl restart mariadb
systemctl reload nginx
systemctl enable mariadb
systemctl disable mariadb
systemctl is-active mariadb
systemctl is-enabled mariadb

enable creates the symlink that starts the service at boot; it does not start it now. systemctl enable --now mariadb does both (available in systemd 220 and later, so on both distributions mentioned here). reload sends the service its reload signal if the unit defines ExecReload; if it does not, systemctl tells you so instead of guessing.

To see everything that failed during the last boot:

systemctl --failed
systemctl list-units --type=service --state=running

journalctl

Every service's stdout and stderr goes to the journal, which is the biggest practical improvement over SysV init. Follow a single service:

journalctl -u nginx -f
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx -b
journalctl -p err -b

-b limits output to the current boot, -p err filters by priority. On CentOS 7 the journal is not persistent by default and is lost on reboot; make it persistent with:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

Set SystemMaxUse=500M in /etc/systemd/journald.conf so it cannot fill the disk.

Overriding a vendor unit

Package-provided unit files live in /usr/lib/systemd/system/ (CentOS) or /lib/systemd/system/ (Ubuntu) and get overwritten on upgrade. I stopped editing them after losing a change that way. Use a drop-in:

systemctl edit mariadb

This opens an editor on /etc/systemd/system/mariadb.service.d/override.conf, where you put only the lines you want to change. The most common one for a database server is the open-files limit:

[Service]
LimitNOFILE=65535

Save, then reload systemd's view of the files and restart the service:

systemctl daemon-reload
systemctl restart mariadb

systemctl cat mariadb shows the merged result, vendor file plus your drop-in.

A unit for your own service

Suppose you have a small Python script that listens on a port and should run as an unprivileged user, restart if it crashes, and start after the network is up. Create /etc/systemd/system/metrics-agent.service:

[Unit]
Description=Internal metrics agent
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=metrics
Group=metrics
WorkingDirectory=/opt/metrics-agent
ExecStart=/usr/bin/python3 /opt/metrics-agent/agent.py --port 9100
Restart=on-failure
RestartSec=5s
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target

A few notes: Type=simple is right for a process that stays in the foreground; use Type=forking with PIDFile= only for old-style daemons. Restart=on-failure restarts on non-zero exit or a signal, but not on a clean exit 0, which is usually what you want. PYTHONUNBUFFERED=1 makes print statements appear in the journal immediately rather than after buffering.

Load and start it:

systemctl daemon-reload
systemctl enable --now metrics-agent

Timers instead of cron

For scheduled tasks that belong to a service, a timer unit keeps the log and the schedule together. Create /etc/systemd/system/db-backup.service with a one-shot type:

[Unit]
Description=Nightly database dump

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/db-backup.sh

and /etc/systemd/system/db-backup.timer:

[Unit]
Description=Run db-backup nightly

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

Persistent=true runs the job at next boot if the machine was off at 02:30. Enable the timer, not the service: systemctl enable --now db-backup.timer, then check systemctl list-timers.

Checking your unit

systemctl status metrics-agent
journalctl -u metrics-agent -n 20
systemd-analyze verify /etc/systemd/system/metrics-agent.service

systemd-analyze verify catches typos in directive names, which systemd otherwise ignores silently with only a warning in the journal. Kill the process with kill -9 $(systemctl show -p MainPID metrics-agent | cut -d= -f2) and watch it come back after five seconds.

Notes and error messages

Warning: metrics-agent.service changed on disk. Run 'systemctl daemon-reload' to reload units.: you edited a unit file and forgot the reload. Harmless, but the running definition is stale until you do it.
Failed to start metrics-agent.service: Unit metrics-agent.service is masked.: someone ran systemctl mask. systemctl unmask metrics-agent removes the /dev/null symlink.
Service starts at boot but fails because the network is not up: After=network.target only means networking is being configured, not that it has an address. Use network-online.target as above, and on CentOS 7 make sure NetworkManager-wait-online.service is enabled.
start request repeated too quickly: the service crashed five times in ten seconds and systemd gave up. Fix the underlying error shown in journalctl -u, then systemctl reset-failed metrics-agent before starting again.

I grumbled about systemd for a while. I don't anymore. systemctl edit and journalctl -u are the two habits that made the difference for me.