DB-유

Writing Your Own systemd Service Unit

· linux, systemd

For too long I kept things alive in a screen session, right up until a reboot took out something someone had handed me as a Java jar. Writing a proper systemd unit takes about ten minutes and gives you start at boot, restart on failure, journal logging and the usual systemctl commands.

Here's a unit for a typical long-running network service, what the directives mean, and why units refuse to start. It assumes CentOS 7, Ubuntu 16.04 or Debian 8+, and an app already installed at /opt/myapp/myapp that runs fine by hand.

A dedicated user

Services should not run as root unless they truly need to. Create a system account without a login shell or home directory:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin myapp
sudo chown -R myapp:myapp /opt/myapp

On CentOS the nologin shell lives at /sbin/nologin.

The unit file

Local units belong in /etc/systemd/system/. Never edit files in /lib/systemd/system/ or /usr/lib/systemd/system/; those belong to packages and are overwritten on upgrade.

sudo nano /etc/systemd/system/myapp.service

[Unit]
Description=My application API server
Documentation=https://wiki.example.com/myapp
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=-/etc/default/myapp
ExecStart=/opt/myapp/myapp --config /etc/myapp/config.yml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
KillMode=mixed

# Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true

[Install]
WantedBy=multi-user.target

The directives worth understanding:

  • Type=simple means systemd considers the service started as soon as the process is forked. Use Type=forking only for daemons that background themselves, and Type=notify if the program uses sd_notify.
  • After= orders startup; Wants= actually pulls the target in. Together they make sure the network is up before the service starts.
  • EnvironmentFile=- with the leading dash means "do not fail if the file is missing." The file holds KEY=value lines and is the right place for secrets you do not want in the unit itself.
  • Restart=on-failure restarts on a non-zero exit or a signal, but not on a clean exit. Restart=always is appropriate if the program is expected to run forever.
  • ProtectSystem=full mounts /usr, /boot and /etc read-only for the service; ProtectHome=true hides /home. These are available on systemd 214 and later, which includes CentOS 7 (219) and Ubuntu 16.04 (229).

Environment file

sudo nano /etc/default/myapp

MYAPP_PORT=8080
MYAPP_LOG_LEVEL=info

Make it unreadable by others if it contains credentials: sudo chmod 640 /etc/default/myapp.

Load, enable, start

Every time you add or change a unit file, systemd needs to reread it. I forget this one constantly:

sudo systemctl daemon-reload
sudo systemctl enable myapp
sudo systemctl start myapp

enable creates a symlink in /etc/systemd/system/multi-user.target.wants/, which is what makes the service start at boot. On systemd 220 and later you can combine both with systemctl enable --now myapp.

Prove it survives

systemctl status myapp
journalctl -u myapp -f

status should show Active: active (running) and the main PID. The journal shows whatever the program writes to stdout and stderr; no log file configuration needed. Confirm it survives a crash by killing it:

sudo kill -9 $(systemctl show -p MainPID --value myapp)
sleep 6
systemctl is-active myapp

After RestartSec seconds it should be active again. Then reboot once and check that it came up on its own. I always do this, because a unit that only works after a manual start is a trap for whoever reboots next.

Why it won't start

  • Failed to start myapp.service: Unit myapp.service not found. You forgot daemon-reload after creating the file, or the file is not in /etc/systemd/system/.
  • myapp.service: Failed at step EXEC spawning /opt/myapp/myapp: Permission denied. The binary is not executable, or on CentOS it has the wrong SELinux context. chmod +x and restorecon -v /opt/myapp/myapp usually settle it. Note that ExecStart requires an absolute path; myapp alone will not work.
  • Start request repeated too quickly. The program is exiting immediately and hitting systemd's default limit of five starts in ten seconds. Look at journalctl -u myapp -n 50 for the actual error, then systemctl reset-failed myapp before trying again.
  • Service starts but cannot resolve hostnames at boot. You used After=network.target, which only means the network stack is configured, not that an address is up. Switch to network-online.target as shown above.

I keep these in version control next to the app. When you want more, systemd.exec(5) and systemd.service(5) are long but worth reading once.