Reading System Logs with journalctl

I resisted the journal for a long time because I couldn't grep it and I missed my /var/log/messages. Then I learned you can filter by unit, priority, time range or boot without any regex, and I stopped missing anything. Once a few flags become habit it's faster than paging through text logs.

These are the invocations I use most on CentOS 7 and Ubuntu 16.04 (anything on systemd 219 or newer), plus the bit of configuration that keeps the journal across reboots and stops it eating the disk. You'll want sudo, or membership in systemd-journal or adm, to see other units' logs.

Viewing and following

With no arguments, journalctl shows everything since the oldest retained entry, piped through less. That is rarely what you want. Start from the end and follow:

sudo journalctl -e          # jump to the end
sudo journalctl -f          # follow, like tail -f
sudo journalctl -n 100      # last 100 lines

Add --no-pager when piping to another command, and -o short-iso if you prefer ISO timestamps over the syslog style:

sudo journalctl -n 50 --no-pager -o short-iso

By unit

This is the filter I use ten times a day. It shows messages the service wrote to stdout/stderr plus systemd's own messages about starting and stopping it:

sudo journalctl -u nginx
sudo journalctl -u nginx -u php7.0-fpm --since today

For a template unit such as getty@tty1, quote the name. To see everything a given process wrote regardless of unit, filter by PID:

sudo journalctl _PID=1234

By time and by boot

--since and --until accept absolute timestamps and a number of relative phrases:

sudo journalctl --since "22:00" --until "23:30"
sudo journalctl --since yesterday
sudo journalctl --since "1 hour ago"

To look at a specific boot, list them first and then pick one. -b alone is the current boot, -b -1 the previous:

sudo journalctl --list-boots
sudo journalctl -b -1 -p err

If --list-boots shows only the current boot, the journal is not being persisted; see the persistence section below.

By priority and text

-p takes a syslog priority name or number and shows that level and everything more severe:

sudo journalctl -p warning -b
sudo journalctl -p err..alert --since today

Kernel messages only, equivalent to dmesg but with timestamps and persistence:

sudo journalctl -k -b

There is no built-in full-text search across fields on the systemd versions shipped by CentOS 7 and Ubuntu 16.04 (the -g option arrived in systemd 237, which neither has), so pipe to grep:

sudo journalctl -u sshd --since today --no-pager | grep -i 'failed password'

For structured output that scripts can parse, use -o json or -o json-pretty. -o verbose shows every field of each entry, which is how you discover field names such as _SYSTEMD_UNIT or _COMM to filter on.

Keeping the journal across reboots

CentOS 7 keeps the journal in /run/log/journal by default, which is tmpfs and vanishes at reboot; rsyslog receives a copy and writes /var/log/messages. Ubuntu 16.04 also uses volatile storage unless the directory exists. To persist it:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Then cap the size in /etc/systemd/journald.conf:

[Journal]
Storage=persistent
SystemMaxUse=500M
SystemMaxFileSize=50M
MaxRetentionSec=1month

Restart systemd-journald after editing. To trim an already large journal right away:

sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=2weeks

Do this before you need it. I learned that the hard way after an unexplained reboot left me with an empty journal.

Quick check

journalctl --disk-usage
journalctl --verify
journalctl --list-boots

--disk-usage should be at or under the cap you set, --verify should print PASS for each file, and after a reboot --list-boots should now show more than one entry.

Notes

• No journal files were found. Either you lack permission (add yourself to systemd-journal on CentOS or adm on Ubuntu, then log in again) or Storage=none is set in journald.conf.
• Hint: You are currently not seeing messages from other users and the system. Same cause: you are reading only your own user journal. Prefix with sudo.
• -- Reboot -- lines but --list-boots shows one boot. Boot IDs are only tracked when storage is persistent. Make the journal persistent as above.
• Journal consumes several gigabytes. SystemMaxUse defaults to 10 percent of the filesystem, capped at 4 GB. Set an explicit limit and vacuum.

Half a dozen flags (-u, -f, -b, -p, --since, -k) cover most of it. My favorite is still journalctl -b -1 -p err after a surprise reboot.