Centralized Logging with rsyslog

When a server is compromised or simply dies, the logs on its own disk are the first thing you lose. Shipping them to a second machine as they are written costs almost nothing and turns a dozen ssh host; grep sessions into a single grep on one box. Before reaching for Elasticsearch or Graylog, it is worth knowing that rsyslog, already installed on CentOS 7 and Ubuntu 18.04, does this job perfectly well for a small fleet.

I've had one log box like this running for a while now. The plan: an rsyslog server that receives over TCP, writes one directory per client, and rotates the result. Clients forward through a disk-backed queue so a network blip loses nothing.

Budget disk on the log server for roughly the sum of /var/log growth across all clients, and open 514/tcp from the client subnet. rsyslog 8.x is on both sides (8.24 on CentOS 7, 8.32 on Ubuntu 18.04).

Server: receive

I use a separate file so the distribution's rsyslog.conf stays untouched.

On the log server, create /etc/rsyslog.d/10-remote.conf:

# Receive on TCP 514
module(load="imtcp")
input(type="imtcp" port="514" ruleset="remote")

# One file per host per program
template(name="RemoteFile" type="string"
         string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")

ruleset(name="remote") {
    action(type="omfile" dynaFile="RemoteFile" dirCreateMode="0755" fileCreateMode="0640")
    stop
}

The ruleset binding is the important part: messages arriving on the imtcp input are handled only by this ruleset and never fall through to the default rules, so remote logs do not also get written into the server's own /var/log/messages. stop makes that explicit.

Create the directory and open the firewall:

sudo mkdir -p /var/log/remote
sudo firewall-cmd --permanent --add-port=514/tcp    # CentOS 7
sudo firewall-cmd --reload
sudo ufw allow from 10.0.1.0/24 to any port 514 proto tcp   # Ubuntu

On CentOS 7 with SELinux, port 514 is already labelled syslogd_port_t, so no semanage step is needed. Validate and restart:

sudo rsyslogd -N1
sudo systemctl restart rsyslog

Server: rotate the remote logs

Files under /var/log/remote/ are not covered by the stock logrotate snippet. Add /etc/logrotate.d/rsyslog-remote:

/var/log/remote/*/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true
    endscript
}

Because rsyslog keeps the dynamic files open, the HUP is required so it reopens them after rotation.

Client: forward

On each client, create /etc/rsyslog.d/90-forward.conf:

# Forward everything to the central server over TCP with a disk-assisted queue
action(type="omfwd"
       target="logs.example.internal" port="514" protocol="tcp"
       queue.type="LinkedList"
       queue.filename="fwd_logs"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on"
       action.resumeRetryCount="-1")

queue.filename enables disk spooling: if the server is unreachable, messages accumulate in /var/lib/rsyslog/ (CentOS) or /var/spool/rsyslog/ (Ubuntu) up to 1 GB and are delivered when it comes back. action.resumeRetryCount="-1" means retry forever rather than dropping after the default 0 retries. Check the work directory is set in the main config (global(workDirectory="/var/lib/rsyslog") or the legacy $WorkDirectory line) and exists.

To forward only specific facilities, wrap the action in a filter:

if $syslogfacility-text == "authpriv" or $programname == "mysqld" then {
    action(type="omfwd" target="logs.example.internal" port="514" protocol="tcp")
}

Restart rsyslog on the client:

sudo rsyslogd -N1
sudo systemctl restart rsyslog

Client: app logs that don't use syslog

MySQL's error log and nginx's access log are plain files. rsyslog can tail them with the imfile module. On the client, add to 90-forward.conf above the action:

module(load="imfile")
input(type="imfile" File="/var/log/nginx/access.log" Tag="nginx-access" Severity="info" Facility="local6")
input(type="imfile" File="/var/log/mysqld.log" Tag="mysqld-error" Severity="notice" Facility="local6")

They arrive on the server tagged nginx-access and mysqld-error, so the template writes them to /var/log/remote/<host>/nginx-access.log and so on.

Did it work?

From a client, send a test message and watch for it on the server:

logger -t logtest "hello from $(hostname)"

On the log server:

ls /var/log/remote/
tail -n 3 /var/log/remote/db01/logtest.log

Then test the queue: stop rsyslog on the server, send a few logger messages from the client, start the server again, and confirm they appear within a few seconds. On the client, ls -la /var/lib/rsyslog/ should show the fwd_logs.* spool files appearing while the server is down and disappearing after.

Gotchas

• rsyslogd: error during parsing file /etc/rsyslog.d/10-remote.conf, on or before line 12: syntax error on token ...: usually a missing closing quote or brace in the RainerScript. rsyslogd -N1 shows the line before you restart into a broken state.
• omfwd: TCPSendBuf error -2027, destruct TCP Connection on the client: the server is not listening, or the firewall is blocking 514/tcp. nc -zv logs.example.internal 514 from the client isolates which.
• All remote messages land under a single hostname directory: the clients are sending their short hostname and it resolves to the same thing, or the template uses %FROMHOST% and the messages pass through a relay. Check hostname on the clients; %HOSTNAME% uses what the client puts in the message.
• imfile inputs produce nothing: rsyslog cannot read the file (nginx logs are adm group readable on Ubuntu; add the syslog user to adm, or on CentOS check SELinux with ausearch -m avc -ts recent).

For a handful of servers, one place to grep is plenty. If it outgrows grep, the same rsyslog server can forward to Elasticsearch with omelasticsearch without touching the clients.