Synology Log Center and Syslog Server

Chasing a problem across three servers with three terminal windows and a lot of scrolling is what made me set up central logging. A Synology running Log Center is a fine syslog server for a small network: always on, plenty of disk, with search, archiving and alerts on matching messages. It's also where DSM's own connection logs live, so the NAS's file access history ends up in the same place.

I'll turn Log Center into a syslog receiver on DSM 6.2, point a CentOS 7 and an Ubuntu 18.04 host at it with rsyslog, then add archiving and one alert rule. Log Center is preinstalled on most DSM 6 units. You need TCP or UDP 514 reachable from the senders (TCP 6514 for TLS) and a shared folder such as syslog to hold the log database.

The syslog receiver

Log Center → Log Receiving → Create:

Name:               linux-hosts
Log format:         IETF (RFC 5424)
Transfer protocol:  TCP
Port:               514
Enable secure connection (SSL): off for now
Save to:            syslog (shared folder)

BSD format (RFC 3164) is what older devices and most routers speak; IETF is what a current rsyslog sends with the right template and carries structured data and precise timestamps. You can create multiple receivers on different ports, one per format, and Log Center tags the source of each.

I stick to TCP over UDP because messages aren't silently dropped when the NAS is busy. Make sure the DSM firewall allows the port: Control Panel → Security → Firewall → Edit Rules → add the built-in application "Log Center" or a custom rule for TCP 514.

rsyslog on the senders

On CentOS 7, rsyslog is already the default logger. Create /etc/rsyslog.d/50-synology.conf:

$ActionQueueType LinkedList
$ActionQueueFileName synology
$ActionQueueMaxDiskSpace 100m
$ActionQueueSaveOnShutdown on
$ActionResumeRetryCount -1
*.* @@10.20.30.5:514;RSYSLOG_SyslogProtocol23Format

The double @@ means TCP; a single @ is UDP. RSYSLOG_SyslogProtocol23Format is rsyslog's built-in RFC 5424 template, matching the IETF receiver. The queue directives keep messages on local disk while the NAS is unreachable and replay them afterwards.

sudo rsyslogd -N1
sudo systemctl restart rsyslog

Ubuntu 18.04 ships rsyslog too; the same file goes in /etc/rsyslog.d/ and the same restart applies. On a host running journald only, enable forwarding in /etc/systemd/journald.conf with ForwardToSyslog=yes and install rsyslog.

If the host also runs nginx or MariaDB, forward their files as well with the imfile module:

module(load="imfile")
input(type="imfile" File="/var/log/nginx/error.log" Tag="nginx-error" Severity="error" Facility="local1")

Routers and switches

Most routers speak BSD syslog over UDP only. Create a second receiver in Log Center with format BSD, protocol UDP, port 5514, and point the router at that. Keeping the two formats on separate ports avoids Log Center misparsing timestamps.

Archiving

Log Center → Archive Settings:

Archive when logs exceed:  10,000,000 entries
or every:                  30 days
Destination:               syslog/archive
Rotate archives, keep:     12

Archives are compressed text files, one per rotation, which you can grep directly from the shell or include in Hyper Backup. The live database stays small enough that the search page stays responsive.

An alert rule

Log Center → Notifications → Create:

Name:      ssh-failures
Keyword:   Failed password
Match on:  Message
Count:     10 in 5 minutes
Send by:   email

Any message containing the keyword across all senders counts towards the threshold. I keep a handful of these: Failed password, segfault, Out of memory, mdadm and I/O error. Notifications go through the channels configured in Control Panel → Notification.

Sending a test message

From each sender:

logger -p local0.info "syslog test from $(hostname)"

Within a second or two, Log Center → Log Search shows the message with the hostname, facility local0 and the right timestamp. Filter by host in the left pane to confirm both machines are reporting. Then check that DSM's own logs are still arriving too: Log Center → Overview shows counts for Connection, File Transfer and System log types.

To see traffic arriving on the NAS side:

sudo netstat -tlnp | grep 514
sudo tcpdump -ni eth0 tcp port 514 -c 5

Odd behaviour I've run into

• Messages arrive with timestamps several hours off. The sender is in a different time zone or using BSD format on the IETF receiver. Move it to the BSD receiver or fix the template; and make sure all hosts sync NTP.
• rsyslogd: omfwd: TCPSendBuf error -2027, destruct TCP Connection to 10.20.30.5:514 on the sender. The NAS closed the connection, usually because the DSM firewall dropped it or the receiver was recreated on another port. Check the rule and the receiver definition.
• Log Center shows the hostname as the IP address. The sender's hostname is not fully qualified, or you have $PreserveFQDN off in rsyslog.conf. Set it on, or accept the IP and use the host filter by address.
• Search becomes slow after a few months. The archive threshold is too high for the volume of logs; lower the entry count so the live database rotates more often, and search archives from the shell with zgrep.

It won't replace an ELK stack or Graylog in a data centre. For a dozen hosts, one search box and one place for alerts is all I want. The syslog share is small enough to sit in the nightly Hyper Backup.