DB-유

Server Monitoring with Netdata

· linux, monitoring, netdata

For a single server or a handful of them, a full Prometheus and Grafana setup is more infrastructure than the thing you are monitoring. Netdata fills the gap: one agent per host, per-second metrics for CPU, memory, disks, network, and hundreds of applications it detects automatically, and a web dashboard that needs no configuration at all. It has been my first install on any new box for a while now, mainly because it shows MySQL and nginx metrics out of the box the moment it finds them.

Below is the install on CentOS 7 and Ubuntu 18.04, nginx with basic auth in front, the MySQL collector, and an email alarm. You'll want curl, about 100 MB of RAM for the agent, and nginx if the dashboard will be reachable beyond localhost.

Install with the kickstart script

The distribution packages are usually old. Netdata's own installer builds from source or uses a static binary, and sets up nightly updates:

bash <(curl -Ss https://my-netdata.io/kickstart.sh)

I read the script before piping it into bash, and you should too; it installs build dependencies with yum or apt, compiles, and installs to /usr/sbin/netdata with configuration in /etc/netdata/ and the web files under /usr/share/netdata/web/. The service is registered with systemd:

systemctl status netdata
ss -tlnp | grep 19999

Netdata listens on port 19999. On CentOS 7 the firewall blocks it by default, which is what we want; we will go through nginx.

Bind it to localhost first

By default Netdata binds to all interfaces, and there is no authentication built in. Fix that before anything else. The configuration is edited with the edit-config helper, which copies the stock file from /usr/lib/netdata/conf.d/ into /etc/netdata/ the first time:

cd /etc/netdata
sudo ./edit-config netdata.conf

[global]
    history = 3600
    memory mode = save

[web]
    bind to = 127.0.0.1

history = 3600 keeps one hour at one-second resolution in RAM; each 1000 metrics costs roughly 15 MB per hour, so size to your memory. memory mode = save writes the metrics to disk on shutdown so a restart does not lose the graphs. Restart the service afterwards:

sudo systemctl restart netdata

nginx in front, with basic auth

sudo apt install apache2-utils      # Ubuntu; provides htpasswd
sudo yum install httpd-tools        # CentOS 7
sudo htpasswd -c /etc/nginx/.htpasswd-netdata admin

Add a location to the server block for the host:

location /netdata/ {
    auth_basic "Netdata";
    auth_basic_user_file /etc/nginx/.htpasswd-netdata;

    proxy_pass http://127.0.0.1:19999/;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header Connection "";
}

The trailing slash on proxy_pass strips the /netdata/ prefix so the dashboard's relative URLs resolve. Then nginx -t && systemctl reload nginx and browse to https://your-host/netdata/.

The MySQL collector

Netdata's Python-based collectors auto-detect services but need credentials to read status variables. Create a read-only user:

CREATE USER 'netdata'@'localhost';
GRANT USAGE, REPLICATION CLIENT, PROCESS ON *.* TO 'netdata'@'localhost';
FLUSH PRIVILEGES;

Then configure the collector:

sudo ./edit-config python.d/mysql.conf

local:
  name: 'local'
  socket: '/var/run/mysqld/mysqld.sock'
  user: 'netdata'

On CentOS 7 the socket is /var/lib/mysql/mysql.sock. Restart Netdata and a "mysql local" section appears in the dashboard with queries per second, InnoDB buffer pool usage, connections, and slow queries. Enabling the nginx collector is the same idea: turn on stub_status in a localhost-only location and point python.d/nginx.conf at it.

Email alarms

Netdata ships with alarms for disk space, load, RAM, and MySQL replication lag. They fire into the dashboard by default and nowhere else. To get email, edit the notification config:

sudo ./edit-config health_alarm_notify.conf

SEND_EMAIL="YES"
DEFAULT_RECIPIENT_EMAIL="ops@example.com"

This uses the local sendmail binary, so the host needs a working MTA (postfix relaying to your mail server is enough). Test it as the netdata user:

sudo -u netdata /usr/libexec/netdata/plugins.d/alarm-notify.sh test

Is it all working?

Load the dashboard and confirm the sections for system, disks, network, and MySQL appear. Then check the API directly and look for collector errors:

curl -s http://127.0.0.1:19999/api/v1/info | head -c 300
curl -s http://127.0.0.1:19999/api/v1/alarms | head -c 300
sudo tail -50 /var/log/netdata/error.log

Generate some load with sysbench or a big SELECT and watch the charts move within a second. Fill a filesystem to 90% on a test box and confirm the disk space alarm email arrives.

Problems I ran into

  • Dashboard behind nginx loads but charts are blank: proxy_set_header Connection "" or proxy_http_version 1.1 is missing, so the streaming responses are being buffered. Add both.
  • mysql: [ERROR] mysql local: cannot connect to MySQL in error.log: wrong socket path, or the netdata OS user cannot read the socket. Use host: 127.0.0.1 and port: 3306 instead if the socket permissions are restrictive.
  • CPU usage of the netdata process itself is high: the Python collectors with one-second update every can add up. Raise update every to 5 in the collector's yaml, or disable ones you do not need in python.d.conf.
  • Nightly auto-update broke the install on CentOS 7 due to a missing dependency: remove /etc/cron.daily/netdata-updater and update manually with netdata-updater.sh when convenient.

An hour of history in RAM won't do capacity planning for you. For "what was this server doing five minutes ago when the alert fired", though, nothing else I've tried is faster.