Prometheus and node_exporter Setup

2019-08-10 · linux, prometheus, monitoring

A disk at 80 percent is a ticket; a disk at 100 percent is an outage. I got fed up with finding out about the second kind from users, so Prometheus is now what I use for monitoring. It pulls metrics over HTTP from small exporters on each host and stores them as time series. node_exporter covers the host basics: CPU, memory, disk, network, filesystems.

I'm using Prometheus 2.11 and node_exporter 0.18 from the release tarballs on Ubuntu 18.04, both as systemd services under unprivileged users. CentOS 7 is the same with useradd and firewall-cmd. You need a Prometheus host with a few gigabytes free for the database, and port 9100 reachable on each monitored host.

node_exporter on each monitored host

cd /tmp
curl -LO https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
tar xzf node_exporter-0.18.1.linux-amd64.tar.gz
sudo cp node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin/
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

Create the unit file /etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus Node Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --collector.filesystem.ignored-mount-points="^/(sys|proc|dev|run|var/lib/docker/.+)($|/)"
Restart=on-failure

[Install]
WantedBy=multi-user.target

The systemd collector adds unit states, which is handy for alerting on failed services. The mount point filter keeps Docker's overlay mounts from producing hundreds of pointless filesystem series.

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://localhost:9100/metrics | head
sudo ufw allow from 10.0.0.10 to any port 9100 proto tcp

Restrict port 9100 to the Prometheus server's address; the metrics are not secret, but there is no reason to publish them.

Prometheus itself

On the monitoring host:

cd /tmp
curl -LO https://github.com/prometheus/prometheus/releases/download/v2.11.1/prometheus-2.11.1.linux-amd64.tar.gz
tar xzf prometheus-2.11.1.linux-amd64.tar.gz
cd prometheus-2.11.1.linux-amd64
sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo cp prometheus promtool /usr/local/bin/
sudo cp -r consoles console_libraries /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Configuration

sudo nano /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets:
          - 'web01.example.com:9100'
          - 'db01.example.com:9100'
        labels:
          env: production

I always run promtool before starting anything; YAML indentation errors are the usual cause of a dead Prometheus:

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
promtool check config /etc/prometheus/prometheus.yml

Running it as a service

/etc/systemd/system/prometheus.service:

[Unit]
Description=Prometheus
After=network-online.target
Wants=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=127.0.0.1:9090
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

I bind the web UI to localhost and reach it through an SSH tunnel or an nginx proxy with authentication; Prometheus has no built-in auth. Thirty days of retention at a 15-second interval for a handful of hosts is a few gigabytes.

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

First look

ssh -L 9090:127.0.0.1:9090 monitor.example.com

Open http://localhost:9090/targets in a browser. Every target should show UP. Then run a query on the Graph page to make sure data is arriving:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

That is CPU utilization per host. Another useful one, free space on the root filesystem in percent:

node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100

Trouble spots

Target shows DOWN with context deadline exceeded. Prometheus cannot reach port 9100: a firewall rule, or node_exporter bound to a different address. curl http://web01.example.com:9100/metrics from the Prometheus host tells you which.
err="opening storage failed: lock DB directory: resource temporarily unavailable". Another Prometheus process is already using /var/lib/prometheus, or a previous one did not exit cleanly. systemctl status prometheus and remove the stale lock file only if you are sure nothing is running.
level=error ... msg="Error loading config" ... yaml: unmarshal errors. Indentation in prometheus.yml. promtool check config points at the line.
node_systemd_* metrics missing. The --collector.systemd flag was not passed, or the user cannot talk to D-Bus. Check journalctl -u node_exporter for couldn't get dbus connection.

Alertmanager and Grafana are next for me. Even before that, a month of per-host history changes how you troubleshoot.