DB-유

Grafana Dashboards for Server Metrics

· linux, grafana, prometheus, monitoring

"Is it up?" was never my real question. I wanted to know why the disk was filling, whether the CPU really spikes every hour on the hour, and when memory started creeping up. That needs history, so I run Prometheus and Grafana: Prometheus scrapes and stores the numbers, Grafana draws them.

Here it all goes on one CentOS 7 host: node_exporter, Prometheus, then Grafana 6.x from the official repo, plus a ready-made dashboard. Ubuntu 18.04 works too. Budget about 1 GB of free RAM and a few GB of disk for retention. Port 3000 (Grafana) must be reachable from your workstation; 9090 and 9100 only need to be reachable locally. Adding more exporters later doesn't change any of this.

node_exporter

node_exporter exposes CPU, memory, disk, network and filesystem metrics over HTTP. It's one static binary:

sudo useradd --no-create-home --shell /sbin/nologin node_exporter
curl -LO https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
tar xzf node_exporter-0.18.1.linux-amd64.tar.gz
sudo cp node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin/

Create /etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter --collector.systemd
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://127.0.0.1:9100/metrics | head

Repeat on every server you want to watch, opening 9100 only to the Prometheus host: firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=192.168.10.50 port port=9100 protocol=tcp accept'.

Prometheus

sudo useradd --no-create-home --shell /sbin/nologin prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
curl -LO https://github.com/prometheus/prometheus/releases/download/v2.12.0/prometheus-2.12.0.linux-amd64.tar.gz
tar xzf prometheus-2.12.0.linux-amd64.tar.gz
cd prometheus-2.12.0.linux-amd64
sudo cp prometheus promtool /usr/local/bin/
sudo cp -r consoles console_libraries /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Write /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets:
          - 'localhost:9100'
          - 'db01.example.internal:9100'
          - 'web01.example.internal:9100'

And /etc/systemd/system/prometheus.service:

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.listen-address=127.0.0.1:9090
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

I bind to 127.0.0.1 because the Prometheus UI has no authentication and I don't want it on the network. Grafana on the same host reaches it locally.

Grafana

Add the Grafana repository as /etc/yum.repos.d/grafana.repo:

[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key

sudo yum -y install grafana
sudo systemctl enable --now grafana-server
sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --reload

On Ubuntu: apt-get install -y apt-transport-https, add deb https://packages.grafana.com/oss/deb stable main to /etc/apt/sources.list.d/grafana.list, import the key with curl https://packages.grafana.com/gpg.key | apt-key add -, then apt-get update && apt-get install grafana.

Change two settings in /etc/grafana/grafana.ini before exposing it to anyone:

[server]
root_url = https://grafana.example.com/

[users]
allow_sign_up = false

Restart with systemctl restart grafana-server. Put nginx in front with TLS for anything beyond a lab.

Data source and dashboard

Log in at http://server:3000 with admin/admin; it forces a password change. Then:

  1. Configuration → Data Sources → Add data source → Prometheus. URL http://127.0.0.1:9090, Access Server (default). Save & Test should report "Data source is working".
  2. Create (the plus icon) → Import → enter dashboard ID 1860 ("Node Exporter Full") → select the Prometheus data source → Import.

You now have a per-host dashboard with CPU, memory, disk I/O, network and filesystem panels, switchable by the instance variable at the top.

A first alert

Grafana 6 has alerting built into graph panels. Edit a panel (root filesystem usage, say), open the Alert tab and set something like WHEN last() OF query(A, 5m, now) IS ABOVE 90. Then add a notification channel under Alerting → Notification channels (email needs [smtp] set in grafana.ini). For anything complicated Alertmanager is better, but panel alerts cover the basics.

Are the targets healthy?

curl -s http://127.0.0.1:9090/api/v1/targets | python -m json.tool | grep -E '"health"|"instance"'

Every target should say "health": "up", and the dashboard should show data within a minute of the first scrape. Stop node_exporter on one host and confirm its panels go blank and the target turns down. I always do this once, because a dashboard that can't show a failure is worse than none.

Things that went wrong

  • context deadline exceeded on a target: the exporter port is blocked by the remote firewall, or the hostname doesn't resolve from the Prometheus server.
  • Data source is working fails with HTTP Error Bad Gateway: Grafana can't reach Prometheus. Check --web.listen-address and that access mode is Server, not Browser.
  • Panels say No data points while Prometheus has data: the dashboard expects a different job label. Dashboard 1860 uses job="node"; match it in prometheus.yml or edit the dashboard variable.
  • nginx in front of Grafana returns 502 on CentOS with SELinux: setsebool -P httpd_can_network_connect 1.
  • Prometheus eats more disk than expected: lower --storage.tsdb.retention.time or scrape noisy jobs less often. Roughly 1 to 2 bytes per sample after compression.

Next on my list: Alertmanager for routing, exporters for MySQL and nginx, and Grafana dashboards provisioned from files so they end up in version control.