DB-유

HAProxy Basics

· linux, haproxy, load-balancing, ubuntu

The first time I had to patch a web server during business hours, I wished for something in front that would stop sending requests to a dead backend. HAProxy does that for HTTP and for plain TCP (MySQL, Redis, SMTP), and it has been doing it reliably for years. The config is one file and easier to read than most.

Here's HAProxy 2.0 on Ubuntu 20.04 balancing HTTP across web servers with health checks, terminating TLS, fronting a MySQL replica pool over TCP, and exposing the stats page. I assume a load balancer host with a public or VIP address, and backends reachable from it, like 192.168.10.31 and 192.168.10.32 on port 8080. For TLS termination you'll want a combined certificate file, covered below.

Install

sudo apt-get update
sudo apt-get install -y haproxy
haproxy -v
sudo systemctl enable haproxy

Ubuntu 20.04 ships the 2.0 LTS branch. On CentOS 7 the base package is 1.5, which lacks features used below; use the centos-sclo-rh collection (rh-haproxy18) or build from source there.

Global and defaults

Back up the shipped config, then replace /etc/haproxy/haproxy.cfg:

global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    user haproxy
    group haproxy
    daemon
    maxconn 20000
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    option  forwardfor
    option  http-server-close
    timeout connect 5s
    timeout client  50s
    timeout server  50s
    timeout http-request 10s
    errorfile 503 /etc/haproxy/errors/503.http

option forwardfor adds X-Forwarded-For so backends see the client address. The stats socket line lets you administer the running process from the shell (more below).

Frontends and a backend

frontend http_in
    bind *:80
    http-request redirect scheme https unless { ssl_fc }

frontend https_in
    bind *:443 ssl crt /etc/haproxy/certs/example.com.pem alpn h2,http/1.1
    http-request set-header X-Forwarded-Proto https
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /healthz HTTP/1.1\r\nHost:\ example.com
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server web01 192.168.10.31:8080 check
    server web02 192.168.10.32:8080 check
    server web03 192.168.10.33:8080 check backup

balance roundrobin alternates requests. I switch to leastconn for long-lived requests, and source gives stickiness by client IP. The health check polls /healthz every 3 seconds; a server goes down after 3 failures and comes back after 2 successes. web03 is a backup that only gets traffic when every primary is down.

For apps that keep sessions locally and need cookie stickiness instead of IP:

    cookie SRV insert indirect nocache
    server web01 192.168.10.31:8080 check cookie w1
    server web02 192.168.10.32:8080 check cookie w2

The TLS certificate

HAProxy wants the chain and private key in a single PEM file:

sudo mkdir -p /etc/haproxy/certs
sudo cat /etc/letsencrypt/live/example.com/fullchain.pem /etc/letsencrypt/live/example.com/privkey.pem \
    | sudo tee /etc/haproxy/certs/example.com.pem > /dev/null
sudo chmod 600 /etc/haproxy/certs/example.com.pem

Put the same concatenation in the Certbot deploy hook, followed by systemctl reload haproxy, so renewals propagate. I forgot this once and found out when the old cert expired.

MySQL replicas over TCP

HAProxy isn't limited to HTTP. To spread read queries across replicas:

listen mysql_ro
    bind 127.0.0.1:3307
    mode tcp
    option tcplog
    balance leastconn
    option mysql-check user haproxy_check
    server db02 192.168.10.42:3306 check
    server db03 192.168.10.43:3306 check

mysql-check needs a user on each replica with no password and no privileges: CREATE USER 'haproxy_check'@'192.168.10.20';. Applications connect to port 3307 on the balancer for reads.

Stats page, then validate

listen stats
    bind 127.0.0.1:8404
    mode http
    stats enable
    stats uri /
    stats refresh 10s
    stats admin if TRUE

I bind it to localhost (or protect it with stats auth user:pass and a firewall rule) because admin mode lets anyone disable servers. Then validate and start:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl restart haproxy
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

-c prints Configuration file is valid or the line number of the mistake. Once running, systemctl reload haproxy applies changes without dropping connections.

Watching it work

curl -I https://example.com/
for i in 1 2 3 4; do curl -s https://example.com/whoami; done
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | cut -d, -f1,2,18 | column -s, -t

The loop should alternate between backends if the app reports its hostname. The socket command lists every server's status (UP, DOWN, MAINT). Stop the app on web01 and watch it flip to DOWN within ten seconds while requests keep succeeding. To drain a server for maintenance:

echo "set server web_servers/web01 state drain" | sudo socat stdio /run/haproxy/admin.sock

Common failures

  • 503 Service Unavailable on every request: all backends are down. Check show stat; usually the health check URL returns 404 or the Host header check fails.
  • [ALERT] ... 'bind *:443' : unable to load SSL private key from PEM file: the combined PEM lacks the key, or the order is wrong (chain first, then key).
  • Backends log every request as coming from the balancer: option forwardfor is set but the backend doesn't trust it. In nginx, use the real-IP module (set_real_ip_from 192.168.10.20; and real_ip_header X-Forwarded-For;).
  • Server web_servers/web01 is DOWN, reason: Layer7 wrong status, code: 301: the backend redirects the health check to HTTPS. Point httpchk at a path that returns 200 directly, or set the Host header the app expects.
  • haproxy.service: Failed with result 'exit-code' after a reload with a bad config: run haproxy -c -f before every reload. A systemd ExecReload refuses a bad file, but a restart leaves you with nothing running.

Next for me would be two HAProxy nodes sharing a virtual IP with keepalived. A single balancer is a single point of failure, which is a bit funny. After that, shipping the logs somewhere useful, since httplog gives per-request timings for free.