Load Balancing with nginx Upstreams

One app server is one point of failure and a hard ceiling on capacity, and I got tired of doing maintenance at 2 a.m. because of it. The cheap fix is two copies of the app with nginx in front. The upstream block gives you several balancing methods, passive health checks, and a way to pull a backend out without dropping connections.

I'll use nginx 1.18 on Ubuntu 20.04 as reverse proxy and balancer for two backends, 10.0.1.11 and 10.0.1.12 on port 8080. Sessions need to live somewhere shared (database, Redis, signed cookies). If your app keeps them in local memory, read the ip_hash part before going live.

The upstream group

Upstreams live in the http context, so put them in their own file that gets included before the server blocks. On Ubuntu, /etc/nginx/conf.d/*.conf is included by the default nginx.conf:

# /etc/nginx/conf.d/upstream-app.conf
upstream app_backend {
    server 10.0.1.11:8080 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.1.12:8080 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.1.13:8080 backup;

    keepalive 32;
}

Requests are distributed round-robin by default. max_fails and fail_timeout implement passive health checking: after three failed attempts within 30 seconds, the server is skipped for 30 seconds and then tried again. The backup server only receives traffic when every non-backup server is unavailable. keepalive 32 keeps a pool of idle connections open to the backends, which noticeably reduces latency for short requests.

Picking a balancing method

Round-robin is right when requests are similar in cost and the application is stateless. The alternatives, each a single directive at the top of the upstream block:

least_conn;                  # send to the server with the fewest active connections
ip_hash;                     # same client IP always goes to the same server
hash $request_uri consistent; # same URL always goes to the same server

I default to least_conn for anything with a mix of slow and fast requests. ip_hash gives sticky sessions for applications that keep state locally, at the cost of uneven distribution behind corporate NATs. hash ... consistent is useful when backends cache per-URL content.

Proxying to it

# /etc/nginx/sites-available/app.example.com
server {
    listen 443 ssl http2;
    server_name app.example.com;

    ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;

    location / {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 5s;
        proxy_read_timeout    60s;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
    }
}

proxy_http_version 1.1 together with an empty Connection header is required for the keepalive pool to work. proxy_next_upstream decides which failures cause nginx to retry the request on another backend; I include 502 and 503 but deliberately not 500, since a 500 is usually the application's fault and will repeat on the next server. Never include non-idempotent methods in retry logic unless you know the application tolerates it; by default nginx does not retry POST requests that already got a response.

sudo ln -s /etc/nginx/sites-available/app.example.com /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Log the backend

I always add the upstream address and timing to the access log. It's the only way to see the distribution and spot a slow node:

log_format upstreamlog '$remote_addr - [$time_local] "$request" $status '
                       'upstream=$upstream_addr rt=$request_time urt=$upstream_response_time';
access_log /var/log/nginx/app-access.log upstreamlog;

$upstream_addr shows every server tried for the request, separated by commas, so a retried request is visible.

Draining a node

Mark it down in the upstream block and reload; existing connections finish, new ones go elsewhere:

server 10.0.1.12:8080 down;

sudo nginx -t && sudo systemctl reload nginx

Reload is graceful: old worker processes finish their requests before exiting.

Trying it

for i in $(seq 1 10); do curl -s -o /dev/null -w '%{http_code}\n' https://app.example.com/; done
sudo tail -n 10 /var/log/nginx/app-access.log | awk '{print $NF, $(NF-2)}'

The log should show upstream= alternating between the two backends. Then stop the application on one backend and repeat: the requests should all succeed, the log should show the surviving backend, and /var/log/nginx/error.log should contain upstream server temporarily disabled while connecting to upstream.

When it goes wrong

• 502 Bad Gateway for every request, error log says connect() failed (111: Connection refused) while connecting to upstream. The backends are not listening on the address and port in the upstream block, or a firewall on them blocks the nginx host. Check with curl http://10.0.1.11:8080/ from the nginx host.
• no live upstreams while connecting to upstream. Every server is marked failed. The backends are unhealthy or the max_fails window is too aggressive for a slow application; raise fail_timeout or fix the app.
• Users get logged out at random. The application stores sessions locally and round-robin is sending them to the other node. Switch to ip_hash as a stopgap and move sessions to shared storage properly.
• nginx: [emerg] "upstream" directive is not allowed here. The block is inside a server context. It must be in http, which is why it belongs in conf.d/.

Active health checks and cookie-based stickiness are nginx Plus features. For two to five nodes I've never really missed them.