DB-유

Zero-Downtime Deployments with nginx and systemd

· linux, nginx, systemd, deployment

My deploys used to be git pull && systemctl restart app, which drops every request that lands during the few seconds of restart. Nobody cares on an internal tool. On a public API it shows up as errors in every client's logs, and I got tired of explaining that. You don't need Kubernetes to fix it. With nginx already in front and systemd already managing the app, run two instances, move traffic between them, and there's never a moment with nothing listening.

This is a blue/green pair using a systemd template unit, an nginx upstream that knows both, and a short deploy script. The example is Node.js, but the pattern is the same for Python, Go, Java, or anything that listens on a port. I assume nginx already proxies to the app on one port, the app reads its port from an environment variable, and there's a /healthz endpoint that returns 200 only when it's ready to serve.

A template unit

A template unit has an @ in its name and receives the part after it as %i. Replace /etc/systemd/system/myapp.service with /etc/systemd/system/myapp@.service:

[Unit]
Description=MyApp instance on port %i
After=network.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/srv/myapp/current
Environment=NODE_ENV=production
Environment=PORT=%i
ExecStart=/usr/bin/node /srv/myapp/current/server.js
Restart=on-failure
RestartSec=3
KillSignal=SIGTERM
TimeoutStopSec=30
SyslogIdentifier=myapp-%i

[Install]
WantedBy=multi-user.target

WorkingDirectory points at a current symlink that the deploy script moves. TimeoutStopSec=30 gives in-flight requests time to finish on stop. Make sure the app handles SIGTERM by closing its listener and finishing outstanding requests, not exiting on the spot.

sudo systemctl daemon-reload
sudo systemctl enable --now myapp@3000 myapp@3001
systemctl status 'myapp@*'

Releases on disk

sudo mkdir -p /srv/myapp/releases
sudo ln -sfn /srv/myapp/releases/20201209-1 /srv/myapp/current

Each deploy unpacks into a new timestamped directory under releases/ and re-points current. Both instances resolve current only at start, so changing the symlink doesn't touch running processes until each restarts. That's what lets you restart them one at a time.

nginx upstream

In /etc/nginx/conf.d/myapp.conf, list both instances with passive health checks and fast failover:

upstream myapp_backend {
    server 127.0.0.1:3000 max_fails=1 fail_timeout=5s;
    server 127.0.0.1:3001 max_fails=1 fail_timeout=5s;
    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name app.example.com;
    # ssl_* directives omitted

    location / {
        proxy_pass http://myapp_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_connect_timeout 2s;
    }
}

proxy_next_upstream is the line that matters. If the instance nginx picked refuses the connection or returns 502/503 because it's restarting, nginx retries on the other instance instead of handing the error to the client. max_fails=1 fail_timeout=5s pulls a failed instance out of rotation for five seconds so later requests don't pay the retry cost.

sudo nginx -t && sudo systemctl reload nginx

nginx's own reload is already zero-downtime: the master starts new workers with the new config and lets old workers finish their connections.

The deploy script

Save as /usr/local/bin/deploy-myapp.sh:

#!/usr/bin/env bash
set -Eeuo pipefail

APP_DIR=/srv/myapp
RELEASE=${1:?usage: deploy-myapp.sh <release-dir-name>}
PORTS=(3000 3001)
HEALTH_PATH=/healthz

wait_healthy() {
    local port=$1
    for _ in $(seq 1 30); do
        if curl -sf "http://127.0.0.1:${port}${HEALTH_PATH}" >/dev/null; then
            return 0
        fi
        sleep 1
    done
    echo "instance on port ${port} did not become healthy" >&2
    return 1
}

[ -d "$APP_DIR/releases/$RELEASE" ] || { echo "release not found" >&2; exit 1; }
ln -sfn "$APP_DIR/releases/$RELEASE" "$APP_DIR/current"

for port in "${PORTS[@]}"; do
    echo "restarting myapp@${port}"
    systemctl restart "myapp@${port}"
    wait_healthy "$port"
done

echo "deployed $RELEASE"
ls -dt "$APP_DIR"/releases/* | tail -n +6 | xargs -r rm -rf

The loop restarts one instance, waits until it answers on /healthz, then moves on. While one restarts, nginx sends everything to the other. The last line keeps five releases, so a rollback is just deploy-myapp.sh <previous-release>.

Shrinking the gap

systemctl restart stops the old process before starting the new one, so for a second or two that port has nothing listening. nginx copes, but the retries add latency. Two refinements if that matters:

  • Use systemctl reload-or-restart with an ExecReload that tells the app to re-exec itself, if the framework supports it (for example a cluster-mode master that forks new workers).
  • Or use systemd socket activation: myapp@.socket holds the listening socket open, and the kernel queues connections while the service restarts, so nothing is refused at all. That requires the app to accept an inherited socket (LISTEN_FDS), which Node and Go can do with a small change.

I've never needed either. Plain restart plus nginx retries has been indistinguishable from zero downtime for me.

Prove it

Run continuous load and deploy in the middle of it:

ab -n 20000 -c 20 https://app.example.com/ > ab.txt &
sudo /usr/local/bin/deploy-myapp.sh 20201209-2
wait; grep -E 'Failed requests|Non-2xx' ab.txt

Both counts should be zero. Also confirm the instances point at the new release:

for p in 3000 3001; do sudo ls -l /proc/$(systemctl show -p MainPID --value myapp@$p)/cwd; done

Things that bit me

  • 502 during deploy even though the other instance is up: proxy_next_upstream lacks http_502, or proxy_next_upstream_tries is 1. Also check the app returns 503, not 200, from /healthz while starting.
  • wait_healthy times out every deploy: the app needs more than 30 seconds to start, or /healthz needs a Host header. Raise the loop count or pass -H 'Host: app.example.com' to curl.
  • systemctl restart myapp@3000 hangs for the full TimeoutStopSec: the app ignores SIGTERM. Add a handler that calls server.close() and exits when connections drain.
  • Both instances still run old code: WorkingDirectory resolves the symlink at start, so they need the restart. Check the script reached the loop (set -e may have exited early on a missing release).
  • nginx logs upstream server temporarily disabled while connecting to upstream: expected in the restart window; that's max_fails working. Outside deploys it means an instance is crashing.

More instances, or instances on other hosts, are just more server lines. Cheap rollbacks are the real payoff; I'll roll back at any hour when it's one command.