DB-유

Auto-Renewing Let's Encrypt Certificates

· linux, tls, letsencrypt, nginx

Let's Encrypt certificates are valid for 90 days. That is deliberate: it forces you to automate renewal, because nobody is going to remember to do it by hand every three months. The first time one of my internal sites went red in the browser because I had issued the certificate manually with certbot certonly and then forgotten about it, I decided to sort out renewal properly on every host.

What follows is my routine on Ubuntu 16.04: certbot, one nginx site, unattended renewal, and the reload hook that most people miss. It assumes a public server with DNS already pointing at it and nginx serving the hostname over HTTP. Port 80 has to be reachable even if the site is HTTPS-only, because the HTTP-01 challenge needs it.

Installing certbot

The letsencrypt package in stock xenial is old (0.4.1), so I use the certbot PPA.

sudo apt install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt update
sudo apt install python-certbot-nginx

python-certbot-nginx pulls in certbot itself plus the nginx plugin that can edit server blocks for you. Check the version afterwards with certbot --version; you want 0.14 or newer.

Issuing the certificate

Let the nginx plugin do the work. It reads your existing server blocks, matches the server_name, and adds the listen 443 ssl configuration:

sudo certbot --nginx -d example.com -d www.example.com

You will be asked for an email address (used for expiry warnings), to agree to the terms, and whether to redirect HTTP to HTTPS. I choose redirect. When it finishes, the certificate files live under /etc/letsencrypt/live/example.com/:

  • fullchain.pem - the certificate plus intermediate, what nginx wants for ssl_certificate
  • privkey.pem - the private key, for ssl_certificate_key

If you would rather manage the nginx configuration yourself, use certbot certonly --webroot -w /var/www/example.com/html -d example.com and reference the files above manually.

How renewal is scheduled

The PPA package installs both a cron job and a systemd timer. Look at them:

cat /etc/cron.d/certbot
systemctl list-timers | grep certbot

The cron file runs certbot -q renew twice a day, but only if systemd is not present; on a systemd system the certbot.timer unit takes over. Either way, certbot renew inspects every certificate under /etc/letsencrypt/renewal/ and renews the ones with fewer than 30 days left. Running it every 12 hours is intentional: if the Let's Encrypt API has an outage during one run, the next one catches up.

If you installed certbot some other way (for example the certbot-auto script) and have no timer, add a cron entry yourself:

sudo crontab -e

17 3,15 * * * /usr/bin/certbot renew --quiet

Pick odd minutes rather than 0 so you are not hitting the API at the same moment as everyone else.

The reload hook

Renewing the file on disk does nothing until nginx re-reads it. This is the step I forgot the first time, and the site kept serving the old certificate until it expired. The nginx plugin handles this when it installed the certificate, but for certonly certificates you must tell certbot what to do afterwards. The cleanest place is the per-certificate renewal config, /etc/letsencrypt/renewal/example.com.conf. Add under [renewalparams]:

renew_hook = systemctl reload nginx

renew_hook runs only when a certificate was actually renewed, unlike post_hook, which runs at the end of every certbot renew invocation even when nothing changed. For a server running several services off the same certificate, a small script works:

renew_hook = /usr/local/sbin/reload-tls.sh

with the script doing systemctl reload nginx and, say, systemctl restart dovecot.

Prove it works

Do a dry run against the staging environment. It exercises the whole renewal path, including hooks, without touching your real certificate or counting against rate limits:

sudo certbot renew --dry-run

You want to see Congratulations, all renewals succeeded at the end. Then check the expiry date nginx is actually presenting, which is the real test of whether the reload hook is doing its job:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -dates

Compare notAfter with the file on disk: openssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem -noout -enddate. They should match. Also confirm certbot's own view with certbot certificates.

Errors I've run into

  • Failed authorization procedure ... urn:acme:error:connection or unauthorized: Let's Encrypt could not reach http://example.com/.well-known/acme-challenge/ on port 80. Check DNS, the firewall (ufw status), and that the HTTP server block still exists if you redirected everything to HTTPS. The redirect is fine as long as the redirect target also serves the challenge path.
  • Attempting to renew cert from /etc/letsencrypt/renewal/example.com.conf produced an unexpected error: The nginx plugin is not working: usually nginx is not running or nginx -t fails. Fix the config and rerun certbot renew.
  • Certificate renewed but the browser still shows the old one: the reload hook is missing or failed. Check /var/log/letsencrypt/letsencrypt.log for the hook output, and reload nginx manually.
  • too many certificates already issued for exact set of domains: you hit the rate limit (5 duplicates per week) by re-issuing repeatedly during testing. Always use --dry-run or --staging while experimenting.

I keep the expiry-warning emails enabled as a backstop. If one ever arrives, the automation broke, and /var/log/letsencrypt/ will say why.