DB-유

Watchtower for Automatic Container Updates on Synology

· synology, docker, updates

Somewhere around container number eight I stopped updating them. The Docker package GUI has no update button: you pull the image, stop and delete the container, and recreate it with the same settings. That's tedious in the GUI and only a bit better with docker-compose. Watchtower does exactly that for you. It runs as a container, polls the registry, and when a new image appears it restarts the old container with identical configuration and the new image.

This is how I run it on DSM 6.2, limited to the containers I choose, at a quiet hour, with an email when something changed. Two conditions: data must live on mounted volumes, since Watchtower recreates containers and anything inside is lost, and images must be pinned to a tag that receives updates. mariadb:10.4 gets 10.4.x patches; mariadb:10.4.12 never changes, so Watchtower finds nothing for it.

Decide the update policy first

Watchtower updates everything it can see by default. That is fine for small utilities and reverse proxies, and a bad idea for databases and anything whose upgrade needs a schema migration. Two ways to control it:

  • Opt-in with labels: start Watchtower with WATCHTOWER_LABEL_ENABLE=true and it only touches containers carrying the label com.centurylinklabs.watchtower.enable=true.
  • Opt-out with labels: leave label mode off and mark the exceptions with com.centurylinklabs.watchtower.enable=false.

I use opt-in. I once let a database auto-update and spent an evening reading migration notes I should have read first, so now anything not labelled stays put until I upgrade it deliberately.

The compose file

version: '2'
services:
  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      TZ: Europe/Berlin
      WATCHTOWER_LABEL_ENABLE: "true"
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_INCLUDE_STOPPED: "false"
      WATCHTOWER_SCHEDULE: "0 0 4 * * 1"
      WATCHTOWER_TIMEOUT: "60s"
      WATCHTOWER_NOTIFICATIONS: email
      WATCHTOWER_NOTIFICATION_EMAIL_FROM: nas@example.com
      WATCHTOWER_NOTIFICATION_EMAIL_TO: admin@example.com
      WATCHTOWER_NOTIFICATION_EMAIL_SERVER: smtp.example.com
      WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT: "587"
      WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER: nas@example.com
      WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD: secret
      WATCHTOWER_NOTIFICATION_EMAIL_DELAY: "5"

Notes:

  • The Docker socket mount is what gives Watchtower control over the daemon. It is equivalent to root on the NAS, which is why I run Watchtower and nothing else with that mount.
  • WATCHTOWER_CLEANUP deletes the old image after a successful update; without it, /volume1/@docker fills up with dangling images.
  • The schedule is a six-field cron expression with seconds first; the one above is Monday 04:00. Weekly is frequent enough for a NAS, and doing it at a fixed time means that if something breaks, I know when.
  • The image is containrrr/watchtower; the older v2tec/watchtower is unmaintained.

Save it as /volume1/docker/watchtower/docker-compose.yml and start it:

cd /volume1/docker/watchtower
sudo docker-compose up -d
sudo docker logs -f watchtower

The first log lines state the schedule and the next run time.

Labelling containers

For a compose-managed service, add the label:

services:
  nginx-proxy:
    image: jwilder/nginx-proxy
    labels:
      com.centurylinklabs.watchtower.enable: "true"

Then docker-compose up -d recreates the container with the label. For a container created from the Docker GUI, the Edit dialog has no label field; recreate it from the shell, or use docker run with --label com.centurylinklabs.watchtower.enable=true. Containers created by the GUI carry a synology-specific label set that Watchtower preserves, so the GUI still shows them after an update.

Containers that depend on each other

If container B depends on A (an application on its database), and both are labelled, Watchtower updates them in an order that respects Docker's --link relationships. It does not know about depends_on in compose. For such pairs I label only the application and update the database on my own schedule, which is what the opt-in policy is for.

Forcing a run to see it work

Force a run without waiting for Monday:

sudo docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower --run-once --label-enable --cleanup

The output lists each labelled container, whether a new image was found, and the actions taken. If an image was updated, docker ps shows the container's creation time as just now and docker images no longer lists the previous digest. The email notification, if configured, arrives within the delay you set.

Check that the recreated container kept its settings:

sudo docker inspect nginx-proxy --format '{{.HostConfig.Binds}} {{.HostConfig.PortBindings}} {{.HostConfig.RestartPolicy.Name}}'

Trouble I've seen

  • Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.38. The Docker daemon in the Synology package is older than Watchtower expects. Set the environment variable DOCKER_API_VERSION=1.38 in the Watchtower service; the exact number to use comes from sudo docker version.
  • Watchtower reports toomanyrequests: You have reached your pull rate limit for Docker Hub. Too many anonymous pulls from your IP. Reduce the number of labelled containers or pass Docker Hub credentials via REPO_USER and REPO_PASS.
  • A container is updated and then loops with a crash. The new image changed its configuration format. Roll back by pinning the previous tag in the compose file and docker-compose up -d, then remove the Watchtower label until you have read the release notes.
  • The GUI shows the container as "unhealthy" or without its icon after an update. Cosmetic; the Docker package caches metadata about images. Restarting the Docker package from Package Center refreshes it.

The weekly email tells me what changed. Containers that hold data still get upgraded when I've read the changelog, not when a registry happens to publish a tag.