Setting Up a Private Docker Registry

2019-04-14 · linux, docker, registry, nginx

I got tired of pushing internal images to Docker Hub and hoping the "private" setting was what I thought it was. Hosting your own takes very little: the registry:2 image is the Docker Distribution software, and one container does the job. What it doesn't do by itself is TLS or authentication I'd put on a network, so I put nginx in front for both.

The end result is registry.example.com: HTTPS only, authenticated pushes and pulls, images on the host's disk. I'm on Ubuntu 18.04 with Docker CE and docker-compose, a certificate from certbot (paths below are Let's Encrypt's), and apache2-utils for htpasswd.

Directories

Start with the layout:

sudo mkdir -p /opt/registry/{data,auth,nginx}
cd /opt/registry

data will hold the image blobs, auth the password file, nginx the proxy configuration.

Users

sudo htpasswd -Bc /opt/registry/auth/htpasswd ci
sudo htpasswd -B /opt/registry/auth/htpasswd yk

-B selects bcrypt, which is the only hash the registry's own htpasswd auth accepts; since nginx will do the authentication here it is not strictly required, but it keeps the file usable by either. Omit -c after the first user or it recreates the file (yes, I wiped the file once adding a second user).

The compose file

version: '3'
services:
  registry:
    image: registry:2.7
    restart: always
    environment:
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
      REGISTRY_STORAGE_DELETE_ENABLED: "true"
    volumes:
      - /opt/registry/data:/var/lib/registry
    ports:
      - "127.0.0.1:5000:5000"

Binding to 127.0.0.1 keeps the unauthenticated registry port off the network; only nginx on the same host can reach it. DELETE_ENABLED allows deleting manifests through the API, which you will want for garbage collection later.

sudo docker-compose up -d
curl -s http://127.0.0.1:5000/v2/

An empty JSON object {} means the registry answers.

nginx in front

Install nginx on the host and create a site:

sudo apt install -y nginx
sudo nano /etc/nginx/sites-available/registry

server {
    listen 443 ssl http2;
    server_name registry.example.com;

    ssl_certificate     /etc/letsencrypt/live/registry.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/registry.example.com/privkey.pem;

    # Image layers can be large.
    client_max_body_size 0;
    chunked_transfer_encoding on;

    location /v2/ {
        # Reject old clients that do not speak the v2 API.
        if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$") {
            return 404;
        }

        auth_basic "Docker Registry";
        auth_basic_user_file /opt/registry/auth/htpasswd;

        add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

        proxy_pass          http://127.0.0.1:5000;
        proxy_set_header    Host              $http_host;
        proxy_set_header    X-Real-IP         $remote_addr;
        proxy_set_header    X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_read_timeout  900;
    }
}

server {
    listen 80;
    server_name registry.example.com;
    return 301 https://$host$request_uri;
}

client_max_body_size 0 is essential; the default 1 MB limit will break every push with a 413. The X-Forwarded-Proto header tells the registry to generate HTTPS URLs in its responses.

sudo ln -s /etc/nginx/sites-available/registry /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Log in and push

On any Docker host:

docker login registry.example.com
docker pull alpine:3.9
docker tag alpine:3.9 registry.example.com/base/alpine:3.9
docker push registry.example.com/base/alpine:3.9

Credentials are saved in ~/.docker/config.json. For CI systems, docker login -u ci --password-stdin reads the password from a pipe so it does not land in shell history.

Looking at what's stored

curl -u yk https://registry.example.com/v2/_catalog
curl -u yk https://registry.example.com/v2/base/alpine/tags/list

The first should list base/alpine, the second its tags. On the registry host, ls /opt/registry/data/docker/registry/v2/repositories/ shows the same names on disk.

Errors

Error response from daemon: Get https://registry.example.com/v2/: x509: certificate signed by unknown authority. The client does not trust the certificate. With Let's Encrypt this means the chain is incomplete; make sure nginx points at fullchain.pem, not cert.pem. For a self-signed certificate, copy it to /etc/docker/certs.d/registry.example.com/ca.crt on each client.
Push fails with 413 Request Entity Too Large. client_max_body_size is missing or set in the wrong block.
unauthorized: authentication required right after a successful login. Usually nginx is not forwarding the Host header, so the registry issues redirect URLs to 127.0.0.1:5000 that the client cannot reach. Check the proxy_set_header Host line.
Disk keeps growing after deleting tags. Deleting a manifest does not remove blobs. Run garbage collection while the registry is stopped or read-only: docker-compose exec registry bin/registry garbage-collect /etc/docker/registry/config.yml.

Back up /opt/registry/data with the rest of the host, and remember the htpasswd entries are credentials like any other.