Securing Your Private Docker Registry by Tokens and LDAP

In a recent article (part 1 and part 2), I discussed how to Build A High Availability Private Docker Registry. Below I am going to show you how to add Docker Auth/Tokens, TLS/SSL, LDAP, to your Private Docker Registry. Docker Token Authentication / Authorization (Over SSL) Flow digram

Generating Certificates to be used by the Registry

Before we start modifying the configuration, we need to first generate, request, sign the SSL certificates. we will be using those throughout the configurations below. Note: Below I am generating/using my own CA. feel free to get a certificate from let’s encrypt which provides free SSL certificates for personnel use, or pay for one of that provide public SSL certificates, the configuration process would be similar. First, lets create a file called cert.conf, this file contains additional certificate setting, in our case primarily DNS alt names.

[req] 
default_bits       = 2048
prompt             = no
default_md         = sha256
distinguished_name = dn
req_extensions     = v3_req

[ dn ]
CN                 = coreos.domain.com
O                  = Company1
OU                 = Ops
L                  = New York
ST                 = New York
C                  = US
emailAddress       = admin@domain.com

[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1              = coreos.domain.com
DNS.2              = coreos1.domain.com
DNS.2              = coreos2.domain.com
DNS.2              = coreos3.domain.com
email              = admin@domain.com

Note: As you can see from the cert.conf above, I added additional DNS(FQDN) names in the alt_names section. Next, lets generate the CA public and private keys.

# Create a working directory.
mkdir cert && cd cert

# Generate the CA private key.
openssl genrsa -out ca-key.pem 2048

# Generate the CA certificate.
openssl req -x509 -new -nodes -key ca-key.pem -days 3650 \
-out ca-crt.pem -subj '/C=US/ST=New York/L=New York/O=domain.com/CN=coreos.domain.com'

Next, lets generate the server/client certificate – request, then signed by CA.

# Generate the server/client private key.
openssl genrsa -out coreos-key.pem 2048

# Generate the server/client certificate request.
openssl req -new -key coreos-key.pem \
-newkey rsa:2048 -nodes \
-subj '/' \
-outform pem -out coreos-req.csr -keyout coreos-req.key

# Sign the server/client certificate request.
openssl x509 -req -in coreos-req.csr -CA ca-crt.pem -CAkey ca-key.pem -CAcreateserial \
-out coreos-cert.pem -days 3650 -extensions v3_req -extfile cert.conf

Verify certificate information like FQDN, Alt-names, etc.

# Cert info
openssl x509 -in coreos-cert.pem -noout -text

Tip: Extra bonus, convert the certificate to pk12 format(optional).

# Convert the certificate to pk12
openssl pkcs12 -export -out coreos-cert.pk12 -inkey coreos-key.pem -in coreos-cert.pem -certfile ca-crt.pem

Now, that we have the certificates, we are ready to use them.

Create required SSL files and directorys

In the next step we are creating an SSL directory, this will be used by both Docker Auth and the Docker Registry. Create the SSL directory and copy the appropriate certificates.

mkdir ssl
cp cert/coreos* ssl/.
cp cert/ca-crt.pem ssl/.

Add certificates to systems trust store

Tip: The below steps work on CoreOS, your Linux distro may require a different process, so make sure to check your distros documentation.

# Absolutely needed to work.
cp ssl/coreos-cert.pem /etc/ssl/certs/
# May not be required
cp ssl/ca-crt.pem /etc/ssl/certs/

# Update the OS (CoreOS) certificate 
update-ca-certificates
# Finally, restart the docker demon.
systemctl restart docker

Configuration changes to making Docker work with SSL

Modify the file below and remove the –insecure-registry... make sure to restart the docker demon for the changes to take effect. /etc/systemd/system/docker.service.d/50-insecure-registry.conf

# From
DOCKER_OPTS='--insecure-registry="10.0.2.0/24"'

# To
DOCKER_OPTS=''

Also Modify /var/lib/coreos-install/user_data DOCKER_OPTS to not include –insecure-registry... something like this Environment=DOCKER_OPTS=”

Adding Docker Auth/Tokens in the mix

In the next step, I am going to create a Docker Auth configuration file. The configuration file uses TLS/SSL for communication, and LDAP for authentication/authorization. Tip: You can use Docker Auth/token process with static users, that works great in a small environment. however, if your setup is a bit more complex configuration, especially if you are already using LDAP for authentication you might find the below setup useful. First, lets create the appropriate directorys.

mkdir config

Next, lets create a token configuration file. The file name used is config/auth_config.yml

server:  # Server settings.
  # Address to listen on.
  addr: ":5001"
  # TLS certificate and key.
  certificate: "/ssl/coreos-cert.pem"
  key: "/ssl/coreos-key.pem"
token:  # Settings for the tokens.
  issuer: "Auth-Service"  # Must match issuer in the Registry config.
  expiration: 900
ldap_auth:
  #addr: "10.10.122.250:1389"
  addr: "10.10.122.250:1636"
  tls: always
  insecure_tls_skip_verify: true
  bind_dn: cn=proxy,ou=profile,dc=domain,dc=com
  bind_password_file: /ssl/pass
  base: "o=company1.com,dc=domain,dc=com"
  filter: (&(uid=${account})(objectClass=posixAccount))
  labels:
    groups:
      attribute: gidNumber
      #attribute: title
  #The below will also work
  #groups:
    #attribute: gidNumber
    #parse_cn: false
acl:
  #- match: {account: "yourName"}  
  - match: {"gidNumber": "6000"}
    #actions: ["push", "pull"]
    actions: ["*"]
    comment: "If you are part of Ops group you can do anything."	
  #- match: {"gidNumber": "7000"}
    #actions: ["pull"]
    #comment: "If you are part of the DevOps group you can do anything."

Note: To use the proxy user you will need to create a password file with your LDAP proxy password (I used a file called pass in the ssl directory like this – ssl/pass). As you can see from the above configuration, its using /ssl/coreos-cert.pem and /ssl/coreos-key.pem for SSL communication. Finally, to start the Docker Auth container run the below. Note: Replace with –v=2 –alsologtostderr for debugging.

docker run -d --name dockerauth -p 5001:5001 \
       --network minio_stack_minio_distributed \
       -v $(pwd)/config:/config:ro \
       -v $(pwd)/ssl:/ssl \
       -v /var/log/docker_auth:/logs \
       --restart=always \
       cesanta/docker_auth /config/auth_config.yml
       # Used for debugging
       # cesanta/docker_auth --v=2 --alsologtostderr /config/auth_config.yml

If things work properly and debugging is turned on, you can something like the below by looking on the logs.

docker logs -f 0bdc293d75c9
I0925 16:51:10.772517       1 main.go:174] docker_auth 1.3 build 20170915-010504/master@2cd3699d
I0925 16:51:10.774178       1 main.go:46] Config from /config/auth_config.yml (0 users, 1 ACL static entries)
I0925 16:51:10.774218       1 acl.go:108] Created ACL Authorizer with 1 entries
I0925 16:51:10.774240       1 main.go:72] Cert file: /ssl/coreos-cert.pem
I0925 16:51:10.774257       1 main.go:73] Key file : /ssl/coreos-key.pem
I0925 16:51:10.774718       1 main.go:103] Serving on :5001
[..] snip
I0925 17:07:50.358024       1 server.go:371] Auth request: {usera:***@10.0.2.12:47228 []}
I0925 17:07:50.358135       1 ldap_auth.go:156] DialTLS: starting...10.10.122.250:1636
I0925 17:07:50.384281       1 ldap_auth.go:106] Bind read-only user (DN = cn=proxy,ou=profile,dc=domain,dc=com)
I0925 17:07:50.388201       1 ldap_auth.go:167] search filter is (&(uid=usera)(objectClass=posixAccount))
I0925 17:07:50.388293       1 ldap_auth.go:177] Searching...basedDN:o=company1.com,dc=domain,dc=com, filter:(&(uid=usera)(objectClass=posixAccount))
I0925 17:07:50.434814       1 ldap_auth.go:198] Entry DN = uid=usera,OU=ops,ou=people,o=company1.com,dc=domain,dc=com
I0925 17:07:50.440484       1 ldap_auth.go:106] Bind read-only user (DN = cn=proxy,ou=profile,dc=domain,dc=com)
I0925 17:07:50.444624       1 server.go:217] Authn LDAP usera -> true, map[], 
[..] snip

Now that Docker Auth, LDAP and SSL is working, lets move on to modify the docker registry configuration. Note: After modifying the Docker registry configuration, you will be able to test / troubleshot the configuration including using curl for api/authentication, more about that below.

Adding SSL to the docker registry configuration

Below is an updated docker-registry-config.yml, with SSL/https, the original configuration is available here in part 2.

version: 0.1
log:
  level: debug
  formatter: text
  fields:
    service: registry
    environment: staging
loglevel: debug
storage:
  s3:
    accesskey: minio
    secretkey: minio123
    region: us-east-1
    regionendpoint: http://minio1:9001
    bucket: docker
    encrypt: false
    secure: true
    v4auth: true
    chunksize: 5242880
    rootdirectory: /
  cache:
    blobdescriptor: redis
  delete:
    enabled: true
  maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
    readonly:
      enabled: false
http:
  addr: :5000
  headers:
       X-Content-Type-Options: [nosniff]
  secret: secretword
  tls:
     certificate: /ssl/coreos-cert.pem
     key: /ssl/coreos-key.pem
  debug:
       addr: localhost:5002
redis:
  addr: registryv2-redis:6379
  db: 0
  dialtimeout: 10ms
  readtimeout: 10ms
  writetimeout: 10ms
  pool:
    maxidle: 16
    maxactive: 64
    idletimeout: 300s
auth:
  token:
    realm: https://coreos1.domain.com:5001/auth
    service: "Docker-registry"
    issuer: "Auth-Service"
    rootcertbundle: /ssl/coreos-cert.pem

Finally start your registry container buy just running the below.

docker run -d --name=registry \
       --network minio_stack_minio_distributed \
       -p 5000:5000 \
       -p 5002:5002 \
       -v $(pwd)/ssl:/ssl \
       -v $(pwd)/docker-registry-config.yml:/etc/docker/registry/config.yml \
       --restart="always" \
       registry:latest

Note: One of the things to check if you run into any issues is your proxy settings (if you have any). in some of my tests docker login did not honer the no_proxy keyword causing traffic to try and pass my proxy breaking the docker login.

1. Use tcpdump if verify traffic is coming in
2. Try removing http_proxy and https_proxy from /etc/environment

Always remember to re-start the docker demon with systemctl restart docker for changes to take effect. To make this configuration highly available, you now have two options.

1. Replace coreos1 on each server with the server name like coreos2.
2. Use something like consul, Nginx, traefik to load balance requests by using the name coreos.domain.com, this will not be an issue since the certificate was generated with the alt-names option.

Using/testing Docker registry using Auth, LDAP

We are now reday to test docker login, by using the SSL, Auth+Tokan, LDAP. The first way to see if things work as expected, is by simply using docker login, something like the below.

docker login -u=usera -p=password https://coreos1.domain.com:5000
Login Succeeded

And the registry logs would look something like the below.

time="2017-09-25T19:30:51.791822884Z" level=debug msg="authorizing request" environment=staging go.version=go1.7.6 http.request.host="coreos1.domain.com:5000" http.request.id=a30431c5-b62f-4f04-a344-6d5dda112f27 http.request.method=GET http.request.remoteaddr="10.0.2.12:45890" http.request.uri="/v2/" http.request.useragent="docker/17.06.1-ce go/go1.8.2 git-commit/874a737 kernel/4.13.0-rc7-coreos os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.1-ce \\(linux\\))" instance.id=7938d98e-7622-4362-b2d9-93bce1a8bf85 service=registry version=v2.6.2 
time="2017-09-25T19:30:51.792089951Z" level=warning msg="error authorizing context: authorization token required" environment=staging go.version=go1.7.6 http.request.host="coreos1.domain.com:5000" http.request.id=a30431c5-b62f-4f04-a344-6d5dda112f27 http.request.method=GET http.request.remoteaddr="10.0.2.12:45890" http.request.uri="/v2/" http.request.useragent="docker/17.06.1-ce go/go1.8.2 git-commit/874a737 kernel/4.13.0-rc7-coreos os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.1-ce \\(linux\\))" instance.id=7938d98e-7622-4362-b2d9-93bce1a8bf85 service=registry version=v2.6.2 
10.0.2.12 - - [25/Sep/2017:19:30:51 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/17.06.1-ce go/go1.8.2 git-commit/874a737 kernel/4.13.0-rc7-coreos os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.1-ce \\(linux\\))"
time="2017-09-25T19:30:51.901709148Z" level=debug msg="authorizing request" environment=staging go.version=go1.7.6 http.request.host="coreos1.domain.com:5000" http.request.id=a912ac15-cd62-46c0-ade3-f9d296291e99 http.request.method=GET http.request.remoteaddr="10.0.2.12:45894" http.request.uri="/v2/" http.request.useragent="docker/17.06.1-ce go/go1.8.2 git-commit/874a737 kernel/4.13.0-rc7-coreos os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.1-ce \\(linux\\))" instance.id=7938d98e-7622-4362-b2d9-93bce1a8bf85 service=registry version=v2.6.2 
time="2017-09-25T19:30:51.902020098Z" level=info msg="response completed" environment=staging go.version=go1.7.6 http.request.host="coreos1.domain.com:5000" http.request.id=a912ac15-cd62-46c0-ade3-f9d296291e99 http.request.method=GET http.request.remoteaddr="10.0.2.12:45894" http.request.uri="/v2/" http.request.useragent="docker/17.06.1-ce go/go1.8.2 git-commit/874a737 kernel/4.13.0-rc7-coreos os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.664713ms http.response.status=200 http.response.written=2 instance.id=7938d98e-7622-4362-b2d9-93bce1a8bf85 service=registry version=v2.6.2 
10.0.2.12 - - [25/Sep/2017:19:30:51 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/17.06.1-ce go/go1.8.2 git-commit/874a737 kernel/4.13.0-rc7-coreos os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.1-ce \\(linux\\))"

Now, lets try a simple push then pull, something like the below should work.

coreos2 # docker tag alpine coreos1.domain.com:5000/alpine
coreos2 # docker push coreos1.domain.com:5000/alpine
The push refers to a repository [coreos1.domain.com:5000/alpine]
5bef08742407: Pushed 
latest: digest: sha256:0930dd4cc97ed5771ebe9be9caf3e8dc5341e0b5e32e8fb143394d7dfdfa100e size: 528
coreos2  # docker pull coreos1.domain.com:5000/alpine
Using default tag: latest
latest: Pulling from alpine
Digest: sha256:0930dd4cc97ed5771ebe9be9caf3e8dc5341e0b5e32e8fb143394d7dfdfa100e
Status: Image is up to date for coreos1.domain.com:5000/alpine:latest

Using API with CURL to manipulate the registry

To simply test your configuration by using curl. Something like the configuration below will work.

#!/bin/bash

user='usera'
password='usr_password'

## Working examples
URL="https://coreos1.domain.com:5000/v2/_catalog"
#URL="https://coreos1.domain.com:5000/v2/alpine/manifests/latest"
#URL="https://coreos1.domain.com:5000/v2/hello-world-6.4/manifests/2"
#URL="https://coreos1.domain.com:5000/v2/hello-world-6.4/tags"
#URL="https://coreos1.domain.com:5000/v2/alpine/tags/list"
#URL="https://coreos1.domain.com:5000/v2/alpine/manifests/latest"

auth=$(curl --include -sk $URL |grep ^Www-Authenticate)

realm=$(echo $auth | grep -o '\(realm\)="[^"]*"' | cut -d '"' -f 2)
service=$(echo $auth | grep -o '\(service\)="[^"]*"' | cut -d '"' -f 2|sed 's, ,%20,')
scope=$(echo $auth | grep -o '\(scope\)="[^"]*' | cut -d '"' -f 2)
authurl="$realm?service=$service&scope=$scope"

token=$(curl --user "$user:$password" -s -k "$authurl" |jq -r .token)

#echo auth: $wwwAuth
#echo URL: $registryURL
#echo realm: $realm
#echo service: $service
#echo scope: $scope
#echo authURL: $authURL
#echo token: $token


curl -sk -H "Authorization: Bearer $token" $URL

And the output would look something like the below.

{"repositories":["alpine","hello-world-1.0"]}

I hope you enjoyed reading Securing Your Docker Registry, give it a thumbs up by rating the article or by just providing feedback. You might also like – Articles related to Docker Kubernetes / micro-services.