High Availability For A Private Docker Registry – Architecture
This is part two on how to configure
Docker Registry in a High Availability configuration.
To read part one click here.
High Availability Docker Registry Digram
Docker Swarm Configuration on CoreOS
To configure
Docker Swarm you will need minimum 3 nodes, below I will use the names/ip’s listed in
part 1.
To initialize the
Docker Swarm cluster, just run the below on the first node
coreos1.
# Use if the server has multiple interfaces
# docker swarm init --advertise-addr 10.0.2.11
docker swarm init
Next, run the below, on the two other nodes (coreos2 and coreos3).
Replace the token with your
Docker Swarm token.
docker swarm join \
--token SWMTKN-1-58uz09763fitew2samqtowgse75dc9px7gxk2qokf5uc5mu9f2-20ve9z6nj34o202ybf9tiqk3j \
10.0.2.11:2377
For fail-over to work properly, in recent versions, run the below on node2 and node3, this is needed to allow the other two node become manger in case the first node fails.
docker node promote coreos2
docker node promote coreos3
Now, lets verify the cluster functionality.
docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
8c6qmlmo9uym11obvyo3irgib coreos1 Ready Active Reachable
ilmw12fa1kdyzsi5nrmfpi8bk coreos3 Ready Active Reachable
ykafu6ww70nbklvrvgpc5qoiy * coreos2 Ready Active Leader
Next, create a
Docker Swarm network.
docker network create \
--driver overlay \
--subnet 10.20.1.0/24 \
--opt encrypted \
services
Next, create
Docker Swarm service.
docker service create \
--replicas 2 \
--name nginx \
--network services \
--publish 80:80 \
nginx
Now, verify the cluster network and service.
docker service ls
docker service ps nginx
docker network ls
Docker helpful examples
A
Docker Swarm busybox example.
docker service create \
--name busybox \
--network services \
busybox \
sleep 3000
Verify the ps busybox
Docker Swarm.
docker service ps busybox
Running the busybox as part of
Docker Swarm.
docker exec -it busybox.1.4czfvm7771qy8lmbiz2bv4nbwdocker /bin/sh
docker service inspect --pretty nginx
Scale the busybox service as part of
Docker Swarm.
docker service scale nginx=3
docker service ps nginx
Destroying the Docker Swarm cluster
Single node to leave the cluster.
docker node demote coreos2
docker node demote coreos3
Last node to leave the cluster – on coreos1.
docker swarm leave --force
Installing Minio Object Store
To install/configure the
Docker Registry in a High Availability architecture, we need to configure the back-end storage to be shared. One of the many options available is using
AWS S3 as the back-end storage.
One of the free open source vendors created an
S3 alike option. in my configuration I will be using the
Minio back-end object store(available from minio.io), I found it to be an awesome option for an
S3 alike back-end storage.
Note: Just for extras, I will also use an
NFS back end shared directory store, helping the fail-over file system always available.
One of the requirements for a
Distributed Minio is running a minimum of a 4 node cluster (i.e. 4 docker instances), since I started my configuration with only 4 VBox VM’s I will continue that route(with only 3 nodes) but I will configure 4 Docker instances to meet the requirements.
Note: Idle in a production environment you should have 4 physical nodes, or there are a few other options like two nodes with 2 or 4 physical disks each, for the full requirements, please visit
the Distributed Minio Documentation.
Enough talk, lets get to work.
A few notes on the Minio setup.
- Minio like S3 requiters an access and secret key. I am using minio and minio123 as these keys, please make sure to change them in your environment
- Minio by default listens to port 9000, in order not to have any issues if more then one Minio instance land on the same host you should make sure each node uses there own port, I used 9001, 9002, and so on
- I used directory /mnt/docker_reg/minio on all Minio instances. for instances 1 I will be using /mnt/docker_reg/minio/f1 and for instances 2 it will be /mnt/docker_reg/minio/f2 and so on..
- Also, normally /mnt/docker_reg/minio/ would be a local mount, for good speed an SSD would make sense. in my case I used /mnt/docker_reg/minio/ as an NFS client mount, then f1, f2,.. are sub directory’s
- Lastly, I ran in to issues with the official distributed Minio image. until this bug-fix is integrated in the official Minio image I am using the nitisht/minio:swarm4 image which works great
Update: In my most recent testing the minio/minio:edge Docker image also worked without any issues, I updated the docker-compose-secrets.yaml.
One additional note: All Minio participating need to have their system clock synced, this was add as part of the cloud-config in
part 1, if you are not using a cloud-config(CoreOS), you can create a schedule job with systemd something like the below.
Create two files in
/etc/systemd/system/, something like the below.
date.service file.
[Unit]
Description=Updates date and time
[Service]
Type=oneshot
ExecStart=/usr/bin/sh -c '/usr/sbin/ntpdate time.bnh.com'
date.timer file.
[Unit]
Description=Run date.service every 1 day
[Timer]
OnBootSec=15min
OnCalendar=*-*-* 00:00:00
[Install]
WantedBy=multi-user.target
Now lets enable and load this in the system
systemctl enable date.timer
systemctl start date.timer
Verify the timer working by running the below. out should look something like the below.
systemctl list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2017-09-27 19:19:10 EDT 4h 0min left Tue 2017-09-26 17:12:48 EDT 22h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.ser
Thu 2017-09-28 00:00:00 EDT 8h left Wed 2017-09-27 14:51:49 EDT 27min ago date.timer date.service
Thu 2017-09-28 00:00:00 EDT 8h left Wed 2017-09-27 00:00:00 EDT 15h ago logrotate.timer logrotate.service
Thu 2017-09-28 02:52:07 EDT 11h left Wed 2017-09-27 14:52:07 EDT 26min ago rkt-gc.timer rkt-gc.service
n/a n/a n/a n/a update-engine-stub.timer update-engine-stub.service
5 timers listed.
Next, create a
Docker Stack file called
docker-compose-secrets.yaml, like the one below.
version: '3.2'
services:
minio1:
image: minio/minio:edge
volumes:
- /mnt/docker_reg/minio/f1:/export
ports:
- "9001:9000"
networks:
- minio_distributed
deploy:
replicas: 1
placement:
constraints:
- node.labels.minio == minio1
restart_policy:
condition: on-failure
delay: 10s
max_attempts: 10
window: 60s
command: server http://minio1/export http://minio2/export http://minio3/export http://minio4/export
secrets:
- secret_key
- access_key
minio2:
image: minio/minio:edge
volumes:
- /mnt/docker_reg/minio/f2:/export
ports:
- "9002:9000"
networks:
- minio_distributed
deploy:
replicas: 1
placement:
constraints:
- node.labels.minio == minio2
restart_policy:
condition: on-failure
delay: 10s
max_attempts: 10
window: 60s
command: server http://minio1/export http://minio2/export http://minio3/export http://minio4/export
secrets:
- secret_key
- access_key
minio3:
image: minio/minio:edge
volumes:
- /mnt/docker_reg/minio/f3:/export
ports:
- "9003:9000"
networks:
- minio_distributed
deploy:
replicas: 1
placement:
constraints:
- node.labels.minio == minio3
restart_policy:
condition: on-failure
delay: 10s
max_attempts: 10
window: 60s
command: server http://minio1/export http://minio2/export http://minio3/export http://minio4/export
secrets:
- secret_key
- access_key
minio4:
image: minio/minio:edge
volumes:
- /mnt/docker_reg/minio/f4:/export
ports:
- "9004:9000"
networks:
- minio_distributed
deploy:
replicas: 1
placement:
constraints:
- node.labels.minio == minio3
restart_policy:
condition: on-failure
delay: 10s
max_attempts: 10
window: 60s
command: server http://minio1/export http://minio2/export http://minio3/export http://minio4/export
secrets:
- secret_key
- access_key
networks:
minio_distributed:
driver: overlay
attachable: true
secrets:
secret_key:
external: true
access_key:
external: true
Tip: Make sure to specfiy version
3.2+ as prior to that
attachable: true was not supported, and you will run in to network issues latter.
The
highlight lines in the configuration above are added to address placement constrains, if you don’t mind Minio instances jumping around while nodes go up/down you can just omit those lines.
Lets create the S3 secret keys (You don’t wont these in your Docker file for security reasons)
echo "minio" | docker secret create access_key -
echo "minio123" | docker secret create secret_key -
We are now going to add labels to the system
The labels are used as placement constrains, this will ensure that each Minio Docker instance stays on this node.
Verify you labels, default is empty unless you already added some labels…
docker node inspect coreos1
[..] snip
"Labels": {},
[..] snip
Create constrain labels.
docker node update --label-add minio=minio1 coreos1
docker node update --label-add minio=minio2 coreos2
docker node update --label-add minio=minio3 coreos3
Verify new labels are added to the system.
docker node inspect coreos1
[..] snip
"Labels": {
"minio": "minio1"
},
[..] snip
Next, lets deploy the stack, you do so by running the below.
docker stack deploy --compose-file=docker-compose-secrets.yaml minio_stack
Check the
minio_stack Docker logs, give it some time to come up…
while :; do docker logs `docker ps -a |grep minio1|head -1|awk '{print $1}'`; sleep 2; done
Last login to the UI to create your bucket, enter your access keys
http://your-host.domain.com:9002/minio/
Once logged in, click on the + sign, like the screen shut below, and create a bucket called
docker.
Next allow read/write privilege for
* by cliking on the 3 dots left side on the bucket name – (of curse you should adjust this production to your need).
Docker Registry Installation and configuration
Redis installation for Cache
Before we start the
Docker Registry installation, we need to install and configure
Redis for caching (Redis is optional).
To install
Redis Cache is simple, just run the below on every node running latter the
Registry. or you can use separate nodes for that if you like.
docker run -d --name="registryv2-redis" \
--network minio_stack_minio_distributed \
-p 6379:6379 \
--restart="always" sameersbn/redis
Docker Registry Setup Using Redis Cache
Now, we are ready to install the
Docker Registry.
To install the
Docker Registry create a file called
docker-registry-config.yml like the below.
version: 0.1
log:
level: debug
formatter: text
fields:
service: registry
environment: staging
loglevel: debug
storage:
s3:
accesskey: minio
secretkey: minio123
region: us-east-1
regionendpoint: http://minio1:9001
bucket: docker
encrypt: false
secure: true
v4auth: true
chunksize: 5242880
rootdirectory: /
cache:
blobdescriptor: redis
delete:
enabled: true
maintenance:
uploadpurging:
enabled: true
age: 168h
interval: 24h
dryrun: false
readonly:
enabled: false
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
debug:
addr: localhost:5001
redis:
addr: registryv2-redis:6379
db: 0
dialtimeout: 10ms
readtimeout: 10ms
writetimeout: 10ms
pool:
maxidle: 16
maxactive: 64
idletimeout: 300s
For a complete reference of all
Docker registry attributes click here
Next, run the below on all nodes running a
Minio instance.
docker run -d --name=registry \
--network minio_stack_minio_distributed \
-p 5000:5000 \
-v $(pwd)/docker-registry-config.yml:/etc/docker/registry/config.yml \
--restart="always" \
registry:latest
Tip: You can run the
Docker Registry on as little as two nodes if that is enough for your workload, each
Registry will access the local
Minio instance.
To verify
Docker Registry is working and using
Redis Cache.
Run the below in the in one windows.
docker exec -t -i `docker ps -a |grep redis|awk '{print $1}'|head -1` redis-cli monitor
In another window run the below. this will pull
hello-world image, then push that in your private
Docker Registry, you should node a bunch of activity in the
Redis window.
docker pull hello-world
docker tag hello-world localhost:5000/hello-world-1.0
docker push localhost:5000/hello-world-1.0
docker pull localhost:5000/hello-world-1.0
Another way to verify
Redis working is, by looking in the
Docker Registry logs, you should see something like the below, look for the words
redis.
docker logs 148cc9088407|less
time="2017-09-08T17:13:05.850504777Z" level=info msg="using redis blob descriptor cache" environment=staging go.version=go1.7.6 instance.id=560567ff-4c00-4552-b176-49a083734f87 service=registry version=v2.6.2
[..] snip
time="2017-09-08T17:13:21.595916263Z" level=info msg="redis: connect registryv2-redis:6379" environment=staging go.version=go1.7.6 instance.id=560567ff-4c00-4552-b176-49a083734f87 redis.connect.duration=3.395927ms service=registry version=v2.6.2
[..] snip
Now, lets verify the object storage working.
see the below screen capture
Important Note: The above configuration is not using
SSL nor is it using any
Authentication/Authorization, token for access control. these are things defiantly required in any production configuration,
Click here – to configure a secure registry using SSL, Tokens and LDAP /a>.
Another way how to run Docker Registry
If you are using CoreOS, you can also use fleetctl to create the Docker Registry, you will do so by creating a file called registry.service with the below content.
Description=Private Docker Registry
After=docker.service
[Service]
TimeoutStartSec=0
Restart=always
RestartSec=10s
ExecStartPre=/usr/bin/docker pull registry:latest
ExecStart=/usr/bin/docker run -d --name=registry --network minio_stack_minio_distributed -p 5000:5000 -v /var/tmp/docker-registry-config.yml:/etc/docker/registry/config.yml --restart="always" registry:latest
ExecStop=/usr/bin/docker stop registry
ExecStopPost=/usr/bin/docker kill registry
ExecStopPost=/usr/bin/docker rm registry
[X-Fleet]
Global=true
Next, run create the service for all nodes
fleetctl start registry.service
Verify the service working.
fleetctl list-units
UNIT MACHINE ACTIVE SUB
registry.service 3e0abc5b.../10.0.2.12 active running
registry.service 4f72b720.../10.0.2.13 active running
registry.service ba90d46d.../10.0.2.11 active running
To remove/destroy the unit just run the below.
fleetctl destroy registry.service
Docker Registry SSL and Authentication
I hope to update this soon.. once I have a chance. in the mean time you can check this out Gotchas / Tips Creating Your Own Private Docker Registry With Self Signed Certificate
Docker Registry High Availability And Load Balancing
The last piece of the
Docker Registry needed for High Availability is adding a load balancer in front of all
Docker Registry instances.
There are many options you can use for Load Balancing, the simplest option might be using something like
Traefik or
Consul and especially if you already have such a setup in place.
You can see details how to –
Using Traefik As a Load Balancer / HTTP Reverse Proxy For Micro-Services or
Using Consul for Service Discovery In Multiple Data Centers – Part 1.
Other widely used options are, using
Nginx or
HAproxy.
Helpful Docker alias I use
dps () { docker ps -a ; }
dl () { docker logs -f $1 & }
drm () { docker stop $1 && docker rm $1; }
In CoreOS you do so by adding the below to the
write_files: section.
- path: /etc/profile.d/aliases.sh
content: |
dps () { docker ps -a ; }
dl () { docker logs -f $1 & }
drm () { docker stop $1 && docker rm $1; }
In part 3, I discuss
Securing A Private Docker Registry By Using An SSL, Tokens, LDAP
I hope you enjoyed reading the
High Availability Docker Registry, give it a thumbs up by rating the article or by just providing feedback.
You might also like –
Other articles related to Docker Kubernetes / micro-services.
Loading...
Next allow read/write privilege for * by cliking on the 3 dots left side on the bucket name – (of curse you should adjust this production to your need).
Important Note: The above configuration is not using SSL nor is it using any Authentication/Authorization, token for access control. these are things defiantly required in any production configuration, Click here – to configure a secure registry using SSL, Tokens and LDAP /a>.