Creating Multi-Container Docker Applications by Using Docker-Compose

Docker Compose helps you defining and running multi-container Docker applications. To run a multi-service application containers, use Docker-Stack. Note: When using Docker Swarm, Docker-Stacks might be a better option.
Below I will show you how to create a simple 2 tier application(NodeJS front-end + MySQL back-end), by using Docker-Compose, I will also address some of the challenges you might run-into.

Installing Docker-Compose

First, make sure you have the latest docker-compose installed, to install the latest docker-compose release, run the below.

curl -L https://github.com/docker/compose/releases/download/1.14.0-rc2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

Note: Below I will be using the Docker-Compose version 3 (not to confuse with Docker-Compose release, which is currently up-to version 1.14.0-rc2). the configuration options might differ from version to version (currently at version 3). Now, that docker-compose is installed (Note: you can test it by typing docker-compose -v), lets start with the application configuration.

Configuring a MySQL docker-compose YAML configuration file

First, lets create a working directory, I am using dc-dev1 dc for docker compose.

mkdir dc-dev1 && cd dc-dev1

For Docker-Compose to work, we need at minimum a docker-compose.yml configuration file and a standard Dockerfile. By default Docker-Compose will look for the docker-compose.yml in the current working directory. the docker-compose.yml will direct the docker-compose, what to build and if directed, check for a Dockerfile (which might have additional build options). Lets start by create a directory for the MySQL specific configuration.

mkdir mysql-db1 mysql-db1/mysql-schema /var/lib/mysql

Next, lets create a MySQL docker-compose.yml as well as an associated Dockerfile. Below is the MySQL initial docker-compose.yml file, create this file in the base directory (dc-dev1/docker-compose.yml).

cat dc-dev1/docker-compose.yml
version: '3'

services:
  mysql-db1:
    image: mysql
    build: ./mysql-db1
    container_name: "mysql-db1"
    ports:
      - "3306:3306"
    environment:
      MYSQL_DATABASE: games
      MYSQL_USER: mysql_root
      MYSQL_ROOT_PASSWORD: 12345678
      MYSQL_PASSWORD: 12345678
      MYSQL_PORT: 3306
    volumes:
      - ./mysql-db1/mysql-schema:/docker-entrypoint-initdb.d:ro
      - /var/lib/mysql:/var/lib/mysql

volumes:
  data-volume:

Now, lets create the MySQL Dockerfile.

cat dc-dev1/mysql-db1/Dockerfile
FROM mysql:latest
MAINTAINER Eli Kleinman

ENV MYSQL_DATABASE games
ENV MYSQL_USER root
ENV MYSQL_ROOT_PASSWORD 12345678
ENV MYSQL_PASSWORD 12345678

Note: I am defining the version as latest. In production, a better option might be to specify the version, if not, you can get different results then expected with a different version then expected. Next, lets create a MySQL schema file, the schema file will create a test database with test records.

cat mysql-schema/setup.sql 
create database test;
use test;

CREATE TABLE testtab
(
	id INTEGER AUTO_INCREMENT,
	name TEXT,
	PRIMARY KEY (id)
) COMMENT='this is my test table';

Now, we are ready for the first MySQL application run, just run the below.

docker-compose up

# or to run in detach mode.
docker-compose up -d

If all went right, you should have a successful MySQL container up and running. The simplest way to test the database is, connect to the container, like below, then just run mysql.

docker exec -it `docker ps -a|grep mysq|awk '{print  $1}'` /bin/bash
root@705b8bc60cb2:/# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.18 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test               |
+--------------------+
5 rows in set (0.00 sec)

mysql>

Lets, break down the docker-compose.yml configuration. Most of the configuration is self explanatory, with a few quirks outlined below… First the simple stuff, the common docker-compose.yml keywords.

• The image keyword refers to the Docker hub image the build is based on.
• The build keyword refers to the directory docker compose will look for a Dockerfile (it might have additional configurations).
• The container_name and ports keyword are defining the container name and exposed port.
• The environment keyword is used to set environment variables used in the image, in our example setting the MySQL details.
• The volumes keyword set the volumes/directors exposed to the container, more on this below.

Notes / Issue about the above configuration

Issue: How can I configure my own database, tables, records, etc..? Solution: In recent versions of Docker MySQL builds, there is a new option added. if you create a directory docker-entrypoint-initdb.d, you have to option to add a database schema file or a sh start-up script, the system will search for those files at startup. For that reason, I create a directory dc-dev1/mysql-db1/mysql-schema, I then created a sql schema file, which is used at first system startup to created the db schema. The directory dc-dev1/mysql-db1/mysql-schema is mapped as docker-entrypoint-initdb.d in the docker image to be available at system startup. Issue: All my files (will) get lost once the Docker image is destroyed, I would like to preserve them. Solution: While there are multiple options available, I am using one of them which I think is the simplest. I use the directory /var/lib/mysql on the main node (the directory can be an an NFS mount), I then map that to store the MySQL data. This option is visible with the second Volume mapping option of /var/lib/mysql:/var/lib/mysql Note: The Volumes order is source directory:detestation directory. One last thing to consider. The MySQL environment variables are specified twice. you can remove the environment variables from the Dockerfile (if you like), as the system will use the docker-compose.yml file to get this information, more on this (and issues concerning this) are discussed latter in this article.

MySQL environment MYSQL_USER configuration

One of the issues I encountered was the error below. to get around this issue, make sure to set the MYSQL_USER to something other then root, I am using in our example the mysql_root user.

[..] snip
mysql-db1    | ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'root'@'%'
mysql-db1 exited with code 1

NodeJS Container Setup and Configuration

Next, I am going to work on to the NodeJS Docker container configuration. Since we are now focusing on the NodeJS setup, lets move the existing docker-compose.yml aside (don’t worry will stitch everything together at the end of this post). To speed up things I am using the NodeJS express module to create the initial directory structure. first lets install the express module, to do so, just run.

npm install -g express-generator

Next, lets create the directory structure by using express, just run the below.

express node-ms1 && cd node-ms1

Now, lets modify the NodeJS express generated configuration. Lets create a simple API route file, will then add this to the NodeJS app.js. First lets create our route file.

cat node-ms1/routes/api-r1.js
var express = require('express');
var router = express.Router();

router.get('/hello', function(req, res) {
   res.send('Hello from Eli');
});

module.exports = router;

Next, lets add the new api route to the app.js You will need add the route in two places, just below the user routes.

cat node-ms1/app.js
[..]snip
var users = require('./routes/users');
var api = require('./routes/api');
[..]snip

app.use('/users', users);
app.use('/api', api);
[..]snip

Next, lets move on in creating the docker-compose.yml and Dockerfile.

NodeJS docker-compose caveats and options

Before continuing. Now that we have seen what it takes to create a >docker-compose.yml in the MySQL setup, one would think creating a docker-compose.yml for NodeJS would just be a simple process. yes it is, but with a few caveats. all this and more are discussed below. The options below are discussed in grater detail

1. • Each Docker container use their own code base, and gets update at every startup
   • Let most of the configuration derive from the Dockerfile
2. • All Docker containers share the code base
   • Let all of the configuration derive from the docker-compose.yml
3. • All Docker containers share part of the code base
   • Let most of the configuration derive from the docker-compose.yml with helper script to complete the process
4. • All Docker containers share the fully code base
   • Let most of the configuration derive from the docker-compose.yml without helper script

Now, lets jump right in.

Option one – Each Docker container use their own code base

The docker-compose.yml file below, contains no specific configuration information, most information comes from the Dockerfile.

cat node-ms1/docker-compose.yml
version: '3'

services:
  node-ms1:
    build: ./node-ms1
    container_name: "node-ms1"
    working_dir: /app
    ports:
      - "9000:3000"
    environment:
      ANY_VARBLE: eli

The Dockerfile file below, will copy the code from the current working directory to the docker container.

cat node-ms1/Dockerfile
FROM node:latest
MAINTAINER Eli Kleinman

ENV http_proxy=http://httpproxy.domain.com/ https_proxy=http://httpproxy.domain.com/

WORKDIR /app

copy ./package.json /app
RUN cd /app && \
    npm install --quiet

copy . /app

CMD ["npm", "start"]

Note: The package.json is copied first, then the rest of the code so less code would need to be re-run in case of change, i.e. all the NodeJS npm stuff will not have to re-run every time.

Option Two – All Docker containers share the code base

cat docker-compose.yml
version: '3'

services:
  node-ms1:
    working_dir: /app
    build: ./node-ms1
    command: bash -c "npm install && npm start"
    container_name: "node-ms1"
    volumes:
      - ./node-ms1/:/app
    ports:
      - "9000:3000"
    environment:
      VARIABLE_NAME: my_variable_value

The docker-compose.yml contains the command keyword to run npm and npm install at startup.

cat node-ms1/Dockerfile
FROM node:latest
MAINTAINER Eli Kleinman

ENV http_proxy=http://httpproxy.domain.com/ https_proxy=http://httpproxy.domain.com/

Option Three – All Docker containers share “part” of the code base – with helper script

version: '3'

services:
  node-ms1:
    working_dir: /app
    build: ./node-ms1
    container_name: "node-ms1"
    volumes:
      - ./node-ms1/:/app
    ports:
      - "9000:3000"
    environment:
      VARIABLE_NAME: my_variable_value

Adding the helper script will insure npm is run when all volumes are properly mounted.

cat node-ms1/Dockerfile
FROM node:latest
MAINTAINER Eli Kleinman

ADD start-script.sh .

CMD ./start-script.sh

Below is an example of the startup helper script.

cat start-script.sh
#!/usr/bin/env bash

until cd /app && npm install
do
    echo "Retrying npm install"
done
npm start

Note: This solution above was taken from Stackoverflow

Option Four – All Docker containers share “fully” the code base

version: '3'

services:
  node-ms1:
    build: ./node-ms1
    container_name: "node-ms1"
    working_dir: /app
    ports:
      - "9000:3000"
    environment:
      ANY_VARBLE: eli
    volumes:
      - ./node-ms1/:/app

volumes:
  data-volume:

Only npm start is used from the Dockerfile

cat ./node-ms1/Dockerfile
FROM node:latest
MAINTAINER Eli Kleinman

ENV http_proxy=http://httpproxy.domain.com/ https_proxy=http://httpproxy.domain.com/

CMD ["npm", "start"]

Feel free to test all the above options, each has their own pros and cons. To bring up the NodeJS container just run the usual, like below.

docker-compose up -d

Now that you have seen all the NodeJS options, lets stitch them together.

MySQL and NodeJS configuration

Below is an updated docker-compose.yml file with MySQL and NodeJS stitch together.

cat docker-compose.yml
version: '3'

services:
  node-ms1:
    build: ./node-ms1
    container_name: "node-ms1"
    working_dir: /app
    depends_on:
      - mysql-db1
    links:
      - mysql-db1:mysql-db1
    ports:
      - "9000:3000"
    environment:
      VARIABLE_NAME: my_variable_value
  mysql-db1:
    image: mysql
    build: ./mysql-db1
    container_name: "mysql-db1"
    ports:
      - "3307:3306"
    environment:
      MYSQL_DATABASE: games
      MYSQL_USER: mysql_root
      MYSQL_ROOT_PASSWORD: 12345678
      MYSQL_PASSWORD: 12345678
      MYSQL_PORT: 3306
    volumes:
      - ./mysql-db1/mysql-schema:/docker-entrypoint-initdb.d:ro
      - /var/lib/mysql:/var/lib/mysql

volumes:
  data-volume:

Note: The docker-compose.yml dependency will only wait till the MySQL container is up. to wait till the MySQL DB is totally up, additional helper scripts might be needed. a simple example is below.

# docker-compose.yml
[..] snip
command: ["./wait-for-it.sh", "db:3306", "--", "npm", "start"]

# wait-for-mysql.sh script
#!/bin/bash
# wait-for-mysql.sh

set -e

host="$1"
shift
cmd="$@"

until mysql -h "$host" -U "mysql_user"; do
  >&2 echo "MySQL is unavailable - sleeping"
  sleep 1
done

>&2 echo "MySQL is up - executing command"
exec $cmd

In docker version 3 you can use the command: keyword to add a script, something like the above, more details are available at Docker Home wait-for-mysql.sh script

Helpful hints on the above configurations

To connect and access MySQL from the local Node, you can use something like the below.

docker exec -it `docker ps -a|grep nodejs|awk '{print  $1}'` /bin/bash

# Then just type mysql
root@705b8bc60cb2:/# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.7.18 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Connecting to MySQL from outside this node, use something like the below.

mysql -h 127.0.0.1 -P 3307 -u root -p

OR specify protocol as tcp (if local)
mysql -P 3307 --protocol=tcp -u root -p

To test NodeJS and MySQL working together, follow the below. Note: You will have to install the MySQL client binary on the NoeJS to use it.

apt-get update
apt-get install mysql-client

# Then try
mysql -u mysql-user -h mysql-db1 -P 3306 -p

Note: You can also do a ping “mysql-db1” or do a getent host mysql-db1, to make sure it works. Whats next. So far I was working with docor-compose to create a container stack. next, I am going to add a load balancer, traefik as part of the mix – helping configure a full DevOps life-cycle environment. Note: Traefik is a micro services load balancer, similar to HAproxy but designed for micro services. To learn how to install and configure a traefik load balancer please check this out Using Traefik Load Balancer HTTP Reverse Proxy Micro-Services
What are your challenges when using docor-compose, please let me know in the comments below.
You might also like Managing Docker On Ubuntu 17.04 Using Rancher Or Portainer Gotchas / Tips Creating Your Own Private Docker Registry With Self Signed Certificate Using ZFS For The Docker COW Storage Layer(s) In Ubuntu 17.04