Configuring Consul for Service Discovery in a Multiple Data Centers

Since my last post on how to configure Consul for Service Discovery in a Multi Data Center, a number of things have changed and got updated. You can see the original post by going here part 1 and here part 2, the original write-up was using version 0.9.2 while the current version is 1.4.2. Below I updated the consul configuration to work with version 1.4 (the most recent version as of this writing). Before looking on the configuration changes, its good to point-out some of the new features added to more recent versions of Consul. Below is a partial list on some of the new or enhance features.

1. ACLs: With recent versions you can now configure ACL(access control lists) who has access to what, this includes the Web-UI, Rest calls as well as the CLI.
2. SSL:Full support for SSL – configured with consul tls…
3. UI Update: The web UI got a major overall facelift.

For the full list of changes / enhancements (and their are many since version 0.9.2) look here. Note: One of the changes / issues I ran in to with the current version(s), are the dismissal of using 0.0.0.0 as the -client bind address(really a Go change), in order for Consul to be available for DNS, etc requests i.e listen to all address, you will have to change the below

# Remove from startup
-client="0.0.0.0"

# Add instead to the json config
"client_addr": "10.150.100.17 127.0.0.1"...

In addition to the above, some configuration parameters changed or got deprecated.

Consul Server Example

The IP address schema used in the consul configuration is below.

The table below lists names and IP address used in this configuration (feel free to replace with your needs).

DC1
Name	IP Addrss
Consul Servers
ConsulMaster1	10.150.100.17
ConsulMaster2	10.150.100.18
ConsulMaster3	10.150.100.19
Consul Client
Dc1Client1	10.150.0.145
End Host
dc1-devops1	10.150.0.106
DC2
Name	IP Address
Consul Servers
ConsulMaster1	10.150.100.17
ConsulMaster2	10.150.100.18
ConsulMaster3	10.150.100.19
Consul Client
Dc2Client1	10.50.0.145
End Host
dc2-devops1	10.50.0.106

Consul Multi Data Center layout used in this article Please take a look on part 1 of my original consul article, for a similar IP Address schema you can use.

Consul server installation and configuration

In the below test I used Solaris zones installation configuration. For a Solaris zone instillation example please take a look on part 1 (using version 0.9.2). First, lets download consul. For a list of latest releases click here. I used version 1.4.2 the latest version as of this writeup.

wget https://github.com/hashicorp/consul/archive/v1.4.2.zip

Next, lets configure user and groups.

groupadd consul
useradd -d /var/consul -g consul -m -s /bin/bash -c "Consul App" consul
mkdir -p /etc/consul.d/{bootstrap,server,client}
 
mkdir /var/consul
chown consul:consul /var/consul

Next, we need to generate an encryption key.

consul keygen
G1Y/7ooXzfuyPmyzj2RlDg==

Finally we need to create the Consul config.json. you do so by running the below. Consul config.json for the Consul Servers Consul Server DC1 – First node config.json Note: Replace DNS and IP address information to reflect your environment.

cat /etc/consul.d/server/config.json
{
  "bind_addr": "10.150.100.17",
  "client_addr": "10.150.100.17 127.0.0.1",
  "datacenter": "dc1",
  "data_dir": "/consul",
  "encrypt": "G1Y/7ooXzfuyPmyzj2RlDg==",
  "log_level": "INFO",
  "enable_debug": true,
  "node_name": "ConsulMaster1",
  "server": true,
  "bootstrap_expect": 3,
  "leave_on_terminate": false,
  "skip_leave_on_interrupt": true,
  "rejoin_after_leave": true,
  "disable_update_check": true,
  "performance": {
    "raft_multiplier": 1
  },
  "recursors": ["8.8.4.4", "8.8.8.8"],
  "retry_join": [
    "10.150.100.17:8301",
    "10.150.100.18:8301",
    "10.150.100.19:8301"
    ],
  "retry_join_wan": [
    "10.50.100.17:8302",
    "10.50.100.18:8302",
    "10.50.100.19:8302"
    ]
}

Consul Server DC2 – First node config.json

{
  "bind_addr": "10.50.100.17",
  "client_addr": "10.50.100.17 127.0.0.1",
  "datacenter": "dc2",
  "data_dir": "/consul",
  "encrypt": "G1Y/7ooXzfuyPmyzj2RlDg==",
  "log_level": "INFO",
  "enable_debug": true,
  "node_name": "ConsulMaster1",
  "server": true,
  "bootstrap_expect": 3,
  "leave_on_terminate": false,
  "skip_leave_on_interrupt": true,
  "rejoin_after_leave": true,
  "disable_update_check": true,
  "performance": {
    "raft_multiplier": 1
  },
  "recursors": ["8.8.4.4", "8.8.8.8"],
  "retry_join": [
    "10.50.100.17:8301",
    "10.50.100.18:8301",
    "10.50.100.19:8301"
    ],
  "retry_join_wan": [
    "10.150.100.17:8302",
    "10.150.100.18:8302",
    "10.150.100.19:8302"
    ]
}

Note: The above config.json is for the first node. Make sure to replace the below fields on the two other nodes, nodes two and three.

1. bind_addr
2. node_name
3. client_addr

Tip: The performance keyword is by default set to 5. the reason for that (to my understanding) is to accommodate AWS t2.tiny configurations, for maximum performance set this to 1. Next, to start the consul servers, just run the below. First on the DC1 3 nodes. then, once up, run on the 3 DC2 nodes. Tip: You can remove the nohup to run in the foreground.

nohup su - consul -c "/usr/bin/consul agent -config-dir /etc/consul.d/server/ -ui >> /var/tmp/consul_log.out 2>&1" &

Note: The above startup enables the Web UI. if you don’t like the Web UI on the Consul servers just remove the -ui option. Now, Lets move to the Consul client configuration. Consul config.json for the Consul Clients Consul Client DC1 – First node config.json

cat <<'EOF' > /etc/consul.d/client/config.json
{
  "bind_addr": "10.150.0.145",
  "client_addr": "10.150.0.145 10.150.100.145 127.0.0.1",
  "datacenter": "dc1",
  "data_dir": "/consul",
  "encrypt": "G1Y/7ooXzfuyPmyzj2RlDg==",
  "log_level": "INFO",
  "enable_debug": true,
  "node_name": "Dc1Client1",
  "enable_script_checks": true,
  "server": false,
  "recursors": ["8.8.4.4" ,"8.8.8.8"],
  "services": [{
                  "id": "dc1-devops1",
                  "name": "rad-6789",
                  "tags": ["dc1-devops1"],
                  "address": "10.150.0.106",
                  "port": 22,
                  "checks": [{
                          "id": "dc1-rad-6789",
                          "name": "DC1-rad-6789",
                          "service_id": "rad-6789",
                          "tcp": "dc1-devops1:6789",
                          "tls_skip_verify": false,
                          "interval": "2s",
                          "timeout": "1s"
                  }]
              }],
  "rejoin_after_leave": true,
  "disable_update_check": true,
  "retry_join": [
    "10.150.100.19:8301",
    "10.150.100.17:8301",
    "10.150.100.18:8301"
    ]
}
EOF

Consul Client DC2 – First node config.json

cat <<'EOF' > /etc/consul.d/client/config.json
{
  "bind_addr": "10.50.0.145",
  "client_addr": "10.50.0.145 10.50.100.145 127.0.0.1",
  "datacenter": "dc2",
  "data_dir": "/consul",
  "encrypt": "G1Y/7ooXzfuyPmyzj2RlDg==",
  "log_level": "INFO",
  "enable_debug": true,
  "node_name": "Dc2Client1",
  "enable_script_checks": true,
  "server": false,
  "recursors": ["8.8.4.4" ,"8.8.8.8"],
  "services": [{
                  "id": "dc2-devops1",
                  "name": "rad-6789",
                  "tags": ["dc2-devops1"],
                  "address": "10.50.0.106",
                  "port": 22,
                  "checks": [{
                          "id": "dc2-rad-6789",
                          "name": "DC2-rad-6789",
                          "service_id": "rad-6789",
                          "tcp": "dc2-devops1:6789",
                          "tls_skip_verify": false,
                          "interval": "2s",
                          "timeout": "1s"
                  }]
              }],
  "rejoin_after_leave": true,
  "disable_update_check": true,
  "retry_join": [
    "10.50.100.19:8301",
    "10.50.100.17:8301",
    "10.50.100.18:8301"
    ]
}
EOF

Note: The Address property under services, can be used to replace the DNS reply address for this service lookup. Create a startup script with the below.

cat consul.sh
#!/bin/bash

case $1 in
'start')
nohup su - consul -c "/usr/bin/consul agent -config-dir /etc/consul.d/client/ -ui >> /var/tmp/consul_log.out 2>&1" &
;;
'stop') pkill -9 -U consul
;;
'restart')
          $0 stop
          sleep 1
          $0 start
;;
*) echo "Usage $0 [start|stop|restart]"
;;
esac

To start consul, just run the below.

consul.sh start

Tip: You can omit the nohup to run in the foreground(for troubleshooting). If all done correctly, you should now have a working Consul cluster. To access the Web UI , just go to any Consul server, port 8500. For example http://10.150.100.17:8500 would bring you to the below screen, pick your DC and continue to node and services selection. List of Consul nodes. A failed Consul node services. To continue reading part two, on how to configure Consul for Multi Data Center click here. Note: This article was update using Consul version 1.4, to access the original article using Consul version 0.9.2 click here. Like what you’re reading? give it a thumbs up by rating the article. You might also like – related to Docker Kubernetes / micro-services.