Installing, configuring 3 node Kubernetes(master) cluster on CentOS 7.5 – configure the Kubernetes VM’s

• Part 1: Initial setup – bear-metal installation, configuration
• Part 2: Installing the Kubernetes VM’s
• Part 3: Installing and configuring Flanneld, CNI plugin and Docker
• Part 4: Installing and configuring kubernetes manifest and kubelet service
• Part 5: Adding CoreDNS as part of the Kubernetes cluster
• Part 6: Adding / Configuring Kubernetes worker nodes
• Part 7: Enabling / Configuring RBAC, TLS Node bootstrapping
• Part 8: Installing / Configuring Helm, Prometheus, Alertmanager, Grafana and Elasticsearch

Installing the Kubernetes VM’s

• Install CentOS 7.5(1804) / (CentOS-7-x86_64-Minimal-1804.iso), on 5 Virtual Box VM’s.
• 3 VM’s will be configured as Masters, and 2 will be configured as worker nodes.
The host names, IP Address’s used in my configuration are below (feel free to come up with your own configuration schema). Configure each VM with the below resources.
1. 1 Virtual CPU is fine.
2. At least 2Gb of RAM.
3. At least 12Gb HDD.
4. Assign Network Switch to network port.
Set the below on each Kubernetes VM (master and Workers). Disable SE Linux by Running the below.

setenforce 0

# Save the config after a reboot
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

Disable swap

swapoff -a

# Save the config after a reboot by commenting in /etc/fstab
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

If you are behind a firewall or corporate proxy, add your proxy to /etc/yum.conf and /etc/environment (for an example check out in part 1).

modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
echo '1' > /proc/sys/net/ipv4/ip_forward

# Run sysctl to verify
sysctl -p

Install Docker packages

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce

Add kubernetes repo

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Install kubernetes and other related packages

yum install -y kubelet kubeadm kubectl flannel epel-release conntrack jq vim bash-completion lsof screen git net-tools && yum update -y

Since we are not going to use kubeadm for our configuration comment out all entry’s in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf like the below example.

cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
#[Service]
#Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
#Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
#EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
#EnvironmentFile=-/etc/sysconfig/kubelet
#ExecStart=
#ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Disable the system firewall by running the below, it would be partially managed by kubernetes.

systemctl disable firewalld.service
systemctl stop firewalld

Note: There are other options to deal with firewall rules, for example just enabling the ports required by kubernetes. Lastly before continuing reboot each vm instances.

Configuring etcd – kubernetes SSL certificates

In order for all communication in kubernetes to be secure we need to create certificates. Note: In order to keep things the simplest possible, I will be using the same certificate key for all components. in production you might consider breaking those out in your production environment.

Creating Kubernetes SSL certificates

Create a cert.conf file with the content below. add other host names/ ip address as needed, also make sure to replace with your domain name.

[req] 
default_bits       = 2048
prompt             = no
default_md         = sha256
distinguished_name = dn
req_extensions     = v3_req
x509_extensions    = v3_ca
 
[ dn ]
C                  = US
ST                 = NY
L                  = New York
O                  = Company1
OU                 = Ops
CN                 = etcd-node
 
[ v3_ca ]
keyUsage = critical,keyCertSign, cRLSign
basicConstraints = critical,CA:TRUE
subjectKeyIdentifier = hash

[ v3_req ]
keyUsage = critical,digitalSignature, keyEncipherment, nonRepudiation
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = critical,CA:FALSE
subjectKeyIdentifier = hash
subjectAltName = @alt_names
 
[ alt_names ]
DNS.1              = kubernetes
DNS.2              = kubernetes.default
DNS.3              = kubernetes.default.svc
DNS.4              = kubernetes.default.svc.cluster.local
DNS.5              = kube-apiserver
DNS.6              = kube-admin
DNS.7              = localhost
DNS.8              = domain.com
DNS.9              = kmaster1
DNS.10              = kmaster2
DNS.11              = kmaster3
DNS.12              = kmaster1.local
DNS.13              = kmaster2.local
DNS.14              = kmaster3.local
DNS.15              = kmaster1.domain.com
DNS.16              = kmaster2.domain.com
DNS.17              = kmaster3.domain.com
DNS.18              = knode1
DNS.19              = knode2
DNS.20              = knode3
DNS.21              = knode1.domain.com
DNS.22              = knode2.domain.com
DNS.23              = knode3.domain.com

IP.1              = 127.0.0.1
IP.2              = 0.0.0.0
IP.3              = 10.3.0.1
IP.4              = 10.3.0.10
IP.5              = 10.3.0.50
IP.6              = 172.20.0.1
IP.7              = 172.20.0.2
IP.8              = 172.20.0.11
IP.9              = 172.20.0.12
IP.10              = 172.20.0.13
IP.11              = 172.20.0.51
IP.12              = 172.20.0.52
IP.13              = 172.20.0.53

email              = admin@example.com

Next, Copy the below and create a file called run.sh in the same directory as cert.conf file.

# Generate the CA private key
openssl genrsa -out ca-key.pem 2048
sed -i 's/^CN.*/CN                 = Etcd/g' cert.conf

# Generate the CA certificate.
openssl req -x509 -new -extensions v3_ca -key ca-key.pem -days 3650 \
-out ca.pem \
-subj '/C=US/ST=New York/L=New York/O=example.com/CN=Etcd' \
-config cert.conf

 # Generate the server/client private key.
openssl genrsa -out etcd-node-key.pem 2048
sed -i 's/^CN.*/CN                 = etcd-node/g' cert.conf

# Generate the server/client certificate request.
openssl req -new -key etcd-node-key.pem \
-newkey rsa:2048 -nodes -config cert.conf \
-subj '/C=US/ST=New York/L=New York/O=example.com/CN=etcd-node' \
-outform pem -out etcd-node-req.pem -keyout etcd-node-req.key
# Sign the server/client certificate request.
openssl x509 -req -in etcd-node-req.pem \
-CA ca.pem -CAkey ca-key.pem -CAcreateserial \
-out etcd-node.pem -days 3650 -extensions v3_req -extfile cert.conf

Next, make the file executable, and run to create the certificates. output would look something like the below.

chmod +x run.sh

./run.sh 
Generating RSA private key, 2048 bit long modulus
......................+++
................................................+++
e is 65537 (0x10001)
Generating RSA private key, 2048 bit long modulus
..................................+++
........................................................................+++
e is 65537 (0x10001)
Signature ok
subject=/C=US/ST=NY/L=New York/O=Company1/OU=Ops/CN=etcd-node
Getting CA Private Key

To verify the certificate, run the below.

openssl x509 -in etcd-node.pem -text -noout

Once completed you should be left with the below list of files.

ca-key.pem  ca.pem  ca.srl  cert.conf  etcd-node-key.pem  etcd-node-req.pem  etcd-node.pem

From the list of above files we only need the ca.pem, ca-key.pem, etcd-node.pem and etcd-node-key.pem. Create the directory if not exist, and copy the certificate files.

mkdir -p /etc/kubernetes/ssl
ln -s /etc/kubernetes/ssl /etc/kubernetes/pki
cp ca.pem ca-key.pem etcd-node.pem etcd-node-key.pem /etc/kubernetes/ssl

Note: I named the most used key certificate as etcd-node-(key).pem, since its the first SSL certificate being used, feel free to rename at your own desire(just remember to update all references).

Etcd installation and configuration

Add to /etc/environment

ETCDCTL_SSL_DIR=/etc/kubernetes/ssl
ETCD_SSL_DIR=/etc/kubernetes/ssl
ETCDCTL_CA_FILE=/etc/kubernetes/ssl/ca.pem
ETCDCTL_CERT_FILE=/etc/kubernetes/ssl/etcd-node.pem
ETCDCTL_KEY_FILE=/etc/kubernetes/ssl/etcd-node-key.pem
ETCDCTL_ENDPOINTS="https://172.20.0.11:2379,https://172.20.0.12:2379,https://172.20.0.13:2379"
ETCD_ENDPOINTS="https://172.20.0.11:2379,https://172.20.0.12:2379,https://172.20.0.13:2379"
FLANNELD_ETCD_ENDPOINTS="https://172.20.0.11:2379,https://172.20.0.12:2379,https://172.20.0.13:2379"
FLANNELD_IFACE="enp0s3"
FLANNELD_ETCD_PREFIX="/coreos.com/network"

If you are behind a corporate firewall or proxy, set the below in /etc/systemd/system/docker.service.d/http-proxy.conf (create the directory if not exist).

Environment="HTTP_PROXY=http://your_proxy_ip_:1234/"
Environment="HTTPS_PROXY=http://your_proxy_ip_:1234/"
Environment="NO_PROXY=.domain.com,127.0.0.1,localhost,kmaster1,kmaster1.domain.com,kmaster2,kmaster2.domain.com,kmaster3,kmaster3.domain.com,172.20.0.12,172.20.0.11,172.20.0.13"

Install etcd on the 3 master servers

Download the latest etcd and configure

mkdir -p /var/lib/etcd
curl -# -LO https://github.com/coreos/etcd/releases/download/v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz
tar xf etcd-v3.3.8-linux-amd64.tar.gz
chown -Rh root:root etcd-v3.3.8-linux-amd64
find etcd-v3.3.8-linux-amd64 -xdev -type f -exec chmod 0755 '{}' \;
cp etcd-v3.3.8-linux-amd64/etcd* /usr/bin/

Create etcd service, like the below. cat /etc/systemd/system/etcd.service Note: Replace on each each master the below values with the proper name/ip.
1. –name=kmaster1
2. –listen-peer-urls=https://172.20.0.11:2380
3. –advertise-client-urls=https://172.20.0.11:2379
4. –initial-advertise-peer-urls=https://172.20.0.11:2380
5. –listen-client-urls=https://172.20.0.11:2379,https://127.0.0.1:2379,https://127.0.0.1:4001
The example below is from master1.

[Unit]
Description=Etcd Server
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/bin/etcd \
--name=kmaster1 \
--cert-file=/etc/kubernetes/pki/etcd-node.pem \
--key-file=/etc/kubernetes/pki/etcd-node-key.pem \
--peer-cert-file=/etc/kubernetes/pki/etcd-node.pem \
--peer-key-file=/etc/kubernetes/pki/etcd-node-key.pem \
--trusted-ca-file=/etc/kubernetes/pki/ca.pem \
--peer-trusted-ca-file=/etc/kubernetes/pki/ca.pem \
--initial-advertise-peer-urls=https://172.20.0.11:2380 \
--listen-peer-urls=https://172.20.0.11:2380 \
--listen-client-urls=https://172.20.0.11:2379,https://127.0.0.1:2379,https://127.0.0.1:4001 \
--advertise-client-urls=https://172.20.0.11:2379 \
--initial-cluster-token=etcd-token \
--initial-cluster=kmaster1=https://172.20.0.11:2380,kmaster2=https://172.20.0.12:2380,kmaster3=https://172.20.0.13:2380 \
--data-dir=/var/lib/etcd \
--initial-cluster-state=new
Restart=always
RestartSec=5s
User=etcd

[Install]
WantedBy=multi-user.target

Create an etcd user

useradd etcd
chown -R etcd:etcd /var/lib/etcd

Add network configuration on all 3 masters, by running the below.

cat <  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
EOF

# Reload stack
sysctl --system

Start and Enable etcd on all 3 masters.

systemctl enable etcd && systemctl start etcd

# Verify if etcd started.
systemctl status etcd
OR
journalctl -u etcd

Verify if etcd works across all nodes.

etcdctl member list
7c8d40e4de52c1b9: name=kmaster2 peerURLs=https://172.20.0.12:2380 clientURLs=https://172.20.0.12:2379 isLeader=false
96e236888999c3ca: name=kmaster3 peerURLs=https://172.20.0.13:2380 clientURLs=https://172.20.0.13:2379 isLeader=true
9e191dbdf076e744: name=kmaster1 peerURLs=https://172.20.0.11:2380 clientURLs=https://172.20.0.11:2379 isLeader=false

Note: The isLeader=true will automatically get elected/updated based on last elected etcd master (in the case above its kmaster3). Create flannel network key in etcd, by running the below.

/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.20.0.0/20", "SubnetLen": 24, "Backend": { "Type": "vxlan", "VNI": 1 } }

Verify that the key got created.

etcdctl get /coreos.com/network/config
{ "Network": "10.20.0.0/20", "SubnetLen": 24, "Backend": { "Type": "vxlan", "VNI": 1 } }

In Part 3 will continue configuring Flannel and Docker. You might also like – Other related articles to Docker and Kubernetes / micro-service. Like what you’re reading? please provide feedback, any feedback is appreciated.