(2 votes, average: 3.00 out of 5)
Loading...Configuring a 3 node(master) Kubernetes cluster
Below I am going to describe the process to install/configure a 3 node Kubernetes cluster. the configuration will contain 3 Kubernetes master nodes (i.e. controller, api, scheduler, proxy), I will also add 1 worker node(additional workers can be added as needed).Note: Most of the examples below are still relevant for the overall kubernetes configuration. However, some of the YAML file examples below are outdated and will only work with kubernetes v1.8.x. For a copy of the most recent CoreOS examples kubernetes version v.1.13.x+, is available on my GitHub repository by going here. You can Also generate your own by using the generator also available on my GitHub repository by going here. A more up-to-date write up is also available Installing a 3 master node Kubernetes cluster on CentOs.
Why not use Tectonic? Two reasons for that.- While Tectonic is great, I am still learning how Kubernetes works, and the only way to understand is, by creating your own cluster from scratch.
- Tectonic comes with a 10 node license limitation.
- Part 1: Initial setup – getting CoreOS, prepare SSL certificates, etc.
- Part 2: Configure Etcd key value store, Configure Flannel.
- Part 3: Configure Kubernetes manifests for controller, api, scheduler and proxy.
- Part 4: Finalize the kubelet configuration to use RKT and Flannel+CNI.
- Part 5: Optional – configure Ingress, kube-dns and kube-dashboard.
- Part 6: Automate the Kubernetes deployment, etcd, kubelet, rkt, flannel, cni and ssl.
This is part 1 – Initial setup – getting CoreOS and prepare SSL certificates
What items, versions are in the configuration below, and what not. Items versions used.- Openssl or Cfssl to generate certificates.
- CoreOS: 1576.1.0
- Kubernetes/kubelet/kubectl: 1.8.1
- Etcd: version 3
- Flannel: version 0.90-0.30
- Default Container engine Rkt: version 1.29.0 (no Docker is being used)
- Use SSL for all communication if possible
- Matchbox configuration
OS configuration – CoreOS preparation
Lets start with the OS, I opted in using CoreOS which can be download form here – CoreOS ISO images. The reason for selecting CoreOS is obvious, CoreOS runs the process in a container(rkt) and has out of the box excellent Kubernetes support plus many more great features. A few notes about recent CoreOS development.- CoreOS used to have this so called cloud-config configuration file, in the recent versions they moved to something they call container linux config (ignition config). If you are migrating from a cloud config to a container linux config, you might find migrating to ignition and ignition config refrence helpful.
- To create an ignition config you will need the ct utility, which you can download from here and a helpfull video how to use it from here
- The ct utility (transpiler) will generate an ignition config which you can then use to configure your nodes.
ct -in-file n1-ign.yaml -platform vagrant-virtualbox -out-file n1-ign.ign ct -in-file n2-ign.yaml -platform vagrant-virtualbox -out-file n2-ign.ign ct -in-file n3-ign.yaml -platform vagrant-virtualbox -out-file n3-ign.ignNote: You can skip and jump directly to a complete ct/ignition ready files for a 3 Node cluster. Click here for Node 1, Node 2 and Node 3. In my configuration below I am using 3 Virtualbox VM’s, to see how to configure those VM’s you can read this.. Note: that using physical hardware would apply the same or similar process and can be fully automated with matchbox. Last, CoreOS has also a project called matchbox, matchbox is an HTTP and gRPC service that renders signed Ignition configs, cloud-configs, network boot configs, and metadata to machines to create Container Linux clusters. matchbox helps fully automate the Ignition config. For more details you can check out CoreOS matchbox documentation here.
Prepare SSL certificates by using Openssl or Cfssl
Since Several components of Kubernetes require SSL, to simplify the certificate creation I created only two certificates.- CA certificate
- General certificate – used for all components
- Optional create worker certificates
| CoreOS Cluster IP Address | |
|---|---|
| Name | IP Addrss |
| coreos1 | 10.0.2.11/24 |
| coreos2 | 10.0.2.12/24 |
| coreos3 | 10.0.2.13/24 |
Generating the Certificates by using Cfssl
Cfssl Reference Since many OS’s do not include the cfssl application, the installation steps are below.
# Use the below apt-get
apt-get install cfssl
# Or use the below
mkdir ~/bin
curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x ~/bin/{cfssl,cfssljson}
export PATH=$PATH:~/bin
# default json files for ca and client (I modified the files below)
cfssl print-defaults config > ca-config.json
cfssl print-defaults csr > ca-csr.json
Now, create the ca-config.json json files. (I modified the expire time)
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"server": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"etcd-node": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
},
"etcd-proxy": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
Next, create the ca-csr.json file – used for CA configuration.
{
"CN": "Etcd",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "US",
"L": "New York",
"O": "Company1",
"OU": "Ops",
"ST": "NY"
}
]
}
Next, create the etcd-node-csr.json – used for etcd and all other components.
Note: Replace with your domain
{
"CN": "etcd-node",
"hosts": [
"0.0.0.0",
"10.0.2.11",
"10.0.2.12",
"10.0.2.13",
"10.0.2.1",
"10.0.2.2",
"10.3.0.1",
"127.0.0.1",
"localhost",
"coreos1",
"coreos2",
"coreos3",
"coreos1.local",
"coreos2.local",
"coreos3.local",
"coreos1.domain.com",
"coreos2.domain.com",
"coreos3.domain.com",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster.local",
"kube-apiserver",
"kube-admin",
"domain.com"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "US",
"L": "New York",
"O": "Company1",
"OU": "Ops",
"ST": "NY"
}
]
}
Create the below if you would like to use a separate key for the kub-proxy communication.
Note: Replace with your domain
{
"CN": "etcd-proxy",
"hosts": [
"0.0.0.0",
"10.0.2.11",
"10.0.2.12",
"10.0.2.13",
"10.0.2.1",
"10.0.2.2",
"10.3.0.1",
"127.0.0.1",
"localhost",
"coreos1",
"coreos2",
"coreos3",
"coreos1.local",
"coreos2.local",
"coreos3.local",
"coreos1.domain.com",
"coreos2.domain.com",
"coreos3.domain.com",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster.local",
"kube-apiserver",
"kube-admin",
"domain.com"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "US",
"L": "New York",
"O": "Company1",
"OU": "Ops",
"ST": "NY"
}
]
}
Now, just run the below which will generate the certificate and keys used latter in the kubernetes configuration.
# Creat CA cfssl gencert -initca ca-csr.json | cfssljson -bare ca # Create Node Certficate cfssl gencert \ -ca=ca.pem \ -ca-key=ca-key.pem \ -config=ca-config.json \ -profile=etcd-node \ etcd-node-csr.json | cfssljson -bare etcd-node cfssl gencert \ -ca=ca.pem \ -ca-key=ca-key.pem \ -config=ca-config.json \ -profile=etcd-proxy \ etcd-proxy-csr.json | cfssljson -bare etcd-proxyYou should now have all the files below.
# ls -l -rw-r--r-- 1 root root 1102 Oct 17 15:04 ca-config.json -rw-r--r-- 1 root root 256 Oct 17 15:04 ca-csr.json -rw------- 1 root root 1675 Oct 17 16:23 ca-key.pem -rw-r--r-- 1 root root 989 Oct 17 16:23 ca.csr -rw-r--r-- 1 root root 1285 Oct 17 16:23 ca.pem -rw-r--r-- 1 root root 879 Oct 17 16:22 etcd-node-csr.json -rw------- 1 root root 1675 Oct 17 16:23 etcd-node-key.pem -rw-r--r-- 1 root root 1476 Oct 17 16:23 etcd-node.csr -rw-r--r-- 1 root root 1826 Oct 17 16:23 etcd-node.pem -rw-r--r-- 1 root root 880 Oct 17 16:22 etcd-proxy-csr.json -rw------- 1 root root 1675 Oct 17 16:23 etcd-proxy-key.pem -rw-r--r-- 1 root root 1476 Oct 17 16:23 etcd-proxy.csr -rw-r--r-- 1 root root 1830 Oct 17 16:23 etcd-proxy.pem -rwxr-xr-x 1 root root 404 Oct 17 15:03 run.sh
Generating the Certificates by using OpenSSL
Since most linux distros include Openssl, I am including steps to use that below. Create your config file cert.conf Note: Replace with your domain[req] default_bits = 2048 prompt = no default_md = sha256 distinguished_name = dn req_extensions = v3_req x509_extensions = v3_ca [ dn ] C = US ST = NY L = New York O = Company1 OU = Ops CN = etcd-node [ v3_ca ] keyUsage = critical,keyCertSign, cRLSign basicConstraints = critical,CA:TRUE subjectKeyIdentifier = hash [ v3_req ] keyUsage = critical,digitalSignature, keyEncipherment, nonRepudiation extendedKeyUsage = clientAuth, serverAuth basicConstraints = critical,CA:FALSE subjectKeyIdentifier = hash subjectAltName = @alt_names [ alt_names ] DNS.1 = domain.com DNS.2 = coreos1.domain.com DNS.3 = coreos2.domain.com DNS.4 = coreos3.domain.com DNS.5 = localhost DNS.6 = coreos1 DNS.7 = coreos2 DNS.8 = coreos3 DNS.9 = coreos1.local DNS.10 = coreos2.local DNS.11 = coreos3.local DNS.12 = kubernetes DNS.13 = kubernetes.default DNS.14 = kubernetes.default.svc DNS.15 = kubernetes.default.svc.cluster.local DNS.16 = kube-apiserver DNS.17 = kube-admin IP.1 = 127.0.0.1 IP.2 = 10.0.2.11 IP.3 = 10.0.2.12 IP.4 = 10.0.2.13 IP.5 = 10.3.0.1 IP.6 = 10.0.2.1 IP.7 = 10.0.2.2 IP.8 = 0.0.0.0 email = admin@domain.comRun the below to generate the certificates CA and general.
# Generate the CA private key. openssl genrsa -out ca-key.pem 2048 # Generate the CA certificate. sed -i 's/^CN.*/CN = Etcd/g' cert.conf openssl req -x509 -new -extensions v3_ca -key ca-key.pem -days 3650 \ -out ca.pem \ -subj '/C=US/ST=New York/L=New York/O=domain.com/CN=Etcd' \ -config cert.conf # Generate the server/client private key. openssl genrsa -out etcd-node-key.pem 2048 # Generate the server/client certificate request. sed -i 's/^CN.*/CN = etcd-node/g' cert.conf openssl req -new -key etcd-node-key.pem \ -newkey rsa:2048 -nodes -config cert.conf \ -subj '/C=US/ST=New York/L=New York/O=domain.com/CN=etcd-node' \ -outform pem -out etcd-node-req.pem -keyout etcd-node-req.key # Sign the server/client certificate request. openssl x509 -req -in etcd-node-req.pem -CA ca.pem -CAkey ca-key.pem -CAcreateserial \ -out etcd-node.pem -days 3650 -extensions v3_req -extfile cert.confYou should now have the below files.
-rw-r--r-- 1 root root 1679 Oct 17 16:20 ca-key.pem -rw-r--r-- 1 root root 1269 Oct 17 16:20 ca.pem -rw-r--r-- 1 root root 17 Oct 17 16:20 ca.srl -rw-r--r-- 1 root root 1569 Oct 17 16:20 cert.conf -rw-r--r-- 1 root root 1679 Oct 17 16:20 etcd-node-key.pem -rw-r--r-- 1 root root 1622 Oct 17 16:20 etcd-node-req.pem -rw-r--r-- 1 root root 1789 Oct 17 16:20 etcd-node.pem -rwxr-xr-x 1 root root 925 Oct 17 16:17 run.shYou are now ready to move to the next step, configuring Etcd – in part 2. You might also like – Other articles related to Docker Kubernetes / micro-service.
Like what you’re reading? please provide feedback, any feedback is appreciated.
Good Morning
Would it be possible to provide an example of “3.Optional create worker certificates”
This guide has been really useful in understanding how to secure K8s in my own test environment, had been struggling to find any relevant information on-line previously.
Sure.
I hope to have an updated post (Part 7 – Enabling / Configuring RBAC, TLS Node bootstrapping, Installing / Configuring Helm), please check back latter today or tomorrow.
The update will include how to bring on-line a new Node including SSL/TLS (auto) Node Bootstraping, which I believe is what you are looking for.