DevTech101

DevTech101
1 Star2 Stars3 Stars4 Stars5 Stars (3 votes, average: 3.67 out of 5)
Loading...
In the last post I went through how to generate the SSL certificates, below I am continuing with the Etcd configuration. I divided the Kubernetes configuration into parts outlined below (still working progress). Note: An up-to-date example is available on my GitHub project page, or generate your own Kubernetes configuration with the Kubernetes generator available here on my GitHub page.

This is part 2 – Configure Etcd key value store – Flannel networking

Note: The etcd version used in the below config is version 3. In order for Etcd communication to work we will have to copy the certificates create in part 1 since all communication is configured to use SSL.

Etcd configuration

Create the required directory, and copy the CA and etcd certificates. 
mkdir -p /var/lib/etcd/ssl

# certificate files 
ls -l /var/lib/etcd/ssl/
total 24
-rw-r--r--. 1 root root 1269 Oct 19 16:11 ca.pem
-rw-r--r--. 1 root root 1679 Oct 19 16:11 etcd-node-key.pem
-rw-r--r--. 1 root root 1789 Oct 19 16:11 etcd-node.pem
If using the ignition config, add the below to your ignition file. 
    - path: /var/lib/etcd/ssl/ca.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIDfTCCAmWgAwIBAgIJANVKHyHC6mCgMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          ...
          -----END CERTIFICATE-----
    - path: /etc/ssl/certs/ca.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIDfTCCAmWgAwIBAgIJANVKHyHC6mCgMA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          ...
          -----END CERTIFICATE-----
    - path: /var/lib/etcd/ssl/etcd-node.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIEfjCCA2agAwIBAgIJALNU+y2bquY5MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          ...
          -----END CERTIFICATE-----
    - path: /var/lib/etcd/ssl/etcd-node-key.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN RSA PRIVATE KEY-----
          MIIEowIBAAKCAQEAwIANJL2mFBpl3yDcLvfoFv/xS3DGZS7tavSBPzmDAiqzu9Ws
          ...
          -----END RSA PRIVATE KEY-----
To configure etcd, like all other components. you have two options. option one: to use the below systemd configuration files by creating it manually at any time, option two: you can generate at installation an ignition config, by using the CoreOS ignition file. I am including both examples below. Note: Replace with the right ip address on each of the 3 nodes. Manually creating the config files cat /etc/systemd/system/etcd-member.service.d/20-clct-etcd-member.conf 
[Service]
Environment="ETCD_IMAGE_TAG=v3.2.8"
ExecStart=
ExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \
  --name="coreos1" \
  --data-dir="/var/lib/etcd" \
  --listen-peer-urls="https://10.0.2.11:2380" \
  --listen-client-urls="https://10.0.2.11:2379,https://127.0.0.1:2379,https://127.0.0.1:4001" \
  --initial-advertise-peer-urls="https://10.0.2.11:2380" \
  --initial-cluster="coreos1=https://10.0.2.11:2380,coreos2=https://10.0.2.12:2380,coreos3=https://10.0.2.13:2380" \
  --initial-cluster-state="new" \
  --initial-cluster-token="etcd-token" \
  --advertise-client-urls="https://10.0.2.11:2379" \
  --auto-compaction-retention=1 \
  --cert-file="/var/lib/etcd/ssl/etcd-node.pem" \
  --key-file="/var/lib/etcd/ssl/etcd-node-key.pem" \
  --client-cert-auth=true \
  --trusted-ca-file="/var/lib/etcd/ssl/ca.pem" \
  --peer-cert-file="/var/lib/etcd/ssl/etcd-node.pem" \
  --peer-key-file="/var/lib/etcd/ssl/etcd-node-key.pem" \
  --peer-client-cert-auth=true \
  --peer-trusted-ca-file="/var/lib/etcd/ssl/ca.pem"
cat /etc/systemd/system/etcd-member.service.d/conf1.conf 
[Service]
Environment="ETCD_SSL_DIR=/var/lib/etcd/ssl"
Using the ignition config at install time Add the below to your ignition config. 
systemd:
  units:
    - name: etcd-member.service
      dropins:
        - name: conf1.conf
          contents: |
            [Service]
            Environment="ETCD_SSL_DIR=/var/lib/etcd/ssl"

etcd:
  version:                     3.2.8
  name:                        coreos1
  data_dir:                    /var/lib/etcd
  listen_client_urls:          https://10.0.2.11:2379,https://127.0.0.1:2379,https://127.0.0.1:4001
  advertise_client_urls:       https://10.0.2.11:2379
  listen_peer_urls:            https://10.0.2.11:2380
  initial_advertise_peer_urls: https://10.0.2.11:2380
  initial_cluster:             coreos1=https://10.0.2.11:2380,coreos2=https://10.0.2.12:2380,coreos3=https://10.0.2.13:2380
  initial_cluster_token:       etcd-token
  initial_cluster_state:       new
  cert_file:                   /var/lib/etcd/ssl/etcd-node.pem
  key_file:                    /var/lib/etcd/ssl/etcd-node-key.pem
  peer_cert_file:              /var/lib/etcd/ssl/etcd-node.pem
  peer_key_file:               /var/lib/etcd/ssl/etcd-node-key.pem
  client_cert_auth:            true
  peer_client_cert_auth:       true
  trusted_ca_file:             /var/lib/etcd/ssl/ca.pem
  peer_trusted_ca_file:        /var/lib/etcd/ssl/ca.pem
  auto_compaction_retention:   1
Loading / enabling the service. 
systemctl daemon-reload
systemctl enable etcd-member
systemctl start etcd-member
systemctl status etcd-member
To verify cluster health just run the below. 
etcdctl cluster-health
## Without any ETCDCTL_ set
# etcdctl --endpoints=https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379 --cert-file=/var/lib/etcd/etcd-node.pem --key-file=/var/lib/etcd/etcd-node-key.pem cluster-health
member 829c4dcf6567e22f is healthy: got healthy result from https://10.0.2.13:2379
member 8ad2e1df4dc66f9a is healthy: got healthy result from https://10.0.2.12:2379
member b12eaa0af14319e0 is healthy: got healthy result from https://10.0.2.11:2379
cluster is healthy
Tip: You can omitted the endpoints and certificates if the environment variables are set properly. Just add the below to your /etc/environment. 
ETCDCTL_ENDPOINTS=https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379
ETCDCTL_KEY_FILE=/var/lib/etcd/ssl/etcd-node-key.pem
ETCDCTL_CERT_FILE=/var/lib/etcd/ssl/etcd-node.pem
Or add to your ignition config. Note: The below config contains extra variables, only the ETCDCTL_ are needed. 
    - path: /etc/environment
      filesystem: root
      mode: 0644
      contents:
        inline: |
          HTTP_PROXY=http://proxy:port
          HTTPS_PROXY=http://proxy:port
          http_proxy=http://proxy:port
          https_proxy=http://proxy:port
          no_proxy=localhost,127.0.0.0/8,127.0.0.1,::1,10.0.2.11,10.0.2.12,10.0.2.13,coreos1,coreos2,coreos3,coreos1.domain.com,coreos2.domain.com,coreos3.domain.com,/var/run/docker.sock
          COREOS_PRIVATE_IPV4=10.0.2.11
          ADVERTISE_IP=10.0.2.11
          ETCDCTL_CA_FILE=/etc/ssl/certs/ca.pem
          ETCDCTL_CERT_FILE=/var/lib/etcd/ssl/etcd-node.pem
          ETCDCTL_KEY_FILE=/var/lib/etcd/ssl/etcd-node-key.pem
          ETCDCTL_ENDPOINTS="https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379"
          FLANNELD_IFACE="enp0s3"
          FLANNELD_ETCD_ENDPOINTS=${ETCDCTL_ENDPOINTS}
          FLANNELD_ETCD_PREFIX="/coreos.com/network"
          ETCD_ENDPOINTS=${ETCDCTL_ENDPOINTS}

Flannel configuration

Same as etcd flannel can also be configured manually or with an ignition config. Below is the ignition config 
systemd:
  units:
    - name: flanneld.service
      dropins:
        - name: 50-network-config.conf
          contents: |
            [Service]
            EnvironmentFile=/etc/environment
            ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.0.0.0/21", "SubnetLen": 24, "Backend": { "Type": "vxlan", "VNI": 1 } }'
            # VXlan/VNI/VLan example
            #ExecStartPre=/usr/bin/etcdctl ${CA} ${CERT} ${KEY} ${EP} set /coreos.com/network/red-db/config '{ "Network": "10.220.0.0/16", "SubnetLen": 24, "SubnetMin": "10.220.110.0", "SubnetMax": "10.220.200.0", "Backend": { "Type": "vxlan", "VNI": 6 } }'
            #ExecStartPre=/usr/bin/etcdctl ${CA} ${CERT} ${KEY} ${EP} set /coreos.com/network/red-app/config '{ "Network": "10.220.0.0/16", "SubnetLen": 24, "SubnetMin": "10.220.110.0", "SubnetMax": "10.220.200.0", "Backend": { "Type": "vxlan", "VNI": 5 } }'
            # This is needed for rkt to start due to some bug
            ExecStartPre=/usr/bin/rkt image fetch coreos.com/rkt/stage1-coreos:1.29.0
            Restart=always

    - name: flanneld.service
      dropins:
        - name: 50-ssl.conf
          contents: |
            [Service]
            Environment="ETCD_SSL_DIR=/var/lib/etcd/ssl"

storage:
  files:
    - path: /etc/systemd/system/flanneld.service.d/40-ExecStartPre-symlink.conf
      filesystem: root
      mode: 0644
      contents:
        inline: |
          [Service]
          ExecStartPre=/usr/bin/ln -sf /etc/flannel/options.env /run/flannel/options.env
          ExecStartPre=/usr/bin/ln -sf /etc/kubernetes/cni/net.d /etc/rkt/net.d

flannel:
  version:        0.9.0-30
  etcd_prefix:    "/coreos.com/network"
  etcd_endpoints: "https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379"
  etcd_cafile:    /etc/ssl/certs/ca.pem
  etcd_certfile:  /var/lib/etcd/ssl/etcd-node.pem
  etcd_keyfile:   /var/lib/etcd/ssl/etcd-node-key.pem
  interface:          enp0s3
To create the flannel config manually, create the below files. /etc/systemd/system/flanneld.service.d/20-clct-flannel.conf 
[Service]
Environment="FLANNEL_IMAGE_TAG=v0.9.0"
ExecStart=
ExecStart=/usr/lib/coreos/flannel-wrapper $FLANNEL_OPTS \
  --etcd-endpoints="https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379" \
  --etcd-cafile="/etc/ssl/certs/ca.pem" \
  --etcd-certfile="/var/lib/etcd/ssl/etcd-node.pem" \
  --etcd-keyfile="/var/lib/etcd/ssl/etcd-node-key.pem" \
  --etcd-prefix="/coreos.com/network" \
  --iface="enp0s3"
/etc/systemd/system/flanneld.service.d/40-ExecStartPre-symlink.conf 
[Service]
ExecStartPre=/usr/bin/ln -sf /etc/flannel/options.env /run/flannel/options.env
ExecStartPre=/usr/bin/ln -sf /etc/kubernetes/cni/net.d /etc/rkt/net.d
/etc/systemd/system/flanneld.service.d/50-network-config.conf 
[Service]
EnvironmentFile=/etc/environment
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.0.0.0/21", "SubnetLen": 24, "Backend": { "Type": "vxlan",
 "VNI": 1 } }'
ExecStartPre=/usr/bin/rkt image fetch coreos.com/rkt/stage1-coreos:1.29.0
Restart=always
/etc/systemd/system/flanneld.service.d/50-ssl.conf 
[Service]
Environment="ETCD_SSL_DIR=/var/lib/etcd/ssl"
Loading / enabling the service. 
systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
systemctl status flanneld
To verify flannel working just check the etcd configuration 
# Network config
etcdctl get /coreos.com/network/config
{ "Network": "10.0.0.0/21", "SubnetLen": 24, "Backend": { "Type": "vxlan", "VNI": 1 } }
# Subnet config
etcdctl get /coreos.com/network/subnets/10.0.1.0-24
{"PublicIP":"10.0.2.11","BackendType":"vxlan","BackendData":{"VtepMAC":"8e:45:24:df:5a:82"}}
You are now ready to move to the next step, Configuring Kubernetes manifests – in part 3. You might also like – Other articles related to Docker Kubernetes / micro-service.
Like what you’re reading? please provide feedback, any feedback is appreciated.
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x
%d bloggers like this: