CoreOS - Container Linux / 3 Node Linux Sample Configs - Node 3

(1 votes, average: 1.00 out of 5)

CoreOS Ignition Configuration Files

In my configuration below I am using a Virtualbox VM, to see how to configure those VM’s you can read this..

Note: The YAML files below are outdated and will only work with kubernetes v1.8.0. For a copy of the most recent examples kubernetes version v.1.13.x+, is available on my GitHub repository by going here. You can Also generate your own by using the generator also available on my GitHub repository by going here.

Note: An up-to-date example is available on my GitHub project page, or generate your own Kubernetes configuration with the Kubernetes generator available here on my GitHub page. You can find at the end of this post the Node 3 ignition Container Linux Config. To generate the final ignition/json file run the below.

ct -in-file n3-ign.yaml -platform vagrant-virtualbox -out-file n3-ign.ign

You then use the n3-ign.ign file to configure / install CoreOS. Boot the VM with the CoreOS ISO, then copy the ign file to the VM and run the below to install the OS to disk with ign config.

export http_proxy=proxy:port
export https_proxy=proxy:port

coreos-install -d /dev/sda -C alpha -i n3-ign.ign 

# Unmount the CD and reboot to CoreOS 
reboot

General Notes: Make sure to replace the certificates with your generated certificates, Replace the ssh-keys with your keys. Replace the users password with yours. Replace or remove the proxy with yours. Notes on the below configuration

If you are re-joining an existing etcd cluster node (i.e. you are re-using the node name/ip), follow the below process to first remove the node from ethcd, then re-add the node.
The ct utility generates an ignition config which also adds to the etcd-member service a Unit with a dependency on coreos-metadata.service.
The issue is on a bear-metal install the coreos-metadata.service wont start causing etcd-member service to wait for the dependency.
To get around this issue, remove the Unit and the 2 top lines from the etcd-member.service.d/20-clct-etcd-member.conf. like the below.
Remove the 3 top lines.
```
[Unit]
Requires=coreos-metadata.service
After=coreos-metadata.service

[Service]
[..] snip
```
Make sure to disable the coreos-metadata.service with systemcctl disable coreos-metadata.service (if its enabled)
(Optional) Copy the kubectl to /opt, by running the below.
```
mkdir /opt; cd /opt; scp -r user@coreos2:/opt/bin .
```
Lastly, Enable the kubelet service with systemctl enable kubelet. then reboot now everything should come up clean.

Note: If you know a workaround how to disable the coreos-metadata.service dependency being added to the etcd-member service, I would appreciate if you let me know in the comments below. To re-join an existing Etcd Memeber First get the member name list, by running the below.

etcdctl member list
1d9b68db3dfbbb61: name=coreos3 peerURLs=https://10.0.2.13:2380 clientURLs=https://10.0.2.13:2379 isLeader=false
8ad2e1df4dc66f9a: name=coreos2 peerURLs=https://10.0.2.12:2380 clientURLs=https://10.0.2.12:2379 isLeader=false
b12eaa0af14319e0: name=coreos1 peerURLs=https://10.0.2.11:2380 clientURLs=https://10.0.2.11:2379 isLeader=true

Next, remove the memebr, in the below example its coreos3

etcdctl member remove 1d9b68db3dfbbb61 
Removed member 1d9b68db3dfbbb61 from cluster

Now, re-add the member to the cluster.

etcdctl member add coreos3 https://10.0.2.13:2380
Added member named coreos3 with ID 805576726f49436d to cluster

ETCD_NAME="coreos3"
ETCD_INITIAL_CLUSTER="coreos3=https://10.0.2.13:2380,coreos2=https://10.0.2.12:2380,coreos1=https://10.0.2.11:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

At the next coreos3 Etcd startup it will re-join the cluster cleanly. Note: Make sure to change the etcd-member config from “new” to “existing” (i.e. –initial-cluster-state=”existing”).

Node 3 full ignition config

---
storage:
  files:
    - filesystem: "root"
      path:       "/etc/metadata/coreos"
      mode:       0644
      contents:
        inline: |
          COREOS_VAGRANT_VIRTUALBOX_PRIVATE_IPV4=10.0.2.13
          COREOS_VAGRANT_VIRTUALBOX_HOSTNAME=coreos3
    - filesystem: "root"
      path:       "/etc/hostname"
      mode:       0644
      contents:
        inline: coreos3
    - filesystem: "root"
      path:       "/etc/hosts"
      mode:       0644
      contents:
        inline: |
          127.0.0.1	localhost
          ::1		localhost
          10.0.2.11 coreos1 coreos1.domain.com
          10.0.2.12 coreos2 coreos2.domain.com
          10.0.2.13 coreos3 coreos3.domain.com
    - path: /etc/docker/ca.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIDfTCCAmWgAwIBAgIJAPgnITmJUxM1MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          BAYTAlVTMQswCQYDVQQIDAJOWTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoM
          B0JIUGhvdG8xDDAKBgNVBAsMA09wczENMAsGA1UEAwwERXRjZDAeFw0xNzEwMTcy
          MDIwNDBaFw0yNzEwMTUyMDIwNDBaMFwxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJO
          WTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoMB0JIUGhvdG8xDDAKBgNVBAsM
          A09wczENMAsGA1UEAwwERXRjZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
          ggEBAJ/zTXlvdaRWH38f8z33RG3CWuY7zFr3q6H8dwEpM8KGab5lZ8SOLpkrBOS7
          8CXZnvoCXh5NigLZJMCCNkV/s3OUeu5IkxMVflS5rNh9bMLoGgKGSS/9jw46ohyD
          /o3XKg9hsrFrQRSizTMxElD/Ccr73gIJzF74sLdyQpt/a7BwRdVjb42jafQrQw88
          9YGNeZpgCjAZ5fkzt+JW+ztkysd8eCXI/Une5Ap8+pZPt+9ayzvzA2DmpTJRH4Zs
          A4OmktDBTwG5hyMePZvrnrafvxeb5x73e72h0J6yRfqBNWqyIu/9oOJoqaevbTNP
          yqtB2zC9huIoV4MZYKkabM9x/ycCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8G
          A1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFOlIxdaKQKPrVAGf2B/aW7++0lPRMA0G
          CSqGSIb3DQEBCwUAA4IBAQBK7741peD9+fFtFhoSUPscl76nxLdXlKBJis7ytBuj
          w3itwOQFsTEE4vzB/jyTHHyBLfr+AmtjZa4JDATDtwhqxXowGymZ9KrDWEsOHzET
          q4edrl9/2EIDFPN5uWi/wue4AZy8f7PiKluStiXfYH49lIwtIvO0jiD+zCGsBWSM
          BxOHNDBP+XYkvBDDRIl7Uu79Djw5DICiP4vT1oGUnMM9/cYDBBVyl+lU54rueJVv
          g4PoOmHaeIPquIqxB5Aqc9m35mbutq4ZpQ65F2VPn1ejXI8jZnqUdOPjLZ1e2Wnx
          ym+NKgfOIgSQjA53hBgOnnFHJEr0S7e/PRvcvjdBdj12
          -----END CERTIFICATE-----
    - path: /etc/docker/server.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIE+zCCA+OgAwIBAgIJANxo/z6xfMb4MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          BAYTAlVTMQswCQYDVQQIDAJOWTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoM
          B0JIUGhvdG8xDDAKBgNVBAsMA09wczENMAsGA1UEAwwERXRjZDAeFw0xNzEwMTcy
          MDIwNDBaFw0yNzEwMTUyMDIwNDBaMGExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJO
          WTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoMB0JIUGhvdG8xDDAKBgNVBAsM
          A09wczESMBAGA1UEAwwJZXRjZC1ub2RlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
          MIIBCgKCAQEAodQQOS9mI42MH8Qk+D280lcBwZAqQQxwI/TMNhkjdd9S/wiCiTTz
          yogkaPVSMceidxSehHVkm2N0+LSFkG9yiLSVBBGjoJ+LR+wlECKiV4a1zNSgAn5/
          17SqTuzEdb6pSE59jKliMlwjCSzlRpsAT6Z1YNMrHqkBwRVSYnxWGibc0BJcCGCs
          Hbb13UnYERC5ce0MuXXX3nUJQXKYvrnBLYpWM5M5PN43VxwXCajBcFQKDngDbnmu
          8oIoWgxan7trAfgYDLo2/j7uqFinj9oqwSYicND1zanMc+XFZTHqc8copdQRKGMY
          b/kfNTNb3ACzZyqm45Cax4QcjtoFNjidTQIDAQABo4IBuTCCAbUwDgYDVR0PAQH/
          BAQDAgXgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAMBgNVHRMBAf8E
          AjAAMB0GA1UdDgQWBBTJdhrvi+oTQoJD/EjhjXK4DJmRRTCCAVUGA1UdEQSCAUww
          ggFIggdibmguY29tgg9jb3Jlb3MxLmJuaC5jb22CD2NvcmVvczIuYm5oLmNvbYIP
          Y29yZW9zMy5ibmguY29tgglsb2NhbGhvc3SCB2NvcmVvczGCB2NvcmVvczKCB2Nv
          cmVvczOCDWNvcmVvczEubG9jYWyCDWNvcmVvczIubG9jYWyCDWNvcmVvczMubG9j
          YWyCCmt1YmVybmV0ZXOCEmt1YmVybmV0ZXMuZGVmYXVsdIIWa3ViZXJuZXRlcy5k
          ZWZhdWx0LnN2Y4Ika3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2Fs
          gg5rdWJlLWFwaXNlcnZlcoIKa3ViZS1hZG1pbocEfwAAAYcECgACC4cECgACDIcE
          CgACDYcECgMAAYcECgACAYcECgACAocEAAAAAIENYWRtaW5AYm5oLmNvbTANBgkq
          hkiG9w0BAQsFAAOCAQEAIyama2mrV4vzcF0Tsw1NghE9GdAx0WpmfFStEC5GrgBH
          +tVjuaqLYIGW8DC7+EIFLq5obQCeXIMZ1OQoQFegfv0X8XwsDFQeaNs15N6jYZ8S
          7Yxe4mFvCecvVfqdc5M4XxykoRQQbIzBBI7AJ5zumTB6c6OZ2l8cRyNPYooj3xHt
          XqHnugDX4LMyBLRxRYyatF9Evku4VnqbawWPpejpfcfGiwyx/ArNNmDgsOikvYYB
          ndmXqiYpBAeatW65kwvIWuGoo4HT3K3BExP0aVxrddAFL8Q35DRxTU6tTM3k3q3b
          gv2CMfTju+IUHyqnqcWy7vIcKwQrYkmfKdjGB4mjzw==
          -----END CERTIFICATE-----
    - path: /etc/docker/server-key.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN RSA PRIVATE KEY-----
          MIIEpQIBAAKCAQEAodQQOS9mI42MH8Qk+D280lcBwZAqQQxwI/TMNhkjdd9S/wiC
          iTTzyogkaPVSMceidxSehHVkm2N0+LSFkG9yiLSVBBGjoJ+LR+wlECKiV4a1zNSg
          An5/17SqTuzEdb6pSE59jKliMlwjCSzlRpsAT6Z1YNMrHqkBwRVSYnxWGibc0BJc
          CGCsHbb13UnYERC5ce0MuXXX3nUJQXKYvrnBLYpWM5M5PN43VxwXCajBcFQKDngD
          bnmu8oIoWgxan7trAfgYDLo2/j7uqFinj9oqwSYicND1zanMc+XFZTHqc8copdQR
          KGMYb/kfNTNb3ACzZyqm45Cax4QcjtoFNjidTQIDAQABAoIBAFkzyMGM1BNS6eN0
          viAr/9I97ZThv7jdf08CmDJVxwspT9mOuk2hZ6TDSLamS2lvLqnPLtW60/snZonV
          EC6ekBa029NDWdunNiIuE/wB61Stx7ahmZXmXtfrGVmp/10b/LoDsgcAM0WpLhLX
          sYYq2JAPJHJ71C/xXysscp4hBWEiHDUx0Uc9Cc4dDAJyICgmxsYEk86NqxX5oekw
          FpKhKWCVOp+7wEzKHbNA3F2L5qRZ1QKKJKZH3mIO/TkNmbHkVM/V949PHsruLDP4
          SeArZV8PEfe3BZwc7zpSNwa/ZNjt/ZuwW2hL2lHrxpPT2USAn8TwhD/HZBYLWUWD
          RQ2hQ40CgYEA1idEi1BfCb6RNQmXoyAkaJL/SewPnhyadjO8tZMTzN3BkLM5pM+W
          a536sj9Cu7HpyphvvVy24JPfZ6EZjtCyWIJWk+mZhHG/UcQyEa5X5Kc3cF6hG3Rg
          m9jgtdDizo3mM64u8x8ScaDaomkO8sA/djCMjIwa1EIVS8/N6npOc1cCgYEAwXNO
          yU5PFRZBuQkOedWKlFCF37/yHaP41akwPNLptVfAlcnHErnPz4pI4BX5Zi6pcyst
          LcuLUGhVpH8hgUw35uiFFRGG8NpfUZbIq0sGZ6yHHnmOz06aSRD8VbxqaLhw82y+
          HiqZOFpyOaYFWYlwxuoRL7gzU4aBxtN+vvNJUfsCgYEAiAAgE6lS8clIRjp1Rm9f
          L9pKrhNwR6U6ot5wdY+V/4DFMLPhY71Bdc66KwDpzHFMXBE6xTNgIHdIZXN127DF
          0lfTy9OKXwYtziue8IOb5eYb2wz9m4Xo5JBzBIRHxY1wKFnyDwwolmjk8Edv6tIU
          e2aRm6BOWYYuKBssVkCeGbkCgYEAut70YKnw4uHZIzGIdlC9yX/vkrEM5W0cVulU
          26yybqcSNp/wWjp8cHOY6j7P6NFyhvpSbkaNkWL4OCtG5mwGJFzBEhUYUVnhciNC
          PZX/3e5fqCSsu72mdKaf2wJjmD1zfvKwxNtJn5JffSjPSirmcoBxkbNSgYERpEtx
          yGujjDkCgYEAmh17ruky/BcEbnRF7sZoc1wBdpgG/m/rOqlEBgf08omEyyYy64he
          X4JpHBgSWR+RikcW3KQze4mzdClKtXFnpeZjSVYUkyOGMd15X9r86f7yTXDH+yaT
          aDxeHW+J8lVnfw2xlU8BVh3kcFRDZFbKPQnv7oAfilr/81t4Z+JBWiM=
          -----END RSA PRIVATE KEY-----
    - path: "/etc/motd"
      filesystem: root
      mode: 0644
      contents:
        inline: | 
          Just anther day!!
    - path: /etc/environment
      filesystem: root
      mode: 0644
      contents:
        inline: |
          COREOS_VAGRANT_VIRTUALBOX_PRIVATE_IPV4=10.0.2.13
          COREOS_VAGRANT_VIRTUALBOX_HOSTNAME=coreos3
          ADVERTISE_IP=10.0.2.13
          COREOS_PRIVATE_IPV4=10.0.2.13
          ETCDCTL_SSL_DIR=/var/lib/etcd/ssl
          ETCD_SSL_DIR=/var/lib/etcd/ssl
          ETCDCTL_CA_FILE=/etc/ssl/certs/ca.pem
          ETCDCTL_CERT_FILE=/var/lib/etcd/ssl/etcd-node.pem
          ETCDCTL_KEY_FILE=/var/lib/etcd/ssl/etcd-node-key.pem
          ETCDCTL_ENDPOINTS="https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379"
          ETCD_ENDPOINTS="https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379"
          FLANNELD_ETCD_ENDPOINTS="https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379"
          FLANNELD_IFACE="enp0s3"
          FLANNELD_ETCD_PREFIX="/coreos.com/network"
          kubernetes_version=1.8.0
          LOCKSMITHCTL_ETCD_CAFILE=/etc/ssl/certs/ca.pem
          LOCKSMITHCTL_ETCD_CERTFILE=/var/lib/etcd/ssl/etcd-node.pem
          LOCKSMITHCTL_ETCD_KEYFILE=/var/lib/etcd/ssl/etcd-node-key.pem
          LOCKSMITHCTL_ENDPOINT="https://10.0.2.13:2379"
          HTTP_PROXY=http://proxy:port
          HTTPS_PROXY=http://proxy:port
          http_proxy=http://proxy:port
          https_proxy=http://proxy:port
          no_proxy=localhost,127.0.0.0/8,127.0.0.1,::1,10.0.2.11,10.0.2.12,10.0.2.13,coreos1,coreos2,coreos3,coreos1.domain.com,coreos2.domain.com,coreos3.domain.com,/var/run/docker.sock
    - path: /etc/systemd/system.conf.d/10-default-env.conf
      filesystem: root
      mode: 0644
      contents:
        inline: |
          [Manager]
          DefaultEnvironment=http_proxy=http://proxy:port
          DefaultEnvironment=https_proxy=http://proxy:port
          DefaultEnvironment=HTTP_PROXY=http://proxy:port
          DefaultEnvironment=HTTPS_PROXY=http://proxy:port
          DefaultEnvironment="no_proxy=localhost,127.0.0.0/8,127.0.0.1,::1,10.0.2.11,10.0.2.12,10.0.2.13,coreos1,coreos2,coreos3,coreos1.domain.com,coreos2.domain.com,coreos3.domain.com,/var/run/docker.sock"
    - path: /etc/systemd/system/docker.service.d/environment.conf
      filesystem: root
      mode: 0644
      contents:
        inline: |
          [Service]
          EnvironmentFile=/etc/environment
    - path: /etc/systemd/timesyncd.conf
      filesystem: root
      mode: 0644
      contents:
        inline: |
          [Time]
          NTP=time.domain.com
    - path: /var/lib/etcd/ssl/ca.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIDfTCCAmWgAwIBAgIJAPgnITmJUxM1MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          BAYTAlVTMQswCQYDVQQIDAJOWTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoM
          B0JIUGhvdG8xDDAKBgNVBAsMA09wczENMAsGA1UEAwwERXRjZDAeFw0xNzEwMTcy
          MDIwNDBaFw0yNzEwMTUyMDIwNDBaMFwxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJO
          WTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoMB0JIUGhvdG8xDDAKBgNVBAsM
          A09wczENMAsGA1UEAwwERXRjZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
          ggEBAJ/zTXlvdaRWH38f8z33RG3CWuY7zFr3q6H8dwEpM8KGab5lZ8SOLpkrBOS7
          8CXZnvoCXh5NigLZJMCCNkV/s3OUeu5IkxMVflS5rNh9bMLoGgKGSS/9jw46ohyD
          /o3XKg9hsrFrQRSizTMxElD/Ccr73gIJzF74sLdyQpt/a7BwRdVjb42jafQrQw88
          9YGNeZpgCjAZ5fkzt+JW+ztkysd8eCXI/Une5Ap8+pZPt+9ayzvzA2DmpTJRH4Zs
          A4OmktDBTwG5hyMePZvrnrafvxeb5x73e72h0J6yRfqBNWqyIu/9oOJoqaevbTNP
          yqtB2zC9huIoV4MZYKkabM9x/ycCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8G
          A1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFOlIxdaKQKPrVAGf2B/aW7++0lPRMA0G
          CSqGSIb3DQEBCwUAA4IBAQBK7741peD9+fFtFhoSUPscl76nxLdXlKBJis7ytBuj
          w3itwOQFsTEE4vzB/jyTHHyBLfr+AmtjZa4JDATDtwhqxXowGymZ9KrDWEsOHzET
          q4edrl9/2EIDFPN5uWi/wue4AZy8f7PiKluStiXfYH49lIwtIvO0jiD+zCGsBWSM
          BxOHNDBP+XYkvBDDRIl7Uu79Djw5DICiP4vT1oGUnMM9/cYDBBVyl+lU54rueJVv
          g4PoOmHaeIPquIqxB5Aqc9m35mbutq4ZpQ65F2VPn1ejXI8jZnqUdOPjLZ1e2Wnx
          ym+NKgfOIgSQjA53hBgOnnFHJEr0S7e/PRvcvjdBdj12
          -----END CERTIFICATE-----
    - path: /etc/ssl/certs/ca.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIDfTCCAmWgAwIBAgIJAPgnITmJUxM1MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          BAYTAlVTMQswCQYDVQQIDAJOWTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoM
          B0JIUGhvdG8xDDAKBgNVBAsMA09wczENMAsGA1UEAwwERXRjZDAeFw0xNzEwMTcy
          MDIwNDBaFw0yNzEwMTUyMDIwNDBaMFwxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJO
          WTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoMB0JIUGhvdG8xDDAKBgNVBAsM
          A09wczENMAsGA1UEAwwERXRjZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
          ggEBAJ/zTXlvdaRWH38f8z33RG3CWuY7zFr3q6H8dwEpM8KGab5lZ8SOLpkrBOS7
          8CXZnvoCXh5NigLZJMCCNkV/s3OUeu5IkxMVflS5rNh9bMLoGgKGSS/9jw46ohyD
          /o3XKg9hsrFrQRSizTMxElD/Ccr73gIJzF74sLdyQpt/a7BwRdVjb42jafQrQw88
          9YGNeZpgCjAZ5fkzt+JW+ztkysd8eCXI/Une5Ap8+pZPt+9ayzvzA2DmpTJRH4Zs
          A4OmktDBTwG5hyMePZvrnrafvxeb5x73e72h0J6yRfqBNWqyIu/9oOJoqaevbTNP
          yqtB2zC9huIoV4MZYKkabM9x/ycCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8G
          A1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFOlIxdaKQKPrVAGf2B/aW7++0lPRMA0G
          CSqGSIb3DQEBCwUAA4IBAQBK7741peD9+fFtFhoSUPscl76nxLdXlKBJis7ytBuj
          w3itwOQFsTEE4vzB/jyTHHyBLfr+AmtjZa4JDATDtwhqxXowGymZ9KrDWEsOHzET
          q4edrl9/2EIDFPN5uWi/wue4AZy8f7PiKluStiXfYH49lIwtIvO0jiD+zCGsBWSM
          BxOHNDBP+XYkvBDDRIl7Uu79Djw5DICiP4vT1oGUnMM9/cYDBBVyl+lU54rueJVv
          g4PoOmHaeIPquIqxB5Aqc9m35mbutq4ZpQ65F2VPn1ejXI8jZnqUdOPjLZ1e2Wnx
          ym+NKgfOIgSQjA53hBgOnnFHJEr0S7e/PRvcvjdBdj12
          -----END CERTIFICATE-----
    - path: /etc/kubernetes/ssl/ca.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIDfTCCAmWgAwIBAgIJAPgnITmJUxM1MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          BAYTAlVTMQswCQYDVQQIDAJOWTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoM
          B0JIUGhvdG8xDDAKBgNVBAsMA09wczENMAsGA1UEAwwERXRjZDAeFw0xNzEwMTcy
          MDIwNDBaFw0yNzEwMTUyMDIwNDBaMFwxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJO
          WTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoMB0JIUGhvdG8xDDAKBgNVBAsM
          A09wczENMAsGA1UEAwwERXRjZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
          ggEBAJ/zTXlvdaRWH38f8z33RG3CWuY7zFr3q6H8dwEpM8KGab5lZ8SOLpkrBOS7
          8CXZnvoCXh5NigLZJMCCNkV/s3OUeu5IkxMVflS5rNh9bMLoGgKGSS/9jw46ohyD
          /o3XKg9hsrFrQRSizTMxElD/Ccr73gIJzF74sLdyQpt/a7BwRdVjb42jafQrQw88
          9YGNeZpgCjAZ5fkzt+JW+ztkysd8eCXI/Une5Ap8+pZPt+9ayzvzA2DmpTJRH4Zs
          A4OmktDBTwG5hyMePZvrnrafvxeb5x73e72h0J6yRfqBNWqyIu/9oOJoqaevbTNP
          yqtB2zC9huIoV4MZYKkabM9x/ycCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8G
          A1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFOlIxdaKQKPrVAGf2B/aW7++0lPRMA0G
          CSqGSIb3DQEBCwUAA4IBAQBK7741peD9+fFtFhoSUPscl76nxLdXlKBJis7ytBuj
          w3itwOQFsTEE4vzB/jyTHHyBLfr+AmtjZa4JDATDtwhqxXowGymZ9KrDWEsOHzET
          q4edrl9/2EIDFPN5uWi/wue4AZy8f7PiKluStiXfYH49lIwtIvO0jiD+zCGsBWSM
          BxOHNDBP+XYkvBDDRIl7Uu79Djw5DICiP4vT1oGUnMM9/cYDBBVyl+lU54rueJVv
          g4PoOmHaeIPquIqxB5Aqc9m35mbutq4ZpQ65F2VPn1ejXI8jZnqUdOPjLZ1e2Wnx
          ym+NKgfOIgSQjA53hBgOnnFHJEr0S7e/PRvcvjdBdj12
          -----END CERTIFICATE-----
    - path: /etc/kubernetes/ssl/etcd-node.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIE+zCCA+OgAwIBAgIJANxo/z6xfMb4MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          BAYTAlVTMQswCQYDVQQIDAJOWTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoM
          B0JIUGhvdG8xDDAKBgNVBAsMA09wczENMAsGA1UEAwwERXRjZDAeFw0xNzEwMTcy
          MDIwNDBaFw0yNzEwMTUyMDIwNDBaMGExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJO
          WTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoMB0JIUGhvdG8xDDAKBgNVBAsM
          A09wczESMBAGA1UEAwwJZXRjZC1ub2RlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
          MIIBCgKCAQEAodQQOS9mI42MH8Qk+D280lcBwZAqQQxwI/TMNhkjdd9S/wiCiTTz
          yogkaPVSMceidxSehHVkm2N0+LSFkG9yiLSVBBGjoJ+LR+wlECKiV4a1zNSgAn5/
          17SqTuzEdb6pSE59jKliMlwjCSzlRpsAT6Z1YNMrHqkBwRVSYnxWGibc0BJcCGCs
          Hbb13UnYERC5ce0MuXXX3nUJQXKYvrnBLYpWM5M5PN43VxwXCajBcFQKDngDbnmu
          8oIoWgxan7trAfgYDLo2/j7uqFinj9oqwSYicND1zanMc+XFZTHqc8copdQRKGMY
          b/kfNTNb3ACzZyqm45Cax4QcjtoFNjidTQIDAQABo4IBuTCCAbUwDgYDVR0PAQH/
          BAQDAgXgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAMBgNVHRMBAf8E
          AjAAMB0GA1UdDgQWBBTJdhrvi+oTQoJD/EjhjXK4DJmRRTCCAVUGA1UdEQSCAUww
          ggFIggdibmguY29tgg9jb3Jlb3MxLmJuaC5jb22CD2NvcmVvczIuYm5oLmNvbYIP
          Y29yZW9zMy5ibmguY29tgglsb2NhbGhvc3SCB2NvcmVvczGCB2NvcmVvczKCB2Nv
          cmVvczOCDWNvcmVvczEubG9jYWyCDWNvcmVvczIubG9jYWyCDWNvcmVvczMubG9j
          YWyCCmt1YmVybmV0ZXOCEmt1YmVybmV0ZXMuZGVmYXVsdIIWa3ViZXJuZXRlcy5k
          ZWZhdWx0LnN2Y4Ika3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2Fs
          gg5rdWJlLWFwaXNlcnZlcoIKa3ViZS1hZG1pbocEfwAAAYcECgACC4cECgACDIcE
          CgACDYcECgMAAYcECgACAYcECgACAocEAAAAAIENYWRtaW5AYm5oLmNvbTANBgkq
          hkiG9w0BAQsFAAOCAQEAIyama2mrV4vzcF0Tsw1NghE9GdAx0WpmfFStEC5GrgBH
          +tVjuaqLYIGW8DC7+EIFLq5obQCeXIMZ1OQoQFegfv0X8XwsDFQeaNs15N6jYZ8S
          7Yxe4mFvCecvVfqdc5M4XxykoRQQbIzBBI7AJ5zumTB6c6OZ2l8cRyNPYooj3xHt
          XqHnugDX4LMyBLRxRYyatF9Evku4VnqbawWPpejpfcfGiwyx/ArNNmDgsOikvYYB
          ndmXqiYpBAeatW65kwvIWuGoo4HT3K3BExP0aVxrddAFL8Q35DRxTU6tTM3k3q3b
          gv2CMfTju+IUHyqnqcWy7vIcKwQrYkmfKdjGB4mjzw==
          -----END CERTIFICATE-----
    - path: /etc/kubernetes/ssl/etcd-node-key.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN RSA PRIVATE KEY-----
          MIIEpQIBAAKCAQEAodQQOS9mI42MH8Qk+D280lcBwZAqQQxwI/TMNhkjdd9S/wiC
          iTTzyogkaPVSMceidxSehHVkm2N0+LSFkG9yiLSVBBGjoJ+LR+wlECKiV4a1zNSg
          An5/17SqTuzEdb6pSE59jKliMlwjCSzlRpsAT6Z1YNMrHqkBwRVSYnxWGibc0BJc
          CGCsHbb13UnYERC5ce0MuXXX3nUJQXKYvrnBLYpWM5M5PN43VxwXCajBcFQKDngD
          bnmu8oIoWgxan7trAfgYDLo2/j7uqFinj9oqwSYicND1zanMc+XFZTHqc8copdQR
          KGMYb/kfNTNb3ACzZyqm45Cax4QcjtoFNjidTQIDAQABAoIBAFkzyMGM1BNS6eN0
          viAr/9I97ZThv7jdf08CmDJVxwspT9mOuk2hZ6TDSLamS2lvLqnPLtW60/snZonV
          EC6ekBa029NDWdunNiIuE/wB61Stx7ahmZXmXtfrGVmp/10b/LoDsgcAM0WpLhLX
          sYYq2JAPJHJ71C/xXysscp4hBWEiHDUx0Uc9Cc4dDAJyICgmxsYEk86NqxX5oekw
          FpKhKWCVOp+7wEzKHbNA3F2L5qRZ1QKKJKZH3mIO/TkNmbHkVM/V949PHsruLDP4
          SeArZV8PEfe3BZwc7zpSNwa/ZNjt/ZuwW2hL2lHrxpPT2USAn8TwhD/HZBYLWUWD
          RQ2hQ40CgYEA1idEi1BfCb6RNQmXoyAkaJL/SewPnhyadjO8tZMTzN3BkLM5pM+W
          a536sj9Cu7HpyphvvVy24JPfZ6EZjtCyWIJWk+mZhHG/UcQyEa5X5Kc3cF6hG3Rg
          m9jgtdDizo3mM64u8x8ScaDaomkO8sA/djCMjIwa1EIVS8/N6npOc1cCgYEAwXNO
          yU5PFRZBuQkOedWKlFCF37/yHaP41akwPNLptVfAlcnHErnPz4pI4BX5Zi6pcyst
          LcuLUGhVpH8hgUw35uiFFRGG8NpfUZbIq0sGZ6yHHnmOz06aSRD8VbxqaLhw82y+
          HiqZOFpyOaYFWYlwxuoRL7gzU4aBxtN+vvNJUfsCgYEAiAAgE6lS8clIRjp1Rm9f
          L9pKrhNwR6U6ot5wdY+V/4DFMLPhY71Bdc66KwDpzHFMXBE6xTNgIHdIZXN127DF
          0lfTy9OKXwYtziue8IOb5eYb2wz9m4Xo5JBzBIRHxY1wKFnyDwwolmjk8Edv6tIU
          e2aRm6BOWYYuKBssVkCeGbkCgYEAut70YKnw4uHZIzGIdlC9yX/vkrEM5W0cVulU
          26yybqcSNp/wWjp8cHOY6j7P6NFyhvpSbkaNkWL4OCtG5mwGJFzBEhUYUVnhciNC
          PZX/3e5fqCSsu72mdKaf2wJjmD1zfvKwxNtJn5JffSjPSirmcoBxkbNSgYERpEtx
          yGujjDkCgYEAmh17ruky/BcEbnRF7sZoc1wBdpgG/m/rOqlEBgf08omEyyYy64he
          X4JpHBgSWR+RikcW3KQze4mzdClKtXFnpeZjSVYUkyOGMd15X9r86f7yTXDH+yaT
          aDxeHW+J8lVnfw2xlU8BVh3kcFRDZFbKPQnv7oAfilr/81t4Z+JBWiM=
          -----END RSA PRIVATE KEY-----
    - path: /var/lib/etcd/ssl/etcd-node.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN CERTIFICATE-----
          MIIE+zCCA+OgAwIBAgIJANxo/z6xfMb4MA0GCSqGSIb3DQEBCwUAMFwxCzAJBgNV
          BAYTAlVTMQswCQYDVQQIDAJOWTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoM
          B0JIUGhvdG8xDDAKBgNVBAsMA09wczENMAsGA1UEAwwERXRjZDAeFw0xNzEwMTcy
          MDIwNDBaFw0yNzEwMTUyMDIwNDBaMGExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJO
          WTERMA8GA1UEBwwITmV3IFlvcmsxEDAOBgNVBAoMB0JIUGhvdG8xDDAKBgNVBAsM
          A09wczESMBAGA1UEAwwJZXRjZC1ub2RlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
          MIIBCgKCAQEAodQQOS9mI42MH8Qk+D280lcBwZAqQQxwI/TMNhkjdd9S/wiCiTTz
          yogkaPVSMceidxSehHVkm2N0+LSFkG9yiLSVBBGjoJ+LR+wlECKiV4a1zNSgAn5/
          17SqTuzEdb6pSE59jKliMlwjCSzlRpsAT6Z1YNMrHqkBwRVSYnxWGibc0BJcCGCs
          Hbb13UnYERC5ce0MuXXX3nUJQXKYvrnBLYpWM5M5PN43VxwXCajBcFQKDngDbnmu
          8oIoWgxan7trAfgYDLo2/j7uqFinj9oqwSYicND1zanMc+XFZTHqc8copdQRKGMY
          b/kfNTNb3ACzZyqm45Cax4QcjtoFNjidTQIDAQABo4IBuTCCAbUwDgYDVR0PAQH/
          BAQDAgXgMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAMBgNVHRMBAf8E
          AjAAMB0GA1UdDgQWBBTJdhrvi+oTQoJD/EjhjXK4DJmRRTCCAVUGA1UdEQSCAUww
          ggFIggdibmguY29tgg9jb3Jlb3MxLmJuaC5jb22CD2NvcmVvczIuYm5oLmNvbYIP
          Y29yZW9zMy5ibmguY29tgglsb2NhbGhvc3SCB2NvcmVvczGCB2NvcmVvczKCB2Nv
          cmVvczOCDWNvcmVvczEubG9jYWyCDWNvcmVvczIubG9jYWyCDWNvcmVvczMubG9j
          YWyCCmt1YmVybmV0ZXOCEmt1YmVybmV0ZXMuZGVmYXVsdIIWa3ViZXJuZXRlcy5k
          ZWZhdWx0LnN2Y4Ika3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2Fs
          gg5rdWJlLWFwaXNlcnZlcoIKa3ViZS1hZG1pbocEfwAAAYcECgACC4cECgACDIcE
          CgACDYcECgMAAYcECgACAYcECgACAocEAAAAAIENYWRtaW5AYm5oLmNvbTANBgkq
          hkiG9w0BAQsFAAOCAQEAIyama2mrV4vzcF0Tsw1NghE9GdAx0WpmfFStEC5GrgBH
          +tVjuaqLYIGW8DC7+EIFLq5obQCeXIMZ1OQoQFegfv0X8XwsDFQeaNs15N6jYZ8S
          7Yxe4mFvCecvVfqdc5M4XxykoRQQbIzBBI7AJ5zumTB6c6OZ2l8cRyNPYooj3xHt
          XqHnugDX4LMyBLRxRYyatF9Evku4VnqbawWPpejpfcfGiwyx/ArNNmDgsOikvYYB
          ndmXqiYpBAeatW65kwvIWuGoo4HT3K3BExP0aVxrddAFL8Q35DRxTU6tTM3k3q3b
          gv2CMfTju+IUHyqnqcWy7vIcKwQrYkmfKdjGB4mjzw==
          -----END CERTIFICATE-----
    - path: /var/lib/etcd/ssl/etcd-node-key.pem
      filesystem: root
      mode: 0644
      contents:
        inline: |
          -----BEGIN RSA PRIVATE KEY-----
          MIIEpQIBAAKCAQEAodQQOS9mI42MH8Qk+D280lcBwZAqQQxwI/TMNhkjdd9S/wiC
          iTTzyogkaPVSMceidxSehHVkm2N0+LSFkG9yiLSVBBGjoJ+LR+wlECKiV4a1zNSg
          An5/17SqTuzEdb6pSE59jKliMlwjCSzlRpsAT6Z1YNMrHqkBwRVSYnxWGibc0BJc
          CGCsHbb13UnYERC5ce0MuXXX3nUJQXKYvrnBLYpWM5M5PN43VxwXCajBcFQKDngD
          bnmu8oIoWgxan7trAfgYDLo2/j7uqFinj9oqwSYicND1zanMc+XFZTHqc8copdQR
          KGMYb/kfNTNb3ACzZyqm45Cax4QcjtoFNjidTQIDAQABAoIBAFkzyMGM1BNS6eN0
          viAr/9I97ZThv7jdf08CmDJVxwspT9mOuk2hZ6TDSLamS2lvLqnPLtW60/snZonV
          EC6ekBa029NDWdunNiIuE/wB61Stx7ahmZXmXtfrGVmp/10b/LoDsgcAM0WpLhLX
          sYYq2JAPJHJ71C/xXysscp4hBWEiHDUx0Uc9Cc4dDAJyICgmxsYEk86NqxX5oekw
          FpKhKWCVOp+7wEzKHbNA3F2L5qRZ1QKKJKZH3mIO/TkNmbHkVM/V949PHsruLDP4
          SeArZV8PEfe3BZwc7zpSNwa/ZNjt/ZuwW2hL2lHrxpPT2USAn8TwhD/HZBYLWUWD
          RQ2hQ40CgYEA1idEi1BfCb6RNQmXoyAkaJL/SewPnhyadjO8tZMTzN3BkLM5pM+W
          a536sj9Cu7HpyphvvVy24JPfZ6EZjtCyWIJWk+mZhHG/UcQyEa5X5Kc3cF6hG3Rg
          m9jgtdDizo3mM64u8x8ScaDaomkO8sA/djCMjIwa1EIVS8/N6npOc1cCgYEAwXNO
          yU5PFRZBuQkOedWKlFCF37/yHaP41akwPNLptVfAlcnHErnPz4pI4BX5Zi6pcyst
          LcuLUGhVpH8hgUw35uiFFRGG8NpfUZbIq0sGZ6yHHnmOz06aSRD8VbxqaLhw82y+
          HiqZOFpyOaYFWYlwxuoRL7gzU4aBxtN+vvNJUfsCgYEAiAAgE6lS8clIRjp1Rm9f
          L9pKrhNwR6U6ot5wdY+V/4DFMLPhY71Bdc66KwDpzHFMXBE6xTNgIHdIZXN127DF
          0lfTy9OKXwYtziue8IOb5eYb2wz9m4Xo5JBzBIRHxY1wKFnyDwwolmjk8Edv6tIU
          e2aRm6BOWYYuKBssVkCeGbkCgYEAut70YKnw4uHZIzGIdlC9yX/vkrEM5W0cVulU
          26yybqcSNp/wWjp8cHOY6j7P6NFyhvpSbkaNkWL4OCtG5mwGJFzBEhUYUVnhciNC
          PZX/3e5fqCSsu72mdKaf2wJjmD1zfvKwxNtJn5JffSjPSirmcoBxkbNSgYERpEtx
          yGujjDkCgYEAmh17ruky/BcEbnRF7sZoc1wBdpgG/m/rOqlEBgf08omEyyYy64he
          X4JpHBgSWR+RikcW3KQze4mzdClKtXFnpeZjSVYUkyOGMd15X9r86f7yTXDH+yaT
          aDxeHW+J8lVnfw2xlU8BVh3kcFRDZFbKPQnv7oAfilr/81t4Z+JBWiM=
          -----END RSA PRIVATE KEY-----
    - path: /etc/profile.d/aliases.sh
      filesystem: root
      mode: 0755
      contents:
        inline: |
          dps () { docker ps -a ; }
          dl () { docker logs -f $1 & }
          drm () { docker stop $1 && docker rm $1; }
          PATH=$PATH:/opt/bin
    - path: /etc/flannel/options.env
      filesystem: root
      mode: 0644
      contents:
        inline: |
          # ETCDCTL parameters
          ADVERTISE_IP=${ADVERTISE_IP}
          ETCDCTL_CA_FILE=/etc/ssl/certs/ca.pem
          ETCDCTL_CERT_FILE=/var/lib/etcd/ssl/etcd-node.pem
          ETCDCTL_KEY_FILE=/var/lib/etcd/ssl/etcd-node-key.pem
          ETCDCTL_ENDPOINTS="https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379"
          FLANNELD_IFACE="enp0s3"
          FLANNELD_ETCD_ENDPOINTS=${ETCDCTL_ENDPOINTS}
          FLANNELD_ETCD_PREFIX="/coreos.com/network"
          kubernetes_version=1.8.0
    - path: /etc/systemd/system/docker.service.d/40-flannel.conf
      filesystem: root
      mode: 0644
      contents:
        inline: |
          [Unit]
          Requires=flanneld.service
          After=flanneld.service
          [Service]
          EnvironmentFile=/etc/kubernetes/cni/docker_opts_cni.env
    - path: /etc/kubernetes/cni/docker_opts_cni.env
      filesystem: root
      mode: 0644
      contents:
        inline: |
          DOCKER_OPT_BIP=""
          DOCKER_OPT_IPMASQ=""
    - path: /etc/kubernetes/cni/net.d/10-containernet.conf
      filesystem: root
      mode: 0644
      contents:
        inline: |
          {
              "name": "podnet",
              "type": "flannel",
              "delegate": {
                  "forceAddress": true,
                  "isDefaultGateway": true,
                  "hairpinMode": true
              }
          }
    - path: /etc/systemd/system/flanneld.service.d/40-ExecStartPre-symlink.conf
      filesystem: root
      mode: 0644
      contents:
        inline: |
          [Service]
          ExecStartPre=/usr/bin/ln -sf /etc/flannel/options.env /run/flannel/options.env
          ExecStartPre=/usr/bin/ln -sf /etc/kubernetes/cni/net.d /etc/rkt/net.d
    - path: /etc/kubernetes/kubelet.env
      filesystem: root
      mode: 0644
      contents:
        inline: |
          KUBELET_IMAGE_URL=quay.io/coreos/hyperkube
          KUBELET_VERSION=v1.8.0_coreos.0
    - path: /etc/kubernetes/worker-kubeconfig.yaml 
      filesystem: root
      mode: 0644
      contents:
        inline: |
          apiVersion: v1
          kind: Config
          clusters:
          - name: local
            cluster:
              server: https://10.0.2.13:443
              certificate-authority: /etc/kubernetes/ssl/ca.pem
          users:
          - name: kubelet
            user:
              client-certificate: /etc/kubernetes/ssl/etcd-node.pem
              client-key: /etc/kubernetes/ssl/etcd-node-key.pem
          contexts:
          - context:
              cluster: local
              user: kubelet
    - path: /etc/kubernetes/manifests/kube-apiserver.yaml
      filesystem: root
      mode: 0644
      contents:
        inline: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-apiserver
            namespace: kube-system
          spec:
            hostNetwork: true
            containers:
            - name: kube-apiserver
              image: quay.io/coreos/hyperkube:v1.8.0_coreos.0
              command:
              - /hyperkube
              - apiserver
              - --bind-address=0.0.0.0
              - --etcd-servers=https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379
              - --allow-privileged=true
              - --service-cluster-ip-range=10.3.0.0/21
              - --secure-port=443
              - --insecure-port=8080
              - --advertise-address=10.0.2.13
              - --storage-backend=etcd3
              - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota
              - --etcd-certfile=/var/lib/etcd/ssl/etcd-node.pem
              - --etcd-keyfile=/var/lib/etcd/ssl/etcd-node-key.pem
              - --tls-cert-file=/var/lib/etcd/ssl/etcd-node.pem
              - --tls-private-key-file=/var/lib/etcd/ssl/etcd-node-key.pem
              - --kubelet-client-certificate=/var/lib/etcd/ssl/etcd-node.pem
              - --kubelet-client-key=/var/lib/etcd/ssl/etcd-node-key.pem
              - --service-account-key-file=/var/lib/etcd/ssl/etcd-node-key.pem
              - --etcd-cafile=/var/lib/etcd/ssl/ca.pem
              - --tls-ca-file=/var/lib/etcd/ssl/ca.pem
              - --client-ca-file=/var/lib/etcd/ssl/ca.pem
              - --runtime-config=extensions/v1beta1/networkpolicies=true,extensions/v1beta1=true
              - --anonymous-auth=false
              - --audit-log-path=/var/log/kubernetes/kube-apiserver-audit.log
              - --audit-log-maxage=30
              - --audit-log-maxbackup=3
              - --audit-log-maxsize=100
              - --v=3
              livenessProbe:
                httpGet:
                  host: 127.0.0.1
                  port: 8080
                  path: /healthz
                initialDelaySeconds: 15
                timeoutSeconds: 15
              ports:
              - containerPort: 443
                hostPort: 443
                name: https
              - containerPort: 8080
                hostPort: 8080
                name: local
              volumeMounts:
              - mountPath: /var/lib/etcd/ssl
                name: ssl-certs-kubernetes
                readOnly: true
              - mountPath: /etc/ssl/certs
                name: ssl-certs-host
                readOnly: true
              - mountPath: /var/log/kubernetes
                name: var-log-kubernetes
                readOnly: false
            volumes:
            - hostPath:
                path: /var/lib/etcd/ssl
              name: ssl-certs-kubernetes
            - hostPath:
                path: /usr/share/ca-certificates
              name: ssl-certs-host
            - hostPath:
                path: /var/log/kubernetes
              name: var-log-kubernetes
    - path: /etc/kubernetes/manifests/kube-controller-manager.yaml
      filesystem: root
      mode: 0644
      contents:
        inline: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-controller-manager
            namespace: kube-system
          spec:
                hostNetwork: true
                containers:
                - name: kube-controller-manager
                  image: quay.io/coreos/hyperkube:v1.8.0_coreos.0
                  command:
                  - ./hyperkube
                  - controller-manager
                  - --master=http://127.0.0.1:8080
                  - --leader-elect=true
                  - --allocate-node-cidrs=true
                  - --cluster-cidr=10.0.0.0/21
                  - --service-account-private-key-file=/var/lib/etcd/ssl/etcd-node-key.pem
                  - --root-ca-file=/var/lib/etcd/ssl/ca.pem
                  livenessProbe:
                    httpGet:
                      host: 127.0.0.1
                      path: /healthz
                      port: 10252  # Note: Using default port. Update if --port option is set differently.
                    initialDelaySeconds: 15
                    timeoutSeconds: 5
                  volumeMounts:
                  - mountPath: /var/lib/etcd/ssl
                    name: secrets
                    readOnly: true
                  - mountPath: /etc/ssl/certs
                    name: ssl-host
                    readOnly: true
                  - mountPath: /var/log/kube-controller-manager.log
                    name: logfile
                    readOnly: false
                volumes:
                - hostPath:
                    path: /var/lib/etcd/ssl
                  name: secrets
                - hostPath:
                    path: /usr/share/ca-certificates
                  name: ssl-host
                - hostPath:
                    path: /var/log/kube-controller-manager.log
                  name: logfile
    - path: /etc/kubernetes/manifests/kube-proxy.yaml
      filesystem: root
      mode: 0644
      contents:
        inline: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-proxy
            namespace: kube-system
            labels:
              tier: node
              k8s-app: kube-proxy
          spec:
                hostNetwork: true
                containers:
                - name: kube-proxy
                  image: quay.io/coreos/hyperkube:v1.8.0_coreos.0
                  command:
                  - ./hyperkube
                  - proxy
                  - --master=http://127.0.0.1:8080
                  - --logtostderr=true
                  - --proxy-mode=iptables
                  - --hostname-override=10.0.2.13
                  - --cluster-cidr=10.0.0.0/21
                  - --v=3
                  env:
                    - name: NODE_NAME
                      valueFrom:
                        fieldRef:
                          fieldPath: spec.nodeName
                  securityContext:
                    privileged: true
                  volumeMounts:
                  - mountPath: /etc/ssl/certs
                    name: ssl-certs-host
                    readOnly: true
                  - name: etc-kubernetes
                    mountPath: /var/lib/etcd/ssl
                    readOnly: true
                tolerations:
                - key: "node-role.kubernetes.io/master"
                  operator: "Exists"
                  effect: "NoSchedule"
                volumes:
                - hostPath:
                    path: /usr/share/ca-certificates
                  name: ssl-certs-host
                - name: etc-kubernetes
                  hostPath:
                    path: /etc/kubernetes
    - path: /etc/kubernetes/manifests/kube-scheduler.yaml
      filesystem: root
      mode: 0644
      contents:
        inline: |
          apiVersion: v1
          kind: Pod
          metadata:
            name: kube-scheduler
            namespace: kube-system
          spec:
            hostNetwork: true
            containers:
            - name: kube-scheduler
              image: quay.io/coreos/hyperkube:v1.8.0_coreos.0
              command:
              - ./hyperkube
              - scheduler
              - --master=http://127.0.0.1:8080
              - --address=0.0.0.0
              - --leader-elect=true
              - --v=3
              livenessProbe:
                  httpGet:
                    host: 127.0.0.1
                    path: /healthz
                    port: 10251  # Note: Using default port. Update if --port option is set differently.
                  initialDelaySeconds: 15
                  timeoutSeconds: 15
              nodeSelector:
                node-role.kubernetes.io/master: ""
              securityContext:
                runAsNonRoot: true
                runAsUser: 65534
              volumeMounts:
              - mountPath: /var/log/kube-scheduler.log
                name: logfile
            volumes:
            - hostPath:
                path: /var/log/kube-scheduler.log
              name: logfile

passwd:
  users:
    - name: usera
      password_hash: "$1$L0gy00B2$ChBC93/KC6BO5zOXZ47gJ1"
      ssh_authorized_keys:
        - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7ChcV4YlUBFoZwhTMHntBmXuoI12wf//C/2791gyay/8ESRYD8H8TrKbUS+wKVBounkaXgaJBJYOdc9kt8NIFj7wCzRDBDvojjH5yckMMfaghuSoCHuIQ/T1h3DDC9IRl8MkXBLDe4uYY2ZDxDWTHhKrsAsBVrxu6lfD0VS3N1T08whBcmQZcaK6iRmWfW+eR4rCX/7o9tzYcK2hypanW+yf/lFVWGvwhhVafmUxP4bbv11Rf4ckvtVRgVA4pbXcuoeM8mQLOIY5qNmFoRwaYUyQbFzkYOUlr1OPgPSpftG6L/sun284ERbc+bfg8BAcADrLHs7TreXOw5CIjB8oN root@dc1-os4"
      home_dir: /home/usera
      groups:
        - sudo
        - docker
      shell: /bin/bash

systemd:
  units:
    - name: etcd-member.service
      dropins:
        - name: conf1.conf
          contents: |
            [Service]
            EnvironmentFile=/etc/environment

    - name: docker-tcp.socket
      enabled: false
      contents: |
        [Unit]
        Description=Docker Socket for the API

        [Socket]
        ListenStream=2375
        BindIPv6Only=both
        Service=docker.service

        [Install]
        WantedBy=sockets.target

    - name: docker.service
      enabled: false
      dropins:
        - name: 50-insecure-registry.conf
          contents: |
            [Service]
            Environment=DOCKER_OPTS=''
            Environment=HTTP_PROXY=http://proxy:port
            Environment=HTTPS_PROXY=http://proxy:port
            Environment=http_proxy=http://proxy:port
            Environment=https_proxy=http://proxy:port
            Environment=no_proxy=localhost,127.0.0.0/8,127.0.0.1,::1,10.0.2.11,10.0.2.12,10.0.2.13,coreos1,coreos2,coreos3,coreos1.domain.com,coreos2.domain.com,coreos3.domain.com,/var/run/docker.sock

    - name: etcd-member.service
      enable: true

    - name: flanneld.service
      dropins:
        - name: 50-network-config.conf
          contents: |
            [Service]
            EnvironmentFile=/etc/flannel/options.env
            ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.0.0.0/21", "SubnetLen": 24, "Backend": { "Type": "vxlan", "VNI": 1 } }'
            # VXlan/VNI/VLan example
            #ExecStartPre=/usr/bin/etcdctl ${CA} ${CERT} ${KEY} ${EP} set /coreos.com/network/red-db/config '{ "Network": "10.220.0.0/16", "SubnetLen": 24, "SubnetMin": "10.220.110.0", "SubnetMax": "10.220.200.0", "Backend": { "Type": "vxlan", "VNI": 6 } }'
            #ExecStartPre=/usr/bin/etcdctl ${CA} ${CERT} ${KEY} ${EP} set /coreos.com/network/red-app/config '{ "Network": "10.220.0.0/16", "SubnetLen": 24, "SubnetMin": "10.220.110.0", "SubnetMax": "10.220.200.0", "Backend": { "Type": "vxlan", "VNI": 5 } }'
            # This is needed for rkt to start due to some bug
            ExecStartPre=/usr/bin/rkt trust --skip-fingerprint-review --prefix=coreos.com/rkt/stage1-coreos
            ExecStartPre=/usr/bin/rkt image fetch coreos.com/rkt/stage1-coreos:1.29.0
            Restart=always

    - name: flanneld.service
      dropins:
        - name: 50-ssl.conf
          contents: |
            [Service]
            Environment="ETCD_SSL_DIR=/var/lib/etcd/ssl"

    - name: etcd-member-config.service
      enabled: true
      contents: |
        [Unit]
        Before=network-pre.target
        Wants=network-pre.target

        [Service]
        Type=oneshot
        ExecStart=/usr/bin/ln -sf /etc/metadata/coreos /run/metadata/coreos
        TimeoutSec=0
        RemainAfterExit=yes

        [Install]
        WantedBy=network.target

    - name: rkt-api-tcp.socket
      enabled: true
      contents: |
        [Unit]
        Description=rkt api service socket
        PartOf=rkt-api.service

        [Socket]
        ListenStream=127.0.0.1:15441
        Service=rkt-api.service
        BindIPv6Only=both

        [Install]
        WantedBy=sockets.target

    - name: rkt-api.service
      enabled: true
      contents: |
        [Unit]
        Description=rkt api service
        Documentation=http://github.com/rkt/rkt
        After=network.target rkt-api-tcp.socket
        Requires=rkt-api-tcp.socket

        [Service]
        ExecStart=/usr/bin/rkt api-service --local-config=/etc/rkt

        [Install]
        WantedBy=multi-user.target

    - name: rkt-gc.service
      enabled: true
      contents: |
        [Unit]
        Description=Garbage Collection for rkt

        [Service]
        Environment=GRACE_PERIOD=24h
        Type=oneshot
        ExecStart=/usr/bin/rkt gc --grace-period=${GRACE_PERIOD}

    - name: rkt-gc.timer
      enabled: true
      contents: |
        [Unit]
        Description=Periodic Garbage Collection for rkt

        [Timer]
        OnActiveSec=0s
        OnUnitActiveSec=12h

        [Install]
        WantedBy=multi-user.target

    - name: rkt-metadata.service
      enabled: true
      contents: |
        [Unit]
        Description=rkt metadata service
        Documentation=http://github.com/rkt/rkt
        After=network.target rkt-metadata.socket
        Requires=rkt-metadata.socket

        [Service]
        ExecStart=/usr/bin/rkt metadata-service

        [Install]
        WantedBy=multi-user.target

    - name: rkt-metadata.socket
      enabled: true
      contents: |
        [Unit]
        Description=rkt metadata service socket
        PartOf=rkt-metadata.service

        [Socket]
        ListenStream=/run/rkt/metadata-svc.sock
        SocketMode=0660
        SocketUser=root
        SocketGroup=root
        RemoveOnStop=true

        [Install]
        WantedBy=sockets.target

    - name: kubelet.service
      enabled: false
      contents: |
        [Unit]
        Description=The primary agent to run pods
        Documentation=http://kubernetes.io/docs/admin/kubelet/
        Requires=etcd-member.service
        After=flanneld.service
        
        [Service]
        Slice=system.slice
        Environment=KUBELET_IMAGE_TAG=v1.8.0_coreos.0
        #Environment=HTTP_PROXY=http://proxy:port
        #Environment=HTTPS_PROXY=http://proxy:port
        #Environment=http_proxy=http://proxy:port
        #Environment=https_proxy=http://proxy:port
        #Environment=no_proxy=localhost,127.0.0.0/8,127.0.0.1,::1,10.0.2.11,10.0.2.12,10.0.2.13,coreos1,coreos2,coreos3,coreos1.domain.com,coreos2.domain.com,coreos3.domain.com,/var/run/docker.sock
        ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/usr/bin/mkdir -p /var/log/containers
        ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/cni/net.d
        ExecStartPre=/usr/bin/mkdir -p /opt/cni/bin
        ExecStartPre=/usr/bin/mkdir -p /var/lib/cni
        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/run/kubelet-pod.uuid \
          --net=host \
          --dns=host \
          --set-env-file=/etc/environment \
          --volume var-lib-rkt,kind=host,source=/var/lib/rkt \
          --mount volume=var-lib-rkt,target=/var/lib/rkt \
          --volume dns,kind=host,source=/etc/resolv.conf \
          --mount volume=dns,target=/etc/resolv.con \
          --volume var-lib-cni,kind=host,source=/var/lib/cni \
          --mount volume=var-lib-cni,target=/var/lib/cni \
          --volume var-log,kind=host,source=/var/log \
          --mount volume=var-log,target=/var/log \
          --volume container,kind=host,source=/var/log/containers \
          --mount volume=container,target=/var/log/containers \
          --volume rkt,kind=host,source=/usr/bin/rkt \
          --mount volume=rkt,target=/usr/bin/rkt"
        ExecStart=/usr/lib/coreos/kubelet-wrapper \
          --address="0.0.0.0" \
          --register-node=true \
          --container-runtime=rkt \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --allow-privileged=true \
          --cert-dir=/etc/kubernetes/ssl \
          --tls-cert-file=/etc/kubernetes/ssl/etcd-node.pem \
          --tls-private-key-file=/etc/kubernetes/ssl/etcd-node-key.pem \
          --client-ca-file=/etc/kubernetes/ssl/ca.pem \
          --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
          --cluster-dns=10.3.0.10 \
          --cluster-domain=cluster.local \
          --network-plugin=cni \
          --cni-conf-dir=/etc/kubernetes/cni/net.d \
          --lock-file=/var/run/lock/kubelet.lock \
          --exit-on-lock-contention \
          --cloud-provider="" \
          --cadvisor-port=4194 \
          --cgroups-per-qos=true \
          --cgroup-root=/ \
          --hostname-override=coreos3
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/run/kubelet-pod.uuid
        Restart=always
        RestartSec=10
        
        [Install]
        WantedBy=multi-user.target

    - name: settimezone.service
      enabled: true
      contents: |
        [Unit]
        Description=Set the time zone

        [Service]
        ExecStart=/usr/bin/timedatectl set-timezone America/New_York
        RemainAfterExit=yes
        Type=oneshot

networkd:
  units:
    - name: 00-eno0.network
      contents: |
        [Match]
        Name=enp0s3

        [Network]
        Address=10.0.2.13/24
        Gateway=10.0.2.2
        DNS=8.8.8.8
        Domains=domain.com

flannel:
  version:        0.9.0
  etcd_prefix:    "/coreos.com/network"
  etcd_endpoints: "https://10.0.2.11:2379,https://10.0.2.12:2379,https://10.0.2.13:2379"
  etcd_cafile:    /etc/ssl/certs/ca.pem
  etcd_certfile:  /var/lib/etcd/ssl/etcd-node.pem
  etcd_keyfile:   /var/lib/etcd/ssl/etcd-node-key.pem
  interface:      enp0s3

etcd:
  version:                     3.2.9
  name:                        "{HOSTNAME}"
  data_dir:                    /var/lib/etcd
  listen_client_urls:          https://{PRIVATE_IPV4}:2379,https://127.0.0.1:2379,https://127.0.0.1:4001
  advertise_client_urls:       https://{PRIVATE_IPV4}:2379
  listen_peer_urls:            https://{PRIVATE_IPV4}:2380
  initial_advertise_peer_urls: https://{PRIVATE_IPV4}:2380
  initial_cluster:             coreos1=https://10.0.2.11:2380,coreos2=https://10.0.2.12:2380,coreos3=https://10.0.2.13:2380
  initial_cluster_token:       etcd-token
  initial_cluster_state:       new
  cert_file:                   /var/lib/etcd/ssl/etcd-node.pem
  key_file:                    /var/lib/etcd/ssl/etcd-node-key.pem
  peer_cert_file:              /var/lib/etcd/ssl/etcd-node.pem
  peer_key_file:               /var/lib/etcd/ssl/etcd-node-key.pem
  client_cert_auth:            true
  peer_client_cert_auth:       true
  trusted_ca_file:             /etc/ssl/certs/ca.pem
  peer_trusted_ca_file:        /etc/ssl/certs/ca.pem
  auto_compaction_retention:   1


update:
  group:  "alpha"
locksmith:
  reboot_strategy: "etcd-lock"
  etcd_endpoints:  "https://10.0.2.13:2379"
  etcd_cafile: /etc/ssl/certs/ca.pem
  etcd_certfile: /var/lib/etcd/ssl/etcd-node.pem
  etcd_keyfile: /var/lib/etcd/ssl/etcd-node-key.pem

Back to Kubernetes Configuration

You might also like – Other articles related to Docker Kubernetes / micro-service.
Like what you’re reading? please provide feedback, any feedback is appreciated.

0 0 votes

Article Rating

CoreOS Ignition Configuration Files

Like this:

Related

CoreOS Ignition Configuration Files

Share this:

Like this:

Related