Ansible Playbook for Kubernetes 

Kubernetes (a.k.a K8s) is the leader platform for container deployment and management.

In this tutorial I present a way to launch a Kubernetes cluster made up of one master server (API, Scheduler, Controller) and n nodes (Pods, kubelet, proxy and Docker) running project Calico to implement the Kubernetes networking model.

Vagrant is used to spin up the virtual machines using the Virtualbox hypervisor and to run the Ansible playbooks to configure the Kubernetes cluster environment. The objective is to be able to provision a development and learning cluster with many workers.

Download de the code from IT Wonder Lab public GitHub repository.

This Ansible playbook and Vagrantfile for installing Kubernetes have been possible with the help from other blogs (https://kubernetes.io/blog and https://docs.projectcalico.org/). I have included a link to the relevant pages in the source code. 

Updates:

  • 6 June 2019: Fix issue: kubectl was not able to recover logs. See new task “Configure node-ip … at kubelet” in ansible

The Ansible playbook follows IT Wonder Lab best practices and can be used to configure a new Kubernetes cluster in a cloud provider or in a different hypervisor as it doesn’t have dependencies with Virtualbox or Vagrant.

Creating a Kubernetes Cluster with Vagrant and Ansible 

Click on the play button to see the execution of Vagrant creating a Kubernetes Cluster with Ansible.

File Structure

The code used to create a Kubernetes Cluster with Vagrant and Ansible is composed of:

ansible-vbox-vagrant-kubernetes - ansible-vbox-vagrant-kubernetes-dir-structure.png

  • Kubernetes Ansible roles:
    • roles/add_packages: Ansible playbook to install / remove packages using APT in an Ubuntu system.
    • roles/k8s/common: installs the needed packages for Kubernetes (delegating in add_packages) and configures the common settings for Kubernetes master and nodes.
    • roles/k8s/master: Ansible playbook to configure a Kubernetes master, it uses the common playbook for shared components between the Kubernetes master and the nodes.
    • roles/k8s/node: Ansible playbook to configure a Kubernetes node, it uses the common playbook for shared components between the Kubernetes master and the nodes.
    • k8s.yml: Ansible playbook that uses the Kubernetes ansible roles.
  • Vagrantfile: contains the definition of the machines (CPU, memory, network and Ansible playbook and properties)
  • .vagrant/: hidden directory for Vagrant tracking. It includes a Vagrant generated Inventory file: vagrant_ansible_inventory that is used by Ansible to match virtual machines and roles.
  • roles/ditwl-k8s-01-join-command: this file is generated by the Kubernetes master and includes a temporary token and the command needed to join Kubernetes nodes to the cluster.

Kubernetes Network Overview

The Kubernetes and Virtualbox network will be composed of at least 3 networks shown at the VirtualBox Kubernetes cluster network diagram:

Kubernetes Network Overview

Kubernetes External IPs

The VirtualBox HOST ONLY network will be the network used to access the Kubernetes master and nodes from outside the network, it can be considered the Kubernetes public network for our development environment.

In the diagram, it is shown in green with connections to each Kubernetes machine and a VirtualBox virtual interface vboxnet:

  • K8S-M-1 at eth1: 192.168.50.11
  • K8S-N-1 at eth1: 192.168.50.12
  • K8S-N-2 at eth1: 192.168.50.13
  • vboxnet0 virtual iface: 192.168.50.1 

VirtualBox creates the necessary routes and the vboxnet0 interface:

Applications published using a Kubernetes NodePort will be available at all the IPs assigned to the Kubernetes servers. Example, for an application published at nodePort 30000 the following URLs will allow access from outside the Kubernetes cluster:

  • http://192.168.50.11:30000/
  • http://192.168.50.12:30000/
  • http://192.168.50.13:30000/

See how to Publish an Application Outside Kubernetes Cluster.

It is also possible to access the Kubernetes servers by ssh using those IPs.

VirtualBox NAT Network 

The NAT network interface, with the same IP (10.0.2.15) for all servers, is assigned to the first interface or each VirtualBox machine, it is used to access the external world (LAN & Internet) from inside the Kubernetes cluster. 

In the diagram, it is shown in yellow with connections to each Kubernetes machine and a NAT router that connects to the LAN and the Internet.

For example it is used during the Kubernetes cluster configuration to download the needed packages. Since it is a NAT interface it doesn’t allow inbound connections by default.

Kubernetes POD Network

The internal connections between Kubernetes PODs use a tunnel network with IPS on the CIDR range 192.168.112.0/20 (as configured by our Ansible playbook)

In the diagram, it is shown in orange with connections to each Kubernetes machine using tunnel interfaces.

Kubernetes will assign IPs from the POD Network to each POD that it creates. POD IPs are not accessible from outside the Kubernetes cluster and will change when PODs are destroyed and created.

Kubernetes Cluster Network (Cluster-IP)

The Kubernetes Cluster Network is a private IP range used inside the cluster to give each Kubernetes service a dedicated IP. 

In the diagram, it is shown in purple.

As shown in the following example, a different CLUSTER-IP is assigned to each service:

Cluster-IPs can’t be accessed from outside the Kubernetes cluster therefore a NodePort is created (or a LoadBalancer in a Cloud provider) to publish an app. NodePort uses the Kubernetes external IPs. 

See “Publish an Application outside Kubernetes Cluster” for an example of exposing an App in Kubernetes.

Vagrantfile

Lines 1 to 7 define the configuration properties of the Kubernetes cluster:

  • IMAGE_NAME: is the box (virtual machine image) that Vagrant will download and use to create the Kubernetes cluster. We will be using a box identified as “bento/ubuntu-18.04”, it corresponds to a minimal installation of Ubuntu 18.04 packaged by the Bento project.
  • K8S_NAME: is the name that our cluster will have, it is used to identify the join-command file. In or case, the name is “ditwl-k8s-01”, its is an acronym for Demo IT Wonder Lab Kubernetes(aka k8s) and 01, the number of cluster.
  • MASTERS_NUM: number of master nodes, it is used to create a high availability Kubernetes cluster by increasing the number of master nodes (not implemented in the example Ansible code). 
  • MEM: amount of memory in megabytes for each k8s node. We are giving 2048 to each server.
  • CPU: amount of CPUs available to the each k8s node. We are giving  2 CPUs to each server.
  • NODES_NUM = number of workers, the Kubernetes nodes run the pods using a contained runtime, in our demo we are creating 2 nodes.
  • IP_BASE = First three octets of the IP address that will be used to define the VirtualBox Host network and assign IP addresses to the external interface of the Kubernetes machines. The example uses 192.168.50. to produce the following associations:
    • 192.168.50.1 for the vboxnet0
    • 192.168.50.11 for k8s-m-1 (Kubernetes master node 1)
    • 192.168.50.12 for k8s-n-1 (Kubernetes worker node 1)
    • 192.168.50.13 for k8s-n-2 (Kubernetes worker node 2)

Lines 12 to 15  configure the memory and CPU count of the machines.

Masters are created in lines 17 to 40, a loop is used to create MASTERS_NUM master machines with the following characteristics:  

  • The name of the machine is created with the expression “k8s-m-#{i}” that identifies a machine as member of Kubernetes (k8s), a master and the number (the loop variable i).
  • Ansible as provisioner using the playbook “roles/k8s.yml”
  • A manually generated Ansible inventory of machines in two groups k8s-master and k8s-node using name patterns:

    • “k8s-master” => [“k8s-m-[1:#{MASTERS_NUM}]”],
    • “k8s-node” => [“k8s-n-[1:#{NODES_NUM}]”]

      The patterns are converted and used at the  Ansible inventory file. This is an Ansible – Vagrant hack and it is necessary to include the masters and the nodes as we need to regenerate the inventory with changes from any of the machines.
  • Ansible extra vars are defined to modify the default values assigned by the Ansible playbooks.
    • k8s_cluster_name: K8S_NAME
    • k8s_master_admin_user: “vagrant”
    • k8s_master_admin_group: “vagrant”
    • k8s_master_apiserver_advertise_address: “#{IP_BASE}#{i + 10}”
    • k8s_master_node_name: “k8s-m-#{i}”

Worker nodes are created in lines 42 to 64 using similar code as the one used in the masters. The name of the worker nodes is created with the expression “k8s-n-#{j}” that identifies a machine as member of Kubernetes (k8s), a node and the number (the loop variable j).

Ansible

k8s/master

The k8s/master ansible Playbook creates the Kubernetes master node, it uses the following configuration that can be redefined in the Vagrantfile (see Vagrantfile Ansible extra vars).

Since this is a small Kubernetes cluster the default 192.168.0.0/16 pod network has been changed to be 192.168.112.0/20 (a smaller network 192.168.112.0 – 192.168.127.255). It could be modified either at the Ansible master defaults file or at the Vagrantfile as an extra var.

The role requires the installation of some packages that are common to master and worker Kubernetes nodes, since there should be an Ansible roles for each task, the Ansible meta folder is used to list role dependencies and pass the value of the variables (each role has its own variables that are assigned at this step). The master Kubernetes Ansible role has a dependency with k8s/common:

Once the dependencies are met, the playbook for the master role is executed:

 

k8s/node

Each worker node needs to be added to the cluster by executing the join command that was generated on the master node. The join command uses kubeadm join with api-server-endpoint (It is located at Kubernetes master server) and the token  and a hash to validate the root CA public key.

The playbook to install a node is very small, it has a dependency with the k8s/common packaged:

Specific worker node tasks:

 

k8s/common

The k8s/common role is used by the Kubernetes master and worker nodes Ansible playbooks.

Using the Ansible meta folder, the add_packages is added as a dependency:

The add_packages will install the packages listed  at the Ansible variable k8s_common_add_packages_names along with its corresponding repositories and public keys.

Definition of variables:

The Ansible playbook for k8s/common:

roles/add_packages

This Ansible role specializes in installation and removal of packages. The first step adds repositories keys, then packages sources, updates the cache, and finally remove and install  packages:

 

First Steps After Kubernetes Installation

We will use vagrant ssh command to access the servers.

 

Check Syslog

Open a Vagrant SSH to k8s-m-1 and check syslog file for errors

Check that Calico is Running 

Open a Vagrant SSH to k8s-m-1 and execute kubectl get pods –all-namespaces

 

Administer the Kubernetes Cluster from your host

Install kubectl to administer the Kubernetes Cluster from your development host

Copy the Kubernetes config to your local home .kube dir

List the Kubernetes cluster nodes using  kubectl from your development host:

 

Deploying Applications in Kubernetes

Create an Application configuration file that defines the name of the application, labels, number of replicas and the docker images that are needed. Save the file as nginx.yaml.

Apply the configuration to the Kubernetes Cluster

Publish an Application Outside Kubernetes Cluster

In order to access the echo server that we just deployed from outside the cluster a NodePort needs to be created, the NodePort will allow us to access the each of the nginx echo servers (in round robin) from outside the cluster.

In a Kubernetes cluster hosted in a cloud Provider like Azure, Google or AWS a cloud native Load Balancer will be used instead of the NodePort.

Add the following definition to nginx.yaml (the file created before)

Apply the configuration to the Kubernetes Cluster

Check to see if the NodePort hast been created

Access the application with curl (or with a web browser) from the development environment using one of the public Kubernetes Cluster IPs (192.168.50.11, 192.168.50.12, 192.168.50.13) and the nodePort (30000):

Repeat the request and see that the value of Hostname changes as Kubernetes is accessing the different instances of the application among the available PODS.

    The hostname corresponds to the NAME of the different PODS for the nginx deployment:

     

    Kubernetes NodePort Diagram

    The diagram shows the different elements that are involved in publishing an application outside the Kubernetes cluster using a NodePort. The Kubernetes master, not shown, executes a proxy as well.

     

     

    Network traffic when doing curl http://192.168.50.12:30000/

    1. A request to access 192.168.50.12:30000 (Node 1 Port NodePort) is received by the vboxnet router.
    2. It is routed the external IP of the Kubernetes node 1 in port 30000
    3. The requests is sent to the Kubernetes Proxy
    4. The Kubernetes Proxy sends the request to the Cluster IP 10.103.75.9 and port 8082 that has been assigned to the nginx-services-np (ngingx service NodePort).
    5. The ngingx service NodePort sends the request to one of the available pods for the application nginx, it can be any of the pods located at the same Kubernetes node or in a different node (like K8N-N-2)

    Useful Vagrant commands

    Next steps

    After installing Kubernetes using Vagrant, add a service mesh to connect all the microservices, see Installing Istio in Kubernetes under VirtualBox.

     

     


    5
    Leave a Reply

    avatar
    5 Comment threads
    0 Thread replies
    0 Followers
     
    Most reacted comment
    Hottest comment thread
    1 Comment authors
    jsanz Recent comment authors

    This site uses Akismet to reduce spam. Learn how your comment data is processed.

      Subscribe  
    newest oldest most voted
    Notify of
    jsanz
    Guest
    jsanz

    Very descriptive and useful. I will definitely use it in my future projects. Thanks!

    trackback

    […] a previous tutorial I showed How to Install a Kubernetes Cluster using Vagrant and Ansible, in this tutorial I show how to add Istio as a service mesh for that Kubernetes […]

    trackback

    […] See Building a Kubernetes Cluster with Vagrant and Ansible (without Minikube) […]

    trackback

    […] Install a Kubernetes Cluster with Vagrant and Ansible (without Minikube) […]

    trackback

    […] this tutorial I present a way to launch a Postgres database instance in our Kubernetes Vagrant cluster. The Postgres instance will have all its data stored in an remote NFS Server to preserve database […]