Terraform Cloud Agents in a Kubernetes Cluster

What are the Terraform Cloud Agents?

Terraform allows companies to provision and manage cloud infrastructure using best practices by defining the infrastructure as code.

Terraform Cloud Agents are small binaries that can be installed in private clouds or on-premises that execute Terraform runs (plan, apply, and destroy) on behalf of the Terraform Cloud.

Terraform Cloud is a SaaS application to provision and manage infrastructure using Terraform through a SOC 2 Type I Compliant and enterprise-ready solution for teams and companies.

Terraform Cloud manages the Terraform state file and pulls changes in infrastructure from VCS providers like GitHub to trigger modifications on infrastructure.

Until now companies that wanted to use Terraform Cloud needed to publish their infrastructure on the Internet, as each Terraform run (plan, apply, destroy) was executed in HashiCorp servers in the public Cloud or had to install an on-premises edition named Terraform Enterprise.

Terraform Cloud Agents are a new functionality announced by HashiCorp on August 12 2020 for its new Terraform Cloud Business Tier along with other enterprise functionality in the security and auditing space.

Why a Company needs a Terraform Agent?

With Terraform Cloud Agents, a company can manage its private infrastructure as code and benefit from all the functionality of Terraform in a SaaS scenario.

Agents are small binaries (or a convenient docker image or a Kubernetes POD) that run inside a company data center (on-premises infrastructure or in public or private cloud or both leveraging hybrid cloud).

Terraform Cloud Business edition can:

• Manage on-premises infrastructure with Terraform.
• Manage private virtual clouds in public clouds like AWS, Azure, or Google.
• Keep all private infrastructure isolated from the Internet.
• Follow a strict configuration management process.
• Enforce better practices.

Terraform Cloud Agents are responsible for contacting the Terraform Cloud server to get instructions and execute the Terraform runs. The company only needs to allow outbound traffic to Terraform Cloud servers and can keep inbound traffic limited or denied.

Managing Internal Infrastructure with Terraform Cloud and its Agents

In this tutorial, we will be using Terraform Cloud Business Edition to manage all the Terraform Infrastructure as code and an agent installed inside a private Kubernetes cluster to execute the plans.

The Kubernetes cluster only has outbound access to the Internet. The agent installed in the cluster will contact Terraform Cloud to get instructions.

Although Terraform is traditionally used to deploy infrastructure and applications are handled with Ansible, the usage of Kubernetes and containers has made it possible to use Terraform to deploy all the infrastructure needed by an Application and the application itself.

Deployment process using Terraform Cloud and its Agents

In our demo, the Color App will be deployed inside our internal Kubernetes Cluster. The Color App created by IT Wonder Lab to test Istio Patterns is a small go web page that changes its background color based on cookies or the value of environment variables.

Updates:

• 18 August 2020: Move PEM Certificates to Terraform Cloud Variables

This tutorial deploys Terraform Agent(s) in Kubernetes using a Terraform plan (of course!). There are not official instructions at Terraform documentation for Kubernetes and IT Wonder Lab has not done extensive testing.

Prerequisites:

• A Terraform Cloud Business account.
• A working Kubernetes Cluster. We will be using the one created in Kubernetes Cluster using Vagrant and Ansible (in 3 minutes).

Terraform Cloud Agent Life Cycle

Our Terraform Cloud Agent will have a life cycle triggered by changes in the infrastructure code of our demo App.

1. A user edits a Terraform plan
2. The changes in the plan are committed to GitHub
3. A hook is executed by GitHub informing Terraform Cloud of the change in code, Terraform Cloud downloads the changes in code and schedules a Terraform run.
4. The Terraform Agent that runs inside the on-premises Kubernetes cluster is continuously asking Terraform Cloud if it has a job for it, now it does because GitHub triggered a Terraform run in a Terraform Cloud workspace. The Terraform Agent pulls the job and the source code
5. The Terraform Agent downloads the needed Terraform binary and the needed providers, in this example our Terraform uses the Kubernetes provider to execute the Terraform plan. It sends the result to Terraform Cloud and if accepted, Terraform Cloud tells the agent to apply the change.
6. The Terraform Agent applies the change. In our Terraform code, we are creating a deployment of the Color App that is stored as a Docker image in a Docker Registry and a Service Port to access the published Application. The results of the execution are sent to Terraform Cloud.

Create a Token for the Terraform Cloud Agent

Open https://app.terraform.io/ and go to Settings -> Agent page to register a new Agent Token.

Click on Manage tokens

Click on the new token

Set a name

We will use ditwl-agent-k8s-01 and click Create token:

The new agent token is shown

Keep the Token Safe: Don't publish or share the Token, it will be used by the Agent to securely register itself with your Terraform Cloud instance.

Store the value of the new agent token, we will use the token and the name as environment variables for our Kubernetes Terraform Cloud Agent deployment.

Deploying the Terraform Cloud Agent in Kubernetes

We will use a local Terraform configuration to deploy the Terraform Cloud Agent in our Kubernetes Cluster.

Create the Terraform Configuration file

Set the values of the Terraform Cloud Agent name and token as ENV variables in the Terraform configuration file:

• TFC_AGENT_NAME
• TFC_AGENT_TOKEN

ditwl-agent-k8s.tf

terraform {
  required_version = "~> 0.12" 
}

#-----------------------------------------
# Default provider: Kubernetes
#-----------------------------------------

provider "kubernetes" {
  #Context to choose from the config file.
  config_context = "kubernetes-admin@ditwl-k8s-01"
  version        = "~> 1.12"
}

#-----------------------------------------
# KUBERNETES DEPLOYMENT COLOR APP
#-----------------------------------------

resource "kubernetes_deployment" "tfc-agent" {
  metadata {
    name = "tfc-agent"
    labels = {
      app   = "tfc-agent"
    } //labels
  }   //metadata

  spec {
    selector {
      match_labels = {
        app   = "tfc-agent"
      } //match_labels
    }   //selector

    #Number of replicas
    replicas = 1

    #Template for the creation of the pod
    template {
      metadata {
        labels = {
          app   = "tfc-agent"
        } //labels
      annotations = {
        "sidecar.istio.io/inject" = "false"
        }
      }   //metadata

      spec {
        container {
          image = "hashicorp/tfc-agent:latest" #Docker image name
          name  = "tfc-agent"         #Name of the container specified as a DNS_LABEL.
                                      #Each container in a pod must have a unique name (DNS_LABEL).
          #Block of string name and value pairs to set in the container's environment
          env {
            name  = "TFC_AGENT_TOKEN"
            value = "GMzAzKlaeajdvQ.atlasv1.YggshHnEiEL5ypyGsXzcS2PMpF6hMd1P7mB0Dgq6fGhF28lMw5byDn6BGdm5DRHjaCU"
          } //env
          env {
            name  = "TFC_AGENT_NAME"
            value = "ditwl-agent-k8s-01"
          } //env
          resources {
            limits {
              cpu    = "1"
              memory = "512Mi"
            } //limits
            requests {
              cpu    = "250m"
              memory = "50Mi"
            } //requests
          }   //resources
        }     //container
      }       //spec
    }         //template
  }           //spec
}             //resource

Initialize, plan, and apply the Terraform Configuration

Run

$terraform init
$terraform plan
$terraform apply

See the execution of terraform apply in our Kubernetes cluster:

Check the Deployment of the Terraform Cloud Agent in the Kubernetes Cluster

Access the Kubernetes Dashboard or use the kubectl command to check the status of the deployment.

Kubernetes dashboard

Check using kubectl

$ kubectl get all
NAME                             READY   STATUS    RESTARTS   AGE
pod/tfc-agent-7ddc594f56-72ljm   1/1     Running   1          72m

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   11d

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tfc-agent   1/1     1            1           72m

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/tfc-agent-7ddc594f56   1         1         1       72m

Check the Status of the Terraform Cloud Agent

Once deployed in Kubernetes, the Terraform Cloud Agent will contact Terraform Cloud and register to receive jobs.

See the agent status

Open https://app.terraform.io/ and go to Settings -> Agent page to see the status.

Our Kubernetes Terraform Cloud Agent has registered and is ready to receive jobs.

Debugging the Terraform Cloud Agent

To debug issues with the Agent or with Terraform, increase the log level, and check agent logs using the Kubernetes Dashboard or the command line.

To increase the log level of the agent and Terraform set a new environment variable in the Terraform Kubernetes Deployment resource and apply the change to Kubernetes using terraform apply

• Name: TFC_AGENT_LOG_LEVEL
• Allowed values: "trace", "debug", "info", "warn", and "error".

Edit ditwl-agent-k8s.tf adding the new TFC_AGENT_LOG_LEVEL ENV variable and its value

          env {
            name  = "TFC_AGENT_NAME"
            value = "ditwl-agent-k8s-01"
          } //env          
         env {
            name  = "TFC_AGENT_LOG_LEVEL"
            value = "DEBUG"
          } //env 

Do not set the environment variable TF_LOG

TF_LOG is used to debug Terraform plans in a command-line execution of Terraform, but if set the agent fails as it is not prepared to deal with debug messages from the Terraform binary in this way. Use instead TFC_AGENT_LOG_LEVEL which will instruct Terraform to output debug information as well.

Apply the change to the Kubernetes cluster

$terraform apply
kubernetes_deployment.tfc-agent: Refreshing state... [id=default/tfc-agent]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
...
kubernetes_deployment.tfc-agent: Modifying... [id=default/tfc-agent]
kubernetes_deployment.tfc-agent: Modifications complete after 1s [id=default/tfc-agent]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Using the command kubectl get the name of the POD running the Agent

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
color-blue-dep-bcb4d8b9c-5qlrw 1/1 Running 0 15m
color-blue-dep-bcb4d8b9c-j5gk4 1/1 Running 0 15m
color-blue-dep-bcb4d8b9c-k4p76 1/1 Running 0 15m
tfc-agent-7cd5545984-szfwx 1/1 Running 0 20m

Stream the POD logs

$kubectl logs -f tfc-agent-7cd5545984-szfwx
2020-08-15T12:57:36.442Z [INFO]  agent: Starting: name=ditwl-agent-k8s-01 version=0.1.2
2020-08-15T12:57:36.442Z [DEBUG] plugin: starting plugin: path=/usr/bin/tfc-agent-core args=[/usr/bin/tfc-agent-core]
2020-08-15T12:57:36.442Z [DEBUG] plugin: plugin started: path=/usr/bin/tfc-agent-core pid=9
2020-08-15T12:57:36.442Z [DEBUG] plugin: waiting for RPC address: path=/usr/bin/tfc-agent-core
2020-08-15T12:57:36.501Z [DEBUG] plugin.tfc-agent-core: plugin address: network=unix address=/tmp/plugin751737478 timestamp=2020-08-15T12:57:36.501Z
2020-08-15T12:57:36.501Z [DEBUG] plugin: using plugin: version=1
2020-08-15T12:57:36.506Z [INFO]  core: Starting: version=0.1.2
2020-08-15T12:57:36.506Z [DEBUG] core: Registering agent...
2020-08-15T12:57:37.053Z [INFO]  core: Agent registered successfully with Terraform Cloud: id=agent-47ao7XsLiLF39btP pool-id=apool-HEdmePLwpjdMwyo7
2020-08-15T12:57:37.069Z [DEBUG] agent: Starting periodic core updates: interval=1h0m0s
2020-08-15T12:57:37.069Z [INFO]  core: Waiting for next job
...
2020-08-15T13:01:42.174Z [DEBUG] terraform.cli: Plan: 2 to add, 0 to change, 0 to destroy.
2020-08-15T13:01:48.582Z [INFO]  terraform: Generating and uploading provider schemas JSON
2020-08-15T13:01:48.582Z [DEBUG] terraform: Running command: cmd="/root/.tfc-agent/component/terraform/runs/run-kxyUydYxPCVA9sBe.plan/bin/terraform providers schema -json"
2020-08-15T13:01:49.371Z [DEBUG] core: Updating status: agent=busy job=running
2020-08-15T13:02:01.676Z [INFO]  terraform: Persisting filesystem to remote storage
2020-08-15T13:02:03.372Z [DEBUG] terraform: Output stream has stopped
2020-08-15T13:02:03.435Z [INFO]  terraform: Finished handling run
2020-08-15T13:02:03.436Z [DEBUG] core: Updating status: agent=idle job=finished
2020-08-15T13:02:03.978Z [INFO]  core: Waiting for next job
2020-08-15T13:02:04.647Z [INFO]  core: Job received: type=apply id=run-kxyUydYxPCVA9sBe
2020-08-15T13:02:04.647Z [INFO]  terraform: Handling run: id=run-kxyUydYxPCVA9sBe type=apply org=ITWonderLab workspace=terraform-cloud-agent-k8s-deploy-app

Color App Terraform Configuration

The following file defines the Terraform configuration that Terraform Cloud will download from the GitHub repository https://github.com/itwonderlab/terraform-cloud-agent-k8s-deploy-app.

This is not a best practice: IT Wonder Lab's best practices for infrastructure include modularizing Terraform configuration. In this example, we define everything in a single file. See other tutorials for Terraform best practices for Kubernetes deployments.

ditwl-k8s-color.tf

# Copyright (C) 2018 - 2020 IT Wonder Lab (https://www.itwonderlab.com)
#
# This software may be modified and distributed under the terms
# of the MIT license.  See the LICENSE file for details.

# -------------------------------- WARNING --------------------------------
# IT Wonder Lab's best practices for infrastructure include modularizing 
# Terraform configuration. 
# In this example, we define everything in a single file. 
# See other tutorials for Terraform best practices for Kubernetes deployments.
# -------------------------------- WARNING --------------------------------

terraform {
  required_version = "~> 0.12"
}

variable "cluster_ca_certificate" {
  description = "CA Certificate PEM Encoded"
}

variable "client_certificate" {
  description = "Client Certificate PEM Encoded"
}

variable "client_key" {
  description = "Client Key PEM Encoded"
}

#-----------------------------------------
# Default provider: Kubernetes
#----------------------------------------- 

provider "kubernetes" {
  version        = "~> 1.12"

  load_config_file = "false"

  host  = "kubernetes.default.svc" #host  = "https://192.168.50.11:6443"

  cluster_ca_certificate =  var.cluster_ca_certificate
  client_certificate =  var.client_certificate
  client_key = var.client_key

}


#-----------------------------------------
# KUBERNETES DEPLOYMENT COLOR APP
#-----------------------------------------

resource "kubernetes_deployment" "color" {
  metadata {
    name = "color-blue-dep"
    labels = {
      app   = "color"
      color = "blue"
    } //labels
  }   //metadata

  spec {
    selector {
      match_labels = {
        app   = "color"
        color = "blue"
      } //match_labels
    }   //selector

    #Number of replicas
    replicas = 1

    #Template for the creation of the pod
    template {
      metadata {
        labels = {
          app   = "color"
          color = "blue"
        } //labels
        annotations = {
          "sidecar.istio.io/inject" = "false"
        }        
      }   //metadata

      spec {
        container {
          image = "itwonderlab/color:latest" #Docker image name
          name  = "color-blue"               #Name of the container specified as a DNS_LABEL. Each container in a pod must have a unique name (DNS_LABEL).

          #Block of string name and value pairs to set in the container's environment
          env {
            name  = "COLOR"
            value = "blue"
          } //env

          #List of ports to expose from the container.
          port {
            container_port = 8080
          } //port          

          resources {
            limits {
              cpu    = "0.5"
              memory = "512Mi"
            } //limits
            requests {
              cpu    = "250m"
              memory = "50Mi"
            } //requests
          }   //resources
        }     //container
      }       //spec
    }         //template
  }           //spec
}             //resource

#-------------------------------------------------
# KUBERNETES DEPLOYMENT COLOR SERVICE NODE PORT
#-------------------------------------------------

resource "kubernetes_service" "color-service-np" {
  metadata {
    name = "color-service-np"
  } //metadata
  spec {
    selector = {
      app = "color"
    } //selector
    session_affinity = "ClientIP"
    port {
      port      = 8080
      node_port = 30085
    } //port
    type = "NodePort"
  } //spec
}   //resource

Deploying the Color App in the Internal Kubernetes Cluster using Terraform Cloud

We can now create a Workspace in Terraform Cloud Business.

Configure the workspace

Open https://app.terraform.io/ and go to Workspaces.

Click new workspace

Under Connect to a version control provider click GitHub.

Select the GitHub repository

The repository has the Terraform configuration that we want to manage with Terraform Cloud, in our example it is:

https://github.com/itwonderlab/terraform-cloud-agent-k8s-deploy-app.

Finish the workspace creation

Click on Create workspace

Wait a few seconds until Terraform Cloud downloads the repository and configures the Workspace.

Configure execution mode

Go to Settings and select that this workspace uses an Agent and click Save Setting.

Add the variables for the Workspace

We will use Terraform Cloud Variables, a functionality that allows us to set the value of some variables in the workspace. This is a great way to remove sensitive values from the Terraform files and the corresponding VCS (GitHub).

Go to Variables and add all the PEM-encoded certificates

In the Variables page of Terraform Cloud add the following variables and their PEM encoded values from .kube/config file:

• cluster_ca_certificate: PEM encoded certificate in certificate-authority-data
• client_certificate: PEM encoded certificate in client-certificate-data
• client_key: PEM encoded certificate in client-key-data

Obtain the values by running the following code that extracts (grep & awk) and encodes (base64 -d -i) the certificates from the $HOME/.kube/config file:

$grep 'certificate-authority-data' $HOME/.kube/config | awk '{print $2}' | base64 -d -i
$grep 'client-certificate-data' $HOME/.kube/config | awk '{print $2}' | base64 -d -i
$grep 'client-key-data' $HOME/.kube/config | awk '{print $2}' | base64 -d -i

The values are multi-line values that need special treatment in Terraform using the shell-style "here doc" syntax.

Place the PEM Certificates between <<EOF and EOF, making sure that there is a blank space before <<EOF, and the lines of the string and the end marker must not be indented.

Example:

[SPACE] <<EOF
-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIB … [REDACTED]
PBI5s7EgYqgwoijVwOZnU=
-----END CERTIFICATE-----
EOF

You can check sensitive options to prevent the value from being shown again.

Execute the Terraform Plan

Go back to Runs or Click Queue plan to deploy the Application in our private Kubernetes Cluster.

Confirm and apply the plan

Check the result

The plan has been applied and the Application has been published in our Kubernetes Cluster:

Access the URL or the Color App

The Terraform configuration for our deployments creates a Kubernetes Service of type NodePort, it is needed to access the Application. See an explanation in Using a NodePort in a Kubernetes Cluster on top of VirtualBox.

The Color App has been published at http://192.168.50.11:30085/ (and in all cluster IPs).

Open http://192.168.50.11:30085/

Trigger a Terraform Cloud Run Publishing a Change in GitHub

A big benefit of using Terraform Cloud (even its free version) is that it forces users to follow a strict configuration management process that involves versioning every infrastructure configuration change before applying it. This configuration management process is critical to rollback and audit changes.

If we edit the Terraform configuration plan for our deployment, for example changing the number of replicas and committing the change to the GitHub repository, Terraform Cloud will start a new Terraform run that will increase the number of replicas of our application.

    #Number of replicas
    replicas = 3

$ git add .
$ git commit -m "Increase replicas to 3"
[master 5cc2b09] Increase replicas to 3
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git push
Enumerating objects: 5, done.
...
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:ITWonderLab/terraform-cloud-agent-k8s-deploy-app.git
   1b659b1..5cc2b09  master -> master 

Terraform plan is running

Terraform Cloud shows a plan running triggered by a change in the VCS.

Confirm and Apply the change to continue.

Kubernetes dashboard shows 3 replicas

Once the plan is confirmed, the agent applies the change that increases the number of replicas to 3 as shown on the Kubernetes Dashboard.

Conclusion

Terraform Cloud Business with its remote agents and the advanced security and auditing functionality is recommended for companies that have used the Terraform free edition and now would like to roll out Terraform across the organization applying better practices and all the requirements that an enterprise software solution provides.

Terraform Agents Need More Functionality: Terraform Agents are new functionality and the cloud platform is not yet able to manage associations between agents and workspaces, something that should be taken into account in enterprises with complex needs or multiple data centers. It is also important to take into account that HashiCorp is transitioning to an enterprise offering that stills require adjustments in processes, software, support, and team experience.