Terraform allows companies to provision and manage cloud infrastructure using best practices by defining the infrastructure as code.
Terraform Cloud Agents are small binaries that can be installed in private clouds or on-premises that execute Terraform runs (plan, apply, and destroy) on behalf of the Terraform Cloud.
Terraform Cloud is a SaaS application to provision and manage infrastructure using Terraform through a SOC 2 Type I Compliant and enterprise-ready solution for teams and companies.
Terraform Cloud manages the Terraform state file and pulls changes in infrastructure from VCS providers like GitHub to trigger modifications on infrastructure.
Until now companies that wanted to use Terraform Cloud needed to publish their infrastructure on the Internet, as each Terraform run (plan, apply, destroy) was executed in HashiCorp servers in the public Cloud or had to install an on-premises edition named Terraform Enterprise.
Terraform Cloud Agents are a new functionality announced by HashiCorp on August 12 2020 for its new Terraform Cloud Business Tier along with other enterprise functionality in the security and auditing space.
With Terraform Cloud Agents, a company can manage its private infrastructure as code and benefit from all the functionality of Terraform in a SaaS scenario.
Agents are small binaries (or a convenient docker image or a Kubernetes POD) that run inside a company data center (on-premises infrastructure or in public or private cloud or both leveraging hybrid cloud).
Terraform Cloud Business edition can:
Terraform Cloud Agents are responsible for contacting the Terraform Cloud server to get instructions and execute the Terraform runs. The company only needs to allow outbound traffic to Terraform Cloud servers and can keep inbound traffic limited or denied.
In this tutorial, we will be using Terraform Cloud Business Edition to manage all the Terraform Infrastructure as code and an agent installed inside a private Kubernetes cluster to execute the plans.
The Kubernetes cluster only has outbound access to the Internet. The agent installed in the cluster will contact Terraform Cloud to get instructions.
Although Terraform is traditionally used to deploy infrastructure and applications are handled with Ansible, the usage of Kubernetes and containers has made it possible to use Terraform to deploy all the infrastructure needed by an Application and the application itself.
In our demo, the Color App will be deployed inside our internal Kubernetes Cluster. The Color App created by IT Wonder Lab to test Istio Patterns is a small go web page that changes its background color based on cookies or the value of environment variables.
Updates:
This tutorial deploys Terraform Agent(s) in Kubernetes using a Terraform plan (of course!). There are not official instructions at Terraform documentation for Kubernetes and IT Wonder Lab has not done extensive testing.
Prerequisites:
Our Terraform Cloud Agent will have a life cycle triggered by changes in the infrastructure code of our demo App.
Open https://app.terraform.io/ and go to Settings -> Agent page to register a new Agent Token.
Click on Manage tokens
Click on the new token
Set a name
We will use ditwl-agent-k8s-01 and click Create token:
The new agent token is shown
Keep the Token Safe: Don't publish or share the Token, it will be used by the Agent to securely register itself with your Terraform Cloud instance.
Store the value of the new agent token, we will use the token and the name as environment variables for our Kubernetes Terraform Cloud Agent deployment.
We will use a local Terraform configuration to deploy the Terraform Cloud Agent in our Kubernetes Cluster.
Set the values of the Terraform Cloud Agent name and token as ENV variables in the Terraform configuration file:
ditwl-agent-k8s.tf
terraform { required_version = "~> 0.12" } #----------------------------------------- # Default provider: Kubernetes #----------------------------------------- provider "kubernetes" { #Context to choose from the config file. config_context = "kubernetes-admin@ditwl-k8s-01" version = "~> 1.12" } #----------------------------------------- # KUBERNETES DEPLOYMENT COLOR APP #----------------------------------------- resource "kubernetes_deployment" "tfc-agent" { metadata { name = "tfc-agent" labels = { app = "tfc-agent" } //labels } //metadata spec { selector { match_labels = { app = "tfc-agent" } //match_labels } //selector #Number of replicas replicas = 1 #Template for the creation of the pod template { metadata { labels = { app = "tfc-agent" } //labels annotations = { "sidecar.istio.io/inject" = "false" } } //metadata spec { container { image = "hashicorp/tfc-agent:latest" #Docker image name name = "tfc-agent" #Name of the container specified as a DNS_LABEL. #Each container in a pod must have a unique name (DNS_LABEL). #Block of string name and value pairs to set in the container's environment env { name = "TFC_AGENT_TOKEN" value = "GMzAzKlaeajdvQ.atlasv1.YggshHnEiEL5ypyGsXzcS2PMpF6hMd1P7mB0Dgq6fGhF28lMw5byDn6BGdm5DRHjaCU" } //env env { name = "TFC_AGENT_NAME" value = "ditwl-agent-k8s-01" } //env resources { limits { cpu = "1" memory = "512Mi" } //limits requests { cpu = "250m" memory = "50Mi" } //requests } //resources } //container } //spec } //template } //spec } //resource
Run
$terraform init $terraform plan $terraform apply
See the execution of terraform apply
in our Kubernetes cluster:
Access the Kubernetes Dashboard or use the kubectl
command to check the status of the deployment.
Kubernetes dashboard
Check using kubectl
$ kubectl get all NAME READY STATUS RESTARTS AGE pod/tfc-agent-7ddc594f56-72ljm 1/1 Running 1 72m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 11d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/tfc-agent 1/1 1 1 72m NAME DESIRED CURRENT READY AGE replicaset.apps/tfc-agent-7ddc594f56 1 1 1 72m
Once deployed in Kubernetes, the Terraform Cloud Agent will contact Terraform Cloud and register to receive jobs.
See the agent status
Open https://app.terraform.io/ and go to Settings -> Agent page to see the status.
Our Kubernetes Terraform Cloud Agent has registered and is ready to receive jobs.
To debug issues with the Agent or with Terraform, increase the log level, and check agent logs using the Kubernetes Dashboard or the command line.
To increase the log level of the agent and Terraform set a new environment variable in the Terraform Kubernetes Deployment resource and apply the change to Kubernetes using terraform apply
Edit ditwl-agent-k8s.tf adding the new TFC_AGENT_LOG_LEVEL ENV variable and its value
env { name = "TFC_AGENT_NAME" value = "ditwl-agent-k8s-01" } //env env { name = "TFC_AGENT_LOG_LEVEL" value = "DEBUG" } //env
Do not set the environment variable TF_LOG
TF_LOG is used to debug Terraform plans in a command-line execution of Terraform, but if set the agent fails as it is not prepared to deal with debug messages from the Terraform binary in this way. Use instead TFC_AGENT_LOG_LEVEL which will instruct Terraform to output debug information as well.
Apply the change to the Kubernetes cluster
$terraform apply kubernetes_deployment.tfc-agent: Refreshing state... [id=default/tfc-agent] An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: ~ update in-place ... kubernetes_deployment.tfc-agent: Modifying... [id=default/tfc-agent] kubernetes_deployment.tfc-agent: Modifications complete after 1s [id=default/tfc-agent] Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Using the command kubectl
get the name of the POD running the Agent
$ kubectl get pods NAME READY STATUS RESTARTS AGE color-blue-dep-bcb4d8b9c-5qlrw 1/1 Running 0 15m color-blue-dep-bcb4d8b9c-j5gk4 1/1 Running 0 15m color-blue-dep-bcb4d8b9c-k4p76 1/1 Running 0 15m tfc-agent-7cd5545984-szfwx 1/1 Running 0 20m
Stream the POD logs
$kubectl logs -f tfc-agent-7cd5545984-szfwx 2020-08-15T12:57:36.442Z [INFO] agent: Starting: name=ditwl-agent-k8s-01 version=0.1.2 2020-08-15T12:57:36.442Z [DEBUG] plugin: starting plugin: path=/usr/bin/tfc-agent-core args=[/usr/bin/tfc-agent-core] 2020-08-15T12:57:36.442Z [DEBUG] plugin: plugin started: path=/usr/bin/tfc-agent-core pid=9 2020-08-15T12:57:36.442Z [DEBUG] plugin: waiting for RPC address: path=/usr/bin/tfc-agent-core 2020-08-15T12:57:36.501Z [DEBUG] plugin.tfc-agent-core: plugin address: network=unix address=/tmp/plugin751737478 timestamp=2020-08-15T12:57:36.501Z 2020-08-15T12:57:36.501Z [DEBUG] plugin: using plugin: version=1 2020-08-15T12:57:36.506Z [INFO] core: Starting: version=0.1.2 2020-08-15T12:57:36.506Z [DEBUG] core: Registering agent... 2020-08-15T12:57:37.053Z [INFO] core: Agent registered successfully with Terraform Cloud: id=agent-47ao7XsLiLF39btP pool-id=apool-HEdmePLwpjdMwyo7 2020-08-15T12:57:37.069Z [DEBUG] agent: Starting periodic core updates: interval=1h0m0s 2020-08-15T12:57:37.069Z [INFO] core: Waiting for next job ... 2020-08-15T13:01:42.174Z [DEBUG] terraform.cli: Plan: 2 to add, 0 to change, 0 to destroy. 2020-08-15T13:01:48.582Z [INFO] terraform: Generating and uploading provider schemas JSON 2020-08-15T13:01:48.582Z [DEBUG] terraform: Running command: cmd="/root/.tfc-agent/component/terraform/runs/run-kxyUydYxPCVA9sBe.plan/bin/terraform providers schema -json" 2020-08-15T13:01:49.371Z [DEBUG] core: Updating status: agent=busy job=running 2020-08-15T13:02:01.676Z [INFO] terraform: Persisting filesystem to remote storage 2020-08-15T13:02:03.372Z [DEBUG] terraform: Output stream has stopped 2020-08-15T13:02:03.435Z [INFO] terraform: Finished handling run 2020-08-15T13:02:03.436Z [DEBUG] core: Updating status: agent=idle job=finished 2020-08-15T13:02:03.978Z [INFO] core: Waiting for next job 2020-08-15T13:02:04.647Z [INFO] core: Job received: type=apply id=run-kxyUydYxPCVA9sBe 2020-08-15T13:02:04.647Z [INFO] terraform: Handling run: id=run-kxyUydYxPCVA9sBe type=apply org=ITWonderLab workspace=terraform-cloud-agent-k8s-deploy-app
The following file defines the Terraform configuration that Terraform Cloud will download from the GitHub repository https://github.com/itwonderlab/terraform-cloud-agent-k8s-deploy-app.
This is not a best practice: IT Wonder Lab's best practices for infrastructure include modularizing Terraform configuration. In this example, we define everything in a single file. See other tutorials for Terraform best practices for Kubernetes deployments.
ditwl-k8s-color.tf
# Copyright (C) 2018 - 2020 IT Wonder Lab (https://www.itwonderlab.com) # # This software may be modified and distributed under the terms # of the MIT license. See the LICENSE file for details. # -------------------------------- WARNING -------------------------------- # IT Wonder Lab's best practices for infrastructure include modularizing # Terraform configuration. # In this example, we define everything in a single file. # See other tutorials for Terraform best practices for Kubernetes deployments. # -------------------------------- WARNING -------------------------------- terraform { required_version = "~> 0.12" } variable "cluster_ca_certificate" { description = "CA Certificate PEM Encoded" } variable "client_certificate" { description = "Client Certificate PEM Encoded" } variable "client_key" { description = "Client Key PEM Encoded" } #----------------------------------------- # Default provider: Kubernetes #----------------------------------------- provider "kubernetes" { version = "~> 1.12" load_config_file = "false" host = "kubernetes.default.svc" #host = "https://192.168.50.11:6443" cluster_ca_certificate = var.cluster_ca_certificate client_certificate = var.client_certificate client_key = var.client_key } #----------------------------------------- # KUBERNETES DEPLOYMENT COLOR APP #----------------------------------------- resource "kubernetes_deployment" "color" { metadata { name = "color-blue-dep" labels = { app = "color" color = "blue" } //labels } //metadata spec { selector { match_labels = { app = "color" color = "blue" } //match_labels } //selector #Number of replicas replicas = 1 #Template for the creation of the pod template { metadata { labels = { app = "color" color = "blue" } //labels annotations = { "sidecar.istio.io/inject" = "false" } } //metadata spec { container { image = "itwonderlab/color:latest" #Docker image name name = "color-blue" #Name of the container specified as a DNS_LABEL. Each container in a pod must have a unique name (DNS_LABEL). #Block of string name and value pairs to set in the container's environment env { name = "COLOR" value = "blue" } //env #List of ports to expose from the container. port { container_port = 8080 } //port resources { limits { cpu = "0.5" memory = "512Mi" } //limits requests { cpu = "250m" memory = "50Mi" } //requests } //resources } //container } //spec } //template } //spec } //resource #------------------------------------------------- # KUBERNETES DEPLOYMENT COLOR SERVICE NODE PORT #------------------------------------------------- resource "kubernetes_service" "color-service-np" { metadata { name = "color-service-np" } //metadata spec { selector = { app = "color" } //selector session_affinity = "ClientIP" port { port = 8080 node_port = 30085 } //port type = "NodePort" } //spec } //resource
We can now create a Workspace in Terraform Cloud Business.
Configure the workspace
Open https://app.terraform.io/ and go to Workspaces.
Click new workspace
Under Connect to a version control provider click GitHub.
Select the GitHub repository
The repository has the Terraform configuration that we want to manage with Terraform Cloud, in our example it is:
https://github.com/itwonderlab/terraform-cloud-agent-k8s-deploy-app.
Finish the workspace creation
Click on Create workspace
Wait a few seconds until Terraform Cloud downloads the repository and configures the Workspace.
Configure execution mode
Go to Settings and select that this workspace uses an Agent and click Save Setting.
Add the variables for the Workspace
We will use Terraform Cloud Variables, a functionality that allows us to set the value of some variables in the workspace. This is a great way to remove sensitive values from the Terraform files and the corresponding VCS (GitHub).
Go to Variables and add all the PEM-encoded certificates
In the Variables page of Terraform Cloud add the following variables and their PEM encoded values from .kube/config file:
Obtain the values by running the following code that extracts (grep & awk
) and encodes (base64 -d -i
) the certificates from the $HOME/.kube/config file:
$grep 'certificate-authority-data' $HOME/.kube/config | awk '{print $2}' | base64 -d -i $grep 'client-certificate-data' $HOME/.kube/config | awk '{print $2}' | base64 -d -i $grep 'client-key-data' $HOME/.kube/config | awk '{print $2}' | base64 -d -i
The values are multi-line values that need special treatment in Terraform using the shell-style "here doc" syntax.
Place the PEM Certificates between <<EOF and EOF, making sure that there is a blank space before <<EOF, and the lines of the string and the end marker must not be indented.
Example:
[SPACE] <<EOF
-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIB … [REDACTED]
PBI5s7EgYqgwoijVwOZnU=
-----END CERTIFICATE-----
EOF
You can check sensitive options to prevent the value from being shown again.
Execute the Terraform Plan
Go back to Runs or Click Queue plan to deploy the Application in our private Kubernetes Cluster.
Confirm and apply the plan
Check the result
The plan has been applied and the Application has been published in our Kubernetes Cluster:
The Terraform configuration for our deployments creates a Kubernetes Service of type NodePort, it is needed to access the Application. See an explanation in Using a NodePort in a Kubernetes Cluster on top of VirtualBox.
The Color App has been published at http://192.168.50.11:30085/ (and in all cluster IPs).
Open http://192.168.50.11:30085/
A big benefit of using Terraform Cloud (even its free version) is that it forces users to follow a strict configuration management process that involves versioning every infrastructure configuration change before applying it. This configuration management process is critical to rollback and audit changes.
If we edit the Terraform configuration plan for our deployment, for example changing the number of replicas and committing the change to the GitHub repository, Terraform Cloud will start a new Terraform run that will increase the number of replicas of our application.
#Number of replicas replicas = 3
$ git add . $ git commit -m "Increase replicas to 3" [master 5cc2b09] Increase replicas to 3 1 file changed, 1 insertion(+), 1 deletion(-) $ git push Enumerating objects: 5, done. ... remote: Resolving deltas: 100% (2/2), completed with 2 local objects. To github.com:ITWonderLab/terraform-cloud-agent-k8s-deploy-app.git 1b659b1..5cc2b09 master -> master
Terraform plan is running
Terraform Cloud shows a plan running triggered by a change in the VCS.
Confirm and Apply the change to continue.
Kubernetes dashboard shows 3 replicas
Once the plan is confirmed, the agent applies the change that increases the number of replicas to 3 as shown on the Kubernetes Dashboard.
Terraform Cloud Business with its remote agents and the advanced security and auditing functionality is recommended for companies that have used the Terraform free edition and now would like to roll out Terraform across the organization applying better practices and all the requirements that an enterprise software solution provides.
Terraform Agents Need More Functionality: Terraform Agents are new functionality and the cloud platform is not yet able to manage associations between agents and workspaces, something that should be taken into account in enterprises with complex needs or multiple data centers. It is also important to take into account that HashiCorp is transitioning to an enterprise offering that stills require adjustments in processes, software, support, and team experience.
IT Wonder Lab tutorials are based on the diverse experience of Javier Ruiz, who founded and bootstrapped a SaaS company in the energy sector. His company, later acquired by a NASDAQ traded company, managed over €2 billion per year of electricity for prominent energy producers across Europe and America. Javier has over 25 years of experience in building and managing IT companies, developing cloud infrastructure, leading cross-functional teams, and transitioning his own company from on-premises, consulting, and custom software development to a successful SaaS model that scaled globally.
Are you looking for cloud automation best practices tailored to your company?