How to use Ansible and Terraform together?

Linking Terraform and Ansible

See Using Terraform to create an AWS VPC with an EC2 Instance and a MariaDB RDS Database for a tutorial on how to create Cloud infrastructure in AWS using Terraform.

The tutorial assumes that you have already completed the deployment of the infrastructure in AWS from the previous articles.

Prerequisites and source code:

• Terraform Setup and AWS VPC Subnet Creation (1/5)
• VPC Subnet Routing and Internet Access with Terraform (2/5)
• AWS Route 53, AMI Lookup and EC2 Creation with Terraform (3/5)
• AWS RDS MariaDB Database and Resource Tags with Terraform (4/5)
• EC2 Instances and Resource Security with Terraform (5/5)

Using AWS Tags with Ansible

In order to link the Terraform infrastructure with Ansible, we will use the AWS tags created with Terraform to identify the elements and apply Ansible playbooks.

The following screenshot from AWS EC2 Console shows the tags applied to an EC2 instance.

Add at least these tags to all resources:

• Name [name]: the name of the instance or resource. It should be unique and follows the Cloud-Resource-Environment-Visibility-Name/ID format (see EC2 Instances and Resource Security for details)
• Private Name [private_name]: The private name for this element, it is used for DNS registration in private zone and should follow a standard and be unique. It can be used for monitoring and server naming.
• Public Name [public_name]: The public name for the element, it can be used in DNS registration in public zones and can be the same for many instances, as instances can be behind load balancers. In RDS it is the same as the private name.
• App [app]: The name of the main application that will be used in the resource.
• App ID [app_id]: A unique characteristic of the application or a number that can be used to differentiate multiple different instances of the same application, for example imagine you have to releases of the same application in the same environment, App ID could be the release number.
• OS [os]: The operating system of the instance, useful for applying basic configuration.
• Environment [environment]: Used for environment identification, it is a 3 letter acronym for the environment:
• Cost Center [cost_center]:  one or many cost centers that this resource should be assigned to. The cost center is used in billing to classify resources, for example if you provide resources for different customers, some resources are shared and other are costs associated to a specific customer.

Values should all be in lowercase without spaces.

Setup

The tutorial assumes that you have already completed the deployment of the AWS infrastructure using Terraform following the tutorial Using Terraform to create an AWS VPC with an EC2 Instance and a MariaDB RDS Database.

Private Key

Copy the private key file created when deploying the infrastructure to ${HOME}/keys/ditwl_kp_infradmin.pem

For simplicity, we are using the same private key for everything, but in a real scenario, I recommend having a different key for each combination of:

• cloud
• environment
• user

See Multiple Environments below for an explanation.

Set the correct access permissions for the private key file:

chmod 600 $HOME/keys/ditwl_kp_infradmin.pem

AWS Inventory Access Permissions

Set the correct access permissions for the inventory file that is part of the downloaded source code:

cd ansible-aws-ec2-terraform-tags
chmod 755 inventory/ec2.py

Ansible Configuration

Review the configuration from the file ditwl_pro.sh

# source this file before using ansible
# source ditwl_pro.sh
export AWS_PROFILE='ditwl_infradmin'
ANSIBLE_PRIVATE_KEY_FILE=${HOME}/keys/ditwl_kp_infradmin.pem

#AWS Region
export EC2_REGION='us-east-1'

#Set
export ANSIBLE_INVENTORY=$(pwd)/inventory/ec2.py
export ANSIBLE_PRIVATE_KEY=$ANSIBLE_PRIVATE_KEY_FILE

The value of AWS_PROFILE is the name used for the profile that has the AWS credentials. It will be stored outside Ansible ~/.aws/credentials and its is the same used in Terraform.

The EC2_REGION is the AWS region where the Cloud services are located. Setting the region speeds up Ansible AWS inventory creation.

The ANSIBLE_INVENTORY is a path to a file containing an Inventory of Hosts or a script. We will be using a script to generate the inventory dynamically by querying the AWS API.

The ANSIBLE_PRIVATE_KEY points to a file that will be used for SSH authentication when connecting to the AWS EC2 Linux hosts.

Before running any ansible commands, source the configuration file ditwl_pro.sh to set the appropriate environment variables.

ansible-aws-ec2-terraform-tags$ source ditwl_pro.sh
ANSIBLE_INVENTORY: /home/jruiz/.../ansible-aws-ec2-terraform-tags/inventory/ec2.py
ANSIBLE_PRIVATE_KEY: /home/jruiz/keys/ditwl_kp_infradmin.pem

Manual Activities for WordPress

Manually create a MariaDB schema for WordPress and delegate the DNS zone to the Terraform created DNS servers, these two activities have not been automated in the example.

Ansible Dynamic Inventory

Ansible will run and cache the results of the dynamic inventory when a playbook is applied.

We can also run the dynamic inventory script to obtain a JSON representation of all the groups and properties of our AWS Infrastructure.

ansible-aws-ec2-terraform-tags$ inventory/ec2.py
{
  "_meta": {
    "hostvars": {
      "ditwl_ec2_pro_pub_wp01_1": {
        "ansible_host": "52.3.235.198", 
        "ec2__in_monitoring_element": false, 
        "ec2_account_id": "368675470651", 
        "ec2_ami_launch_index": "0", 
        "ec2_architecture": "x86_64", 
        "ec2_block_devices": {
          "sda1": "vol-07c617da3623854c0"
        }, 
        "ec2_client_token": "", 
        "ec2_dns_name": "ec2-52-3-235-198.compute-1.amazonaws.com", 
        "ec2_ebs_optimized": false, 
        "ec2_eventsSet": "", 
        "ec2_group_name": "", 
        "ec2_hypervisor": "xen", 
        "ec2_id": "i-071566e036e992733", 
        "ec2_image_id": "ami-43a15f3e", 
        "ec2_instance_profile": "", 
        "ec2_instance_type": "t2.micro", 
        "ec2_ip_address": "52.3.235.198", 
        "ec2_item": "", 
        "ec2_kernel": "", 
        "ec2_key_name": "ditwl_kp_infradmin", 
        "ec2_launch_time": "2018-03-15T14:11:57.000Z", 
        "ec2_monitored": false, 
        "ec2_monitoring": "", 
        "ec2_monitoring_state": "disabled", 
        "ec2_persistent": false, 
        "ec2_placement": "us-east-1a", 
        "ec2_platform": "", 
        "ec2_previous_state": "", 
        "ec2_previous_state_code": 0, 
        "ec2_private_dns_name": "ip-172-17-32-217.ec2.internal", 
        "ec2_private_ip_address": "172.17.32.217", 
        "ec2_public_dns_name": "ec2-52-3-235-198.compute-1.amazonaws.com", 
        "ec2_ramdisk": "", 
        "ec2_reason": "", 
        "ec2_region": "us-east-1", 
        "ec2_requester_id": "", 
        "ec2_root_device_name": "/dev/sda1", 
        "ec2_root_device_type": "ebs", 
        "ec2_security_group_ids": "sg-0ec2a678,sg-1fd2b669", 
        "ec2_security_group_names": "ditwl-sg-ec2-pro-pub-01,ditwl-sg-ec2-def", 
        "ec2_sourceDestCheck": "true", 
        "ec2_spot_instance_request_id": "", 
        "ec2_state": "running", 
        "ec2_state_code": 16, 
        "ec2_state_reason": "", 
        "ec2_subnet_id": "subnet-23b4be47", 
        "ec2_tag_Name": "ditwl-ec2-pro-pub-wp01-1", 
        "ec2_tag_app": "wp", 
        "ec2_tag_app_id": "wp-01", 
        "ec2_tag_cost_center": "ditwl-permanent", 
        "ec2_tag_environment": "pro", 
        "ec2_tag_os": "ubuntu", 
        "ec2_tag_os_id": "ubuntu-16", 
        "ec2_tag_private_name": "ditwl-ec2-pro-pub-wp-01", 
        "ec2_tag_public_name": "www", 
        "ec2_virtualization_type": "hvm", 
        "ec2_vpc_id": "vpc-a970cbd2"
      }
    }
  }, 
  "ami_43a15f3e": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "ec2": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "i-071566e036e992733": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "key_ditwl_kp_infradmin": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "platform_undefined": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "security_group_ditwl_sg_ec2_def": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "security_group_ditwl_sg_ec2_pro_pub_01": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_Name_ditwl_ec2_pro_pub_wp01_1": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_app_id_wp_01": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_app_wp": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_cost_center_ditwl_permanent": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_environment_pro": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_os_id_ubuntu_16": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_os_ubuntu": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_private_name_ditwl_ec2_pro_pub_wp_01": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_public_name_www": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "type_t2_micro": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "us-east-1": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "us-east-1a": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "vpc_id_vpc_a970cbd2": [
    "ditwl_ec2_pro_pub_wp01_1"
  ]
}

hostvars

For each EC2 instance, a node is added to hostvars, the node contains properties that will be available when running a playbook.

Examples:

In Ansible Dynamic Inventory the EC2 instance “ditwl_ec2_pro_pub_wp01_1” has the following properties:

{
  "_meta": {
    "hostvars": {
      "ditwl_ec2_pro_pub_wp01_1": {
        "ansible_host": "52.3.235.198", 
        ...
        "ec2_id": "i-071566e036e992733", 
        ...
        "ec2_private_ip_address": "172.17.32.217", 
        ...
        "ec2_tag_Name": "ditwl-ec2-pro-pub-wp01-1", 
        ...
      }
    }
  },

Those properties are used inside Ansible Playbooks, in the following example the ec2_tag_Name is used to set the instance hostname:

- name: Set hostname
  hostname:
    name: "{{ ec2_tag_Name }}"

The value of ec2_tag_Name will be extracted from the corresponding Inventory host entry.

Tags

Instance Tags are added to the hostvars properties as shown in the previous entry and also a group is created.

The Ansible Dynamic Inventory creates groups for each tag that is present in the AWS VPC and adds the name of the hosts inside the group.

This grouping is used to target our ansible playbooks to selected operating systems, applications and releases.

The pattern starts with the name “tag” and adds the name of the tag after that.

tag_[tag_name]_[tag_value]:[
 "host_a"
 "host_b"
 "host_c"
]

Example:

{
  "_meta": {
  ...
  "tag_environment_pro": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_os_id_ubuntu_16": [
    "ditwl_ec2_pro_pub_wp01_1"
  ], 
  "tag_os_ubuntu": [
    "ditwl_ec2_pro_pub_wp01_1"
  ],
  ...
}

The group tag_environment_pro corresponds to all EC2 instances that have a tag with the name environment and value pro.

The group tag_os_ubuntu corresponds to all EC2 instances that have a tag with name os and value ubuntu.

The group tag_os_id_ubuntu_16 corresponds to all EC2 instances that have a tag with name os_id and value ubuntu_16.

From our Terraform configuration file, we see that the EC2 instance aws_ec2_pro_pub_wp_01 has those tags.

  aws_ec2_pro_pub_wp_01 = {
    name              = "ditwl-ec2-pro-pub-wp01"
    ....
    tag_private_name  = "ditwl-ec2-pro-pub-wp-01"
    tag_public_name   = "www"
    tag_app           = "wp"
    tag_app_id        = "wp-01"
    tag_os            = "ubuntu"
    tag_os_id         = "ubuntu-16"
    tags_environment  = "pro"
    tag_cost_center   = "ditwl-permanent"
    tags_volume       = "ditwl-ec2-pro-pub-wp-01-root"

  }

Ansible converts the underscore “_” in values to “-” in Inventory as can be seen in the OS_ID tag.

If we had EC2 instances in a pre environment, the inventory will show another group called tag_environment_pre with the instances.

Example:

{
  "_meta": {
  ...
  "tag_environment_pre": [
    "ditwl_ec2_pre_pub_abc01_1",
    "ditwl_ec2_pre_pub_abc01_2",
    "ditwl_ec2_pre_pub_abc01_3"
  ],
  "tag_environment_pro": [
    "ditwl_ec2_pro_pub_wp01_1"
  ]
  ...
}

Using the AWS Tags as Ansible Targets

Ansible playbooks use the hosts filter to select the target hosts, those are the hosts that will have each role applied.

All resources created with Terraform have tags that will be used by Ansible for:

Configuration:

• Name [name]: Will only be used for naming of instances but will never be used for Ansible group selection.
• Private Name [private_name]:  Will never be used for Ansible group selection but can be used for configuration of a package that needs the private DNS name of the instance.
• Public Name [public_name]: Will never be used for Ansible group selection but can be used for configuration of a package that needs the public DNS name of the instance.
• Cost Center [cost_center]:  Not currently used in Ansible.

Group Selection:

• App [app]:  To select all hosts that run the same application, for example to select all hosts that will run a WordPress instance.
• App ID [app_id]: App ID is an specialization of the App selector, it is used to select all hosts that run the same application when there are many “flavours” or “releases” of the application, for example to select all hosts that will run WordPress 4.9.4.
• OS [os]:  To select all hosts that share the same Operating System.
• OS ID [os_id]:  OS ID is an specialization of the OS. It is used to select all hosts that share the same Operating System release.
• Environment [environment]: Used to select all hosts in the same environment.

We like to have a single playbook file for the environment so that the infrastructure can be configured with a single command.

The file ditwl_pro.yml defines the hosts’ selectors and the roles that should be applied to each one. We use :& to AND groups for selectors.

  - hosts: tag_os_ubuntu:&tag_environment_pro
    become: yes
    roles:
      - { role: linux/pam_limits, tags: [ 'pam_limits'] }
      - { role: linux/hosts_file, tags: [ 'hosts_file'] }
      - { role: linux/host_name, tags: [ 'host_name'] }
    tags:
      - common

  - hosts: tag_os_id_ubuntu_16:&tag_environment_pro
    become: yes
    roles:
      - { role: linux/add_packages, tags: [ 'add_packages'] }
    tags:
      - ubuntu_16

  - hosts: tag_app_wp:&tag_environment_pro
    become: yes
    roles:
      - { role: linux/wordpress, tags: [ 'wordpress'] }
    tags:
      - wordpress

Line 1 starts with a hosts selector that will select hosts that are members of the dynamic inventory group tag_os_ubuntu AND also from the group tag_environment_pro.

Resulting hosts are the ones created by Terraform with the tags:

• environment=pro
• os=ubuntu

Ansible will apply the roles:

• linux/pam_limits
• linux/hosts_file
• linux/host_name

Line 10 selects the hosts that have a specific release (or ID) of Ubuntu that are in environment production, in our case the ones tagged by Terraform with:

• os_id = ubuntu-16
• environment=pro

Those hosts will have the role linux/add_packages applied.

This example is used to show how easy is to have a specific set of roles that are only applied to a specific Operating System release or ID.

Line 17 selects the hosts that are tagged as application wp and are in a production environment, it applies the role linux/wordpress.

If we have a more complex example where we need to differentiate from multiple hosts that have the same application tag but have some kind of difference we could use the app_id tag to further select.

Multiple Environments

In case we had a pre environment we could create a new playbook file ditwl_pre.yml and use the tag_environment_pre instead of the tag_environment_pro for hosts selectors.

Each environment should have:

A playbook file that uses tag_environment_ENV as selector, example:

  - hosts: tag_os_ubuntu:&tag_environment_ENV
    become: yes
    roles:
      - { role: linux/pam_limits, tags: [ 'pam_limits'] }
      - { role: linux/hosts_file, tags: [ 'hosts_file'] }
      - { role: linux/host_name, tags: [ 'host_name'] }
    tags:
      - common
  ...

Environment variables for sourcing with the correct SSH key file.

Imagine applying the PRE environment configuration to the PRO environment. It will be a terrible error.

To prevent that from happening we like to have different SSH Key files for each environment and for each user that has to have SSH access either by console or using Ansible.

The file that has the private key for SSH is configured as and environment variable ANSIBLE_PRIVATE_KEY_FILE in the ditwl_ENV.sh file

export AWS_PROFILE='ditwl_infradmin'
ANSIBLE_PRIVATE_KEY_FILE=${HOME}/keys/ditwl_kp_ENV_infradmin.pem
...

Since the ditwl_ENV.sh is stored in the shared source code repository it should not have specific user information, as it points to the private key file, we should instead agree on the location (i.e. ${HOME}/keys/ditwl_kp_ENV_infradmin.pem) but have our own private key as its content.

Each user, therefore, needs to create a PEM file with the agreed name and place its private key inside.

Ansible Playbook Structure

Ansible Playbook structure is defined in official documentation but the recommended way to group hosts and apply roles is something that each user would have to decide for itself.

• Avoid using individual hostnames to select hosts. In the cloud, all hosts should be treated as cattle, not as pets.
• Create groups of hosts by:
  • Operating System and mayor release
  • Application and application release
  • Environment
• Give safe defaults to all roles and use the group_vars to redefine values
• Use a single playbook definition file for each environment

Ansible Roles Granularity

We recommend building Ansible roles that are highly reusable by configuration, but we also recommend pragmatism as our highest priority.

Roles for systems and applications designed to be standalone, or before the Cloud was around, are called “non-native cloud applications”, often those are stateful applications that can not be clustered without sharing the underlying storage and in general are not designed to be recreated or distribute the load between servers.

For the “non-native cloud applications” apply pragmatism and don’t make a big effort in creating reusable roles, instead, create a single role that makes all the necessary changes and configurations for the application even if it has actions available in other roles. It is better to have a working role than spending a huge amount of time fixing complicated dependencies.

You can find an example of such a role in the installation of WordPress. That role configures Nginx, PHP, WordPress and adds Let’s Encrypt – Free SSL/TLS Certificates by requesting a certificate and automating the necessary renew tasks.

Using a single role instead of three or four roles avoids the complexity that a single Nginx role will have, needing to support too many configurations to be reusable for WordPress and also allowing for a high degree of specialization.

An example of a role defined for re-usability is the add_package role, it was designed to install default packages defined in all group_vars and it is also used as a dependency for the WordPress role.

By defining a dependency with parameters inside the meta folder, we add other roles that should be executed before.

The add_packages role is used at the begging of the playbook installing common software in all machines, and later on by using dependencies to add the specific packages needed for role.

dependencies:
  - { role: add_packages,
    linux_add_packages_repositories: "{{ wordpress_add_packages_repositories }}",
    linux_add_packages_keys: "{{ wordpress_add_packages_keys }}",
    linux_add_packages_names: "{{wordpress_add_packages_names }}"
    }

Run the playbook

Click on the play button to see the execution of Ansible playbook.

Configure WordPress

Open the URL https://www.demo.itwonderlab.com/