AWS with Terraform: The Essential Guide (3/21) - Terraform AWS Provider

The AWS Terraform Provider

Terraform providers are plugins that link Terraform with infrastructure providers by translating HCL into actionable API calls for the specific provider.

Welcome to our tutorial series where we dive into cloud infrastructure deployment using Terraform or OpenTofu on AWS. In this series, the fundamentals are shown, guiding you through the process of minimizing resource usage and simplifying the deployment complexities associated with cloud infrastructure. This tutorial series is a work in progress.

This comprehensive tutorial guides you step-by-step through creating infrastructure in AWS using Terraform.

Terraform Basics, AWS Basics, Terraform AWS Provider, AWS VPC, AWS Subnets, AWS Internet Gateway, AWS NAT Gateway, AWS Routing Tables, AWS Security Groups, AWS Key Pairs, AWS AMIs, AWS EC2 Instances, AWS RDS, AWS Route 53 (DNS), AWS Auto Scaling, AWS Load Balancers, Terraform AWS & Ansible, Terraform Modules, Terraform Backends, Terraform Tools, Terraform CI/CD.

Infrastructure as Code (IaC) helps maintain consistency, enables version control, enhances collaboration among teams, allows for easier replication of environments, streamlines the deployment and management of infrastructure boosting efficiency, and reducing errors in managing complex systems.

How to start building AWS infrastructure with Terraform: Terraform AWS Provider

  1. Prerequisites

    Read previous sections of the tutorial: AWS with Terraform: The Essential Guide

  2. Terraform AWS Provider Introduction

    A basic introduction to the Terraform AWS Provider.

  3. Terraform AWS Provider Authentication

    How to configure the Terraform AWS Provider to authenticate with the AWS API to access and modify infrastructure.

  4. Terraform Remote Backend

    Terraform manages its current state, keeping track of resources either locally or in a remote backend.

  5. Terraform State Management

    How to access the Terraform state.

  6. Common Terraform AWS Provider Questions

    Can Terraform manage existing AWS resources? How do I manage multiple AWS environments with Terraform?

  7. Next Steps

    Other tutorials for creating infrastructure in AWS using Terraform.

Prerequisites

Terraform AWS Provider Introduction

AWS is a Cloud provider that offers over 200 services. Many services are equivalent to traditional infrastructure services or products run in an on-premises data center but with added features unavailable at most traditional data centers. All services are available through APIs or the AWS Web Console.

The Terraform official AWS provider acts as an abstraction layer that lets Terraform configurations written in HCL define AWS services and infrastructure using code (IaC). Internally Terraform and the AWS provider handle authentication, and make the necessary AWS API calls to query, create, modify, and destroy the resources.

Companies opt to use Terraform and its declarative configuration language HCL as a single IaC tool to manage multi-cloud and hybrid environment infrastructure.

Terraform AWS Provider Authentication

The Terraform AWS Provider needs to authenticate with the AWS API to access and modify infrastructure. There are multiple authentication options, Terraform will use the first available option.

Parameters in the provider configuration

This option is insecure as it requires clear text and shared credential storage inside Terraform files. Since the Terraform should be versioned in git or a similar version control system the credentials could potentially leak outside the organization.

Example of parameter configuration.

Terraform file:

provider "aws" {
  region     = "us-west-1"
  access_key = "access-key"
  secret_key = "secret-key"
}

Environment variables

The AWS region, access key, and secret key can be provided by defining environment variables: AWS_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY or an AWS_PROFILE or AWS_CONFIG_FILE.

Example of environment variable configuration

Shell:

$ export AWS_REGION="us-west-1"
$ export AWS_ACCESS_KEY_ID="access-key"
$ export AWS_SECRET_ACCESS_KEY="secret-key"

Terraform file:

provider "aws" {}

This approach could be appropriate for CI/CD pipelines where the environment is constantly recreated and not shared between projects.

Using environment variables is not recommended in developer machines as many times the developer is working on different projects or switching between environments. Using the wrong environment variables could lead to changes in the wrong account.

Shared configuration and credentials files

The AWS CLI stores frequently used configuration settings and credentials in local files that are maintained by the AWS CLI. The Terraform AWS provider can make use of the same files for authentication and configuration.

In the tutorial, we will make use of the shared configuration and credentials files by defining the profile to use.

See How to Use and Configure AWS CLI Security for profile definition.

Terraform file specifying only the profile to use "ditwl_infradmin":

#-----------------------------------------
# Provider: AWS
#-----------------------------------------
provider "aws" {
  profile                  = "ditwl_infradmin"
}

AWS CLI shared credentials file /home/user/.aws/credentials specifying the access key, secret access key, and account id for the profile "ditwl_infradmin":

[ditwl_infradmin]
aws_access_key_id = ABCDEFGHIJKLMN
aws_secret_access_key = Yjhasdiuyi13283dahsudyas7123njas6d2132
account_id = 1234567890

AWS CLI shared config file /home/user/.aws/config specifying the region us-east-1 for the profile "ditwl_infradmin":

[profile ditwl_infradmin]
region = us-east-1

Other authentication methods

Container credentials

If Terraform is running inside AWS CodeBuild, ECS, or EKS and the appropriate IAM Task Roles have been configured then Terraform will use the environment variables set by the roles.

Instance profile credentials and region

If Terraform is running inside AWS an EC2 instance with an IAM Instance Profile set, the provider can get the credentials from the EC2 Instance Metadaservice.

IAM role

An IAM role can be assumed once valid credentials have been provided either by authentication or with a token from a web identity provider.

External Credentials Process

An external process or script can be used for authentication.

Terraform Remote Backend

The Terraform state file is a record of the infrastructure managed by Terraform. It keeps track of the resources Terraform manages and their current state. This file is essential for Terraform to understand the changes made to infrastructure over time and to plan and execute updates accurately. The file can be stored locally (local state file) or shared with the team in a remote repository (remote backend).

Local

The default configuration for the state file is to use a local file (.terraform/terraform.tfstate). This file is stored in a subdirectory (.terraform) inside the Terraform configuration.

Remote

A backend used for cloud integration with TACOS (Terraform Automation and Collaboration Software) like env0.

Other

Backend implementations using Object Storage in Azure, Tencent Cloud, Google Cloud, and Alibaba Cloud. Postgres database or Kubernetes secrets.

S3 & DynamoDB

Uses an AWS S3 bucket for storing the state file. The S3 bucket needs to exist before it is used (can't be created in the same Terraform configuration plan).

In the tutorial, we will make use of the S3 remote backend.

Terraform State Management

Access the Terraform state using the terraform state command. Available subcommand:

  • list,
  • show,
  • pull

Common Terraform AWS Provider Questions

Can Terraform manage existing AWS resources?

Yes, by importing existing resources using terraform import.

How do I manage multiple AWS environments with Terraform?

There are multiple options:

  • Utilize Terraform workspaces.
  • Maintain separate configuration files for each environment.
  • Use a TACOS (Terraform Automation and Collaboration Software) like Terraform Cloud or env0.

Which provider should be used for AWS Terraform provider?

Terraform and OpenTofu make use of an online registry that contains providers and modules.

  • registry.terraform.io for Terraform
  • registry.opentofu.org for OpenTofu

Providers and modules in both are mostly equivalent as the registry acts as a delivery point for the binaries of the providers written by the respective maintainers. For the AWS Terraform provider, the maintainer is HashiCorp and OpenTofu generates binaries from a fork.

To differentiate between the two providers, the source name of the provider can include the provider name, for example:

  • "terraform/aws", will download the Terraform AWS Provider by terraform.
  • "opentofu/aws", will download the Terraform AWS Provider by opentofu.
  • "aws": depending on OpenTofu version will use "terraform/aws" or "opentofu/aws"

Currently (January 2024) it doesn't matter which provider you choose as long as you don't switch between Terraform and OpenTofu.

Definition of Providers in the Terraform configuration:

#-----------------------------------------
# Define Terrraform Providers and Backend
#-----------------------------------------
terraform {
  required_version = "> 1.5"

  required_providers {
    aws = {
      source  = "opentofu/aws"
      version = "~> 5.0"
    }
  }
}

"Potential provider misconfiguration" warning shown when switching providers:

$ tofu init

Initializing the backend...

Initializing provider plugins...
- Finding opentofu/aws versions matching "~> 5.0"...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.31.0
- Installing opentofu/aws v5.32.1...
- Installed opentofu/aws v5.32.1 (signed, key ID 0C0AF313E5FD9F80)

Providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://opentofu.org/docs/cli/plugins/signing/

OpenTofu has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

╷
│ Warning: Potential provider misconfiguration
│ 
│ OpenTofu has detected multiple providers of type aws (opentofu/aws, hashicorp/aws) which may be a
│ misconfiguration.
│ 
│ If this is intentional you can ignore this warning

Next Steps

This tutorial series is a work in progress and will have these sections:

Terraform Basics, AWS Basics, Terraform AWS Provider, AWS VPC, AWS Subnets, AWS Internet Gateway, AWS NAT Gateway, AWS Routing Tables, AWS Security Groups, AWS Key Pairs, AWS AMIs, AWS EC2 Instances, AWS RDS, AWS Route 53 (DNS), AWS Auto Scaling, AWS Load Balancers, Terraform AWS & Ansible, Terraform Modules, Terraform Backends, Terraform Tools, Terraform CI/CD.

AWS with Terraform: The Essential Guide: Sections

  1. AWS EC2 Instances,
  2. AWS RDS,
  3. AWS Route 53 (DNS),
  4. AWS Auto Scaling,
  5. AWS Load Balancers,
  6. Terraform, AWS & Ansible,
  7. Terraform Modules,
  8. Terraform Backends,
  9. Terraform Tools,
  10. Terraform CI/CD

Leave a Reply

Your email address will not be published. Required fields are marked *


Related Cloud Tutorials

Control traffic to AWS resources using security groups
How to configure and use the Terraform aws_security_group and aws_security_group_rule resource blocks to create and manage AWS Security Groups and secure the infrastructure.
AWS Routing Tables with Terraform
How to configure and use the Terraform aws_route_table, aws_route, and aws_main_route_table_association resource blocks to create and manage AWS Routing Tables.
AWS NAT Gateway
How to configure and use the Terraform aws_nat_gateway and aws_eip resource blocks to create and manage AWS NAT Gateway and its corresponding Public IPs inside each availability zone to enable Internet access from instances in private subnets.
An AWS NAT Gateway is a managed service that allows instances in a private subnet to connect to the Internet while keeping them secure. It provides network address translation (NAT) for outbound traffic, allowing resources in a private subnet to access the internet while maintaining a private IP address.
AWS Internet Gateway
How to configure and use the Terraform aws_internet_gateway resource block to create and manage AWS Internet Gateway inside a VPC to enable instances access to and from the Internet.
Javier Ruiz Cloud and SaaS Expert

Javier Ruiz

IT Wonder Lab tutorials are based on the diverse experience of Javier Ruiz, who founded and bootstrapped a SaaS company in the energy sector. His company, later acquired by a NASDAQ traded company, managed over €2 billion per year of electricity for prominent energy producers across Europe and America. Javier has over 25 years of experience in building and managing IT companies, developing cloud infrastructure, leading cross-functional teams, and transitioning his own company from on-premises, consulting, and custom software development to a successful SaaS model that scaled globally.

Are you looking for cloud automation best practices tailored to your company?

linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram