AWS with Terraform Tutorial: Terraform AWS Provider (3)

The AWS Terraform Provider #

Terraform providers are plugins that link Terraform with infrastructure providers by translating HCL into actionable API calls for the specific provider.

Welcome to our tutorial series where we dive into cloud infrastructure deployment using Terraform or OpenTofu on AWS. In this series, the fundamentals are shown, guiding you through the process of minimizing resource usage and simplifying the deployment complexities associated with cloud infrastructure.

This comprehensive OpenTofu and Terraform tutorial guides you step-by-step through creating infrastructure in AWS using Terraform.

Infrastructure as Code (IaC) helps maintain consistency, enables version control, enhances collaboration among teams, allows for easier replication of environments, streamlines the deployment and management of infrastructure boosting efficiency, and reducing errors in managing complex systems.

Prerequisites #

Read the previous sections of the tutorial, listed in the series index at the end of this page.

Terraform AWS Provider Introduction #

AWS is a cloud provider that offers more than 200 services. Many services are equivalent to traditional infrastructure services or products run in an on-premises data center but with added features unavailable at most traditional data centers. All services are available through APIs or the AWS Web Console.

The Terraform official AWS provider acts as an abstraction layer that lets Terraform configurations written in HCL define AWS services and infrastructure using code (IaC). Internally Terraform and the AWS provider handle authentication, and make the necessary AWS API calls to query, create, modify, and destroy the resources.

Companies opt to use Terraform and its declarative configuration language HCL as a single IaC tool to manage multi-cloud and hybrid environment infrastructure.

Terraform AWS Provider Authentication #

The Terraform AWS Provider needs to authenticate with the AWS API to access and modify infrastructure. There are multiple authentication options, Terraform will use the first available option.

Parameters in the provider configuration #

This option is insecure as it requires clear text and shared credential storage inside Terraform files. Since Terraform files should be versioned in Git or a similar version control system, the credentials could leak outside the organization.

Example of parameter configuration.

Terraform file:

provider "aws" {
  region     = "us-west-1"
  access_key = "access-key"
  secret_key = "secret-key"
}

Environment variables #

The AWS region, access key, and secret key can be provided by defining environment variables: AWS_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY or an AWS_PROFILE or AWS_CONFIG_FILE.

Example of environment variable configuration

Shell:

$ export AWS_REGION="us-west-1"
$ export AWS_ACCESS_KEY_ID="access-key"
$ export AWS_SECRET_ACCESS_KEY="secret-key"

Terraform file:

provider "aws" {}

This approach could be appropriate for CI/CD pipelines where the environment is constantly recreated and not shared between projects.

Using environment variables is not recommended in developer machines as many times the developer is working on different projects or switching between environments. Using the wrong environment variables could lead to changes in the wrong account.

Shared configuration and credentials files #

The AWS CLI stores frequently used configuration settings and credentials in local files that are maintained by the AWS CLI. The Terraform AWS provider can make use of the same files for authentication and configuration.

In the tutorial, we will make use of the shared configuration and credentials files by defining the profile to use.

See How to Use and Configure AWS CLI Security for profile definition.

Terraform file specifying only the profile to use “ditwl_infradmin”:

#-----------------------------------------
# Provider: AWS
#-----------------------------------------
provider "aws" {
  profile                  = "ditwl_infradmin"
}

AWS CLI shared credentials file /home/user/.aws/credentials specifying the access key, secret access key, and account id for the profile “ditwl_infradmin”:

[ditwl_infradmin]
aws_access_key_id = ABCDEFGHIJKLMN
aws_secret_access_key = Yjhasdiuyi13283dahsudyas7123njas6d2132
account_id = 1234567890

AWS CLI shared config file /home/user/.aws/config specifying the region us-east-1 for the profile “ditwl_infradmin”:

[profile ditwl_infradmin]
region = us-east-1

Other authentication methods #

Container credentials

If Terraform is running inside AWS CodeBuild, ECS, or EKS and the appropriate IAM Task Roles have been configured then Terraform will use the environment variables set by the roles.

Instance profile credentials and region

If Terraform is running inside an AWS EC2 instance with an IAM instance profile set, the provider can get the credentials from the EC2 Instance Metadata Service.

IAM role

An IAM role can be assumed once valid credentials have been provided either by authentication or with a token from a web identity provider.

External Credentials Process

An external process or script can be used for authentication.

IAM Identity Center (SSO) and web identity (OIDC)

For people, a profile created with aws configure sso gets temporary credentials after a browser login, so no access keys are stored in the computer. For pipelines, a web identity token (OIDC) lets a CI/CD job assume a role without any stored key, as shown in Terraform CI/CD. Both are safer than long-lived access keys.

Terraform Remote Backend #

The Terraform state file is a record of the infrastructure managed by Terraform. It keeps track of the resources Terraform manages and their current state. This file is essential for Terraform to understand the changes made to infrastructure over time and to plan and execute updates accurately. The file can be stored locally (local state file) or shared with the team in a remote repository (remote backend).

Local #

The default configuration for the state file is to use a local file named terraform.tfstate, stored in the directory where Terraform runs. Do not commit it to Git: it can contain sensitive data.

Remote #

A backend used for cloud integration with TACOS (Terraform Automation and Collaboration Software) like env0 or HCP Terraform (the cloud block).

Other #

Backend implementations using Object Storage in Azure, Tencent Cloud, Google Cloud, and Alibaba Cloud. Postgres database or Kubernetes secrets.

S3 & DynamoDB #

Uses an AWS S3 bucket for storing the state file, and either an S3 lock file or a DynamoDB table for state locking. The S3 bucket needs to exist before it is used, so it is created by a separate project.

In the tutorial, we will make use of the S3 remote backend, see Terraform Backends.

Terraform State Management #

Access the Terraform state using the terraform state command. Available subcommands include:

• list,
• show,
• pull,
• mv (rename or move a resource in the state),
• rm (stop managing a resource without destroying it).

Common Terraform AWS Provider Questions #

How to configure the provider for new AWS regions? #

The provider has a list of regions and checks if the configured region is listed. For new regions, it takes some time for the AWS provider to update the list and sometimes users don’t have downloaded the latest version.

Error: Invalid AWS Region: ap-southeast-5

To fix:

• Update the AWS provider to the latest version (check the latest version number on the AWS Provider page)
• Bump the version number in the required_providers configuration
• If it fails, set skip_region_validation = true at the provider block (the example below also shows default_tags).

Example:

#-----------------------------------------
# Define Terrraform Providers and Backend
#-----------------------------------------
terraform {
  required_version = "> 1.7"

  required_providers {
    aws = {
      source  = "aws"
      version = "> 5.6"
    }
  }
}

#-----------------------------------------
# Provider: AWS
#-----------------------------------------
provider "aws" {
  profile = "awarala_infradmin"
  skip_region_validation = true
  default_tags {
    tags = {
      environment = "pro"
      cost_center = "HR-department"
      owner       = "IT Wonder Lab"
    }
  }
}

Can Terraform manage existing AWS resources? #

Yes, by importing them into the state with an import block (or the terraform import / tofu import command). See Terraform Tools.

How do I manage multiple AWS environments with Terraform? #

There are multiple options:

• Utilize Terraform workspaces.
• Maintain separate directories (and states) for each environment that use the same modules.
• Use a TACOS (Terraform Automation and Collaboration Software) like HCP Terraform, env0 or Spacelift.

Which provider should be used for AWS Terraform provider? #

Terraform and OpenTofu make use of an online registry that contains providers and modules.

• registry.terraform.io for Terraform
• registry.opentofu.org for OpenTofu

Providers and modules in both are mostly equivalent as the registry acts as a delivery point for the binaries of the providers written by the respective maintainers. For the AWS Terraform provider, the maintainer is HashiCorp and OpenTofu generates binaries from a fork.

The source of the provider identifies where it is downloaded from:

• hashicorp/aws: the AWS provider published by HashiCorp. It can be downloaded from both registries and works with Terraform and OpenTofu, it is the source used in the later sections of the tutorial.
• opentofu/aws: the build of the fork that the OpenTofu registry published in its early versions.
• aws: a short form without a namespace. Terraform understands it as hashicorp/aws; older versions of OpenTofu used opentofu/aws.

When this was written (January 2024) it did not matter which source you chose as long as you did not switch between Terraform and OpenTofu, and the same is true now if you keep the same source in all your projects.

Definition of Providers in the Terraform configuration:

#-----------------------------------------
# Define Terrraform Providers and Backend
#-----------------------------------------
terraform {
  required_version = "> 1.5"

  required_providers {
    aws = {
      source  = "opentofu/aws"
      version = "~> 5.0"
    }
  }
}

The “Potential provider misconfiguration” warning shown when switching providers (here with the early opentofu/aws source):

$ tofu init

Initializing the backend...

Initializing provider plugins...
- Finding opentofu/aws versions matching "~> 5.0"...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.31.0
- Installing opentofu/aws v5.32.1...
- Installed opentofu/aws v5.32.1 (signed, key ID 0C0AF313E5FD9F80)

Providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://opentofu.org/docs/cli/plugins/signing/

OpenTofu has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

╷
│ Warning: Potential provider misconfiguration
│ 
│ OpenTofu has detected multiple providers of type aws (opentofu/aws, hashicorp/aws) which may be a
│ misconfiguration.
│ 
│ If this is intentional you can ignore this warning

Next Steps #

Continue with AWS VPC, or see the full series index.