Terraform providers are plugins that link Terraform with infrastructure providers by translating HCL into actionable API calls for the specific provider.
Welcome to our tutorial series where we dive into cloud infrastructure deployment using Terraform or OpenTofu on AWS. In this series, the fundamentals are shown, guiding you through the process of minimizing resource usage and simplifying the deployment complexities associated with cloud infrastructure. This tutorial series is a work in progress.
This comprehensive OpenTofu and Terraform tutorial guides you step-by-step through creating infrastructure in AWS using Terraform.
Infrastructure as Code (IaC) helps maintain consistency, enables version control, enhances collaboration among teams, allows for easier replication of environments, streamlines the deployment and management of infrastructure boosting efficiency, and reducing errors in managing complex systems.
How to start building AWS infrastructure with Terraform: Terraform AWS Provider
Read previous sections of the tutorial: AWS with Terraform Tutorial
Terraform AWS Provider Introduction
A basic introduction to the Terraform AWS Provider.
Terraform AWS Provider Authentication
How to configure the Terraform AWS Provider to authenticate with the AWS API to access and modify infrastructure.
Terraform manages its current state, keeping track of resources either locally or in a remote backend.
How to access the Terraform state.
Common Terraform AWS Provider Questions
Can Terraform manage existing AWS resources? How do I manage multiple AWS environments with Terraform?
Other tutorials for creating infrastructure in AWS using Terraform.
Read previous sections of the tutorial:
AWS is a Cloud provider that offers over 200 services. Many services are equivalent to traditional infrastructure services or products run in an on-premises data center but with added features unavailable at most traditional data centers. All services are available through APIs or the AWS Web Console.
The Terraform official AWS provider acts as an abstraction layer that lets Terraform configurations written in HCL define AWS services and infrastructure using code (IaC). Internally Terraform and the AWS provider handle authentication, and make the necessary AWS API calls to query, create, modify, and destroy the resources.
Companies opt to use Terraform and its declarative configuration language HCL as a single IaC tool to manage multi-cloud and hybrid environment infrastructure.
The Terraform AWS Provider needs to authenticate with the AWS API to access and modify infrastructure. There are multiple authentication options, Terraform will use the first available option.
This option is insecure as it requires clear text and shared credential storage inside Terraform files. Since the Terraform should be versioned in git or a similar version control system the credentials could potentially leak outside the organization.
Example of parameter configuration.
Terraform file:
provider "aws" {
region = "us-west-1"
access_key = "access-key"
secret_key = "secret-key"
}The AWS region, access key, and secret key can be provided by defining environment variables: AWS_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY or an AWS_PROFILE or AWS_CONFIG_FILE.
Example of environment variable configuration
Shell:
$ export AWS_REGION="us-west-1" $ export AWS_ACCESS_KEY_ID="access-key" $ export AWS_SECRET_ACCESS_KEY="secret-key"
Terraform file:
provider "aws" {}This approach could be appropriate for CI/CD pipelines where the environment is constantly recreated and not shared between projects.
Using environment variables is not recommended in developer machines as many times the developer is working on different projects or switching between environments. Using the wrong environment variables could lead to changes in the wrong account.
The AWS CLI stores frequently used configuration settings and credentials in local files that are maintained by the AWS CLI. The Terraform AWS provider can make use of the same files for authentication and configuration.
In the tutorial, we will make use of the shared configuration and credentials files by defining the profile to use.
See How to Use and Configure AWS CLI Security for profile definition.
Terraform file specifying only the profile to use "ditwl_infradmin":
#-----------------------------------------
# Provider: AWS
#-----------------------------------------
provider "aws" {
profile = "ditwl_infradmin"
}AWS CLI shared credentials file /home/user/.aws/credentials specifying the access key, secret access key, and account id for the profile "ditwl_infradmin":
[ditwl_infradmin] aws_access_key_id = ABCDEFGHIJKLMN aws_secret_access_key = Yjhasdiuyi13283dahsudyas7123njas6d2132 account_id = 1234567890
AWS CLI shared config file /home/user/.aws/config specifying the region us-east-1 for the profile "ditwl_infradmin":
[profile ditwl_infradmin] region = us-east-1
If Terraform is running inside AWS CodeBuild, ECS, or EKS and the appropriate IAM Task Roles have been configured then Terraform will use the environment variables set by the roles.
Instance profile credentials and region
If Terraform is running inside AWS an EC2 instance with an IAM Instance Profile set, the provider can get the credentials from the EC2 Instance Metadaservice.
An IAM role can be assumed once valid credentials have been provided either by authentication or with a token from a web identity provider.
An external process or script can be used for authentication.
The Terraform state file is a record of the infrastructure managed by Terraform. It keeps track of the resources Terraform manages and their current state. This file is essential for Terraform to understand the changes made to infrastructure over time and to plan and execute updates accurately. The file can be stored locally (local state file) or shared with the team in a remote repository (remote backend).
The default configuration for the state file is to use a local file (.terraform/terraform.tfstate). This file is stored in a subdirectory (.terraform) inside the Terraform configuration.
A backend used for cloud integration with TACOS (Terraform Automation and Collaboration Software) like env0.
Backend implementations using Object Storage in Azure, Tencent Cloud, Google Cloud, and Alibaba Cloud. Postgres database or Kubernetes secrets.
Uses an AWS S3 bucket for storing the state file. The S3 bucket needs to exist before it is used (can't be created in the same Terraform configuration plan).
In the tutorial, we will make use of the S3 remote backend.
Access the Terraform state using the terraform state command. Available subcommand:
The provider has a list of regions and checks if the configured region is listed. For new regions, it takes some time for the AWS provider to update the list and sometimes users don't have downloaded the latest version.
Error: Invalid AWS Region: ap-southeast-5
To fix:
skip_requesting_account_id = true at the provider block.Example:
#-----------------------------------------
# Define Terrraform Providers and Backend
#-----------------------------------------
terraform {
required_version = "> 1.7"
required_providers {
aws = {
source = "aws"
version = "> 5.6"
}
}
}
#-----------------------------------------
# Provider: AWS
#-----------------------------------------
provider "aws" {
profile = "awarala_infradmin"
skip_region_validation = true
default_tags {
tags = {
environment = "pro"
cost_center = "HR-department"
owner = "IT Wonder Lab"
}
}
}Yes, by importing existing resources using terraform import.
There are multiple options:
Terraform and OpenTofu make use of an online registry that contains providers and modules.
Providers and modules in both are mostly equivalent as the registry acts as a delivery point for the binaries of the providers written by the respective maintainers. For the AWS Terraform provider, the maintainer is HashiCorp and OpenTofu generates binaries from a fork.
To differentiate between the two providers, the source name of the provider can include the provider name, for example:
Currently (January 2024) it doesn't matter which provider you choose as long as you don't switch between Terraform and OpenTofu.
Definition of Providers in the Terraform configuration:
#-----------------------------------------
# Define Terrraform Providers and Backend
#-----------------------------------------
terraform {
required_version = "> 1.5"
required_providers {
aws = {
source = "opentofu/aws"
version = "~> 5.0"
}
}
}"Potential provider misconfiguration" warning shown when switching providers:
$ tofu init Initializing the backend... Initializing provider plugins... - Finding opentofu/aws versions matching "~> 5.0"... - Reusing previous version of hashicorp/aws from the dependency lock file - Using previously-installed hashicorp/aws v5.31.0 - Installing opentofu/aws v5.32.1... - Installed opentofu/aws v5.32.1 (signed, key ID 0C0AF313E5FD9F80) Providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://opentofu.org/docs/cli/plugins/signing/ OpenTofu has made some changes to the provider dependency selections recorded in the .terraform.lock.hcl file. Review those changes and commit them to your version control system if they represent changes you intended to make. ╷ │ Warning: Potential provider misconfiguration │ │ OpenTofu has detected multiple providers of type aws (opentofu/aws, hashicorp/aws) which may be a │ misconfiguration. │ │ If this is intentional you can ignore this warning
This tutorial series is a work in progress and will have these sections:
AWS with Terraform: The Essential Guide: Sections
IT Wonder Lab tutorials are based on the diverse experience of Javier Ruiz, who founded and bootstrapped a SaaS company in the energy sector. His company, later acquired by a NASDAQ traded company, managed over €2 billion per year of electricity for prominent energy producers across Europe and America. Javier has over 25 years of experience in building and managing IT companies, developing cloud infrastructure, leading cross-functional teams, and transitioning his own company from on-premises, consulting, and custom software development to a successful SaaS model that scaled globally.
Are you looking for cloud automation best practices tailored to your company?