How to Share Infrastructure in Multiple Terraform Projects?

Share and manage AWS infrastructure with Terraform

Sharing infrastructure across multiple Terraform projects can help maintain consistency, reduce redundancy, and promote collaboration.

Terraform Data Sources

This is the most common and recommended approach. Data Sources allow you to import information about existing resources from another project and use them in your configuration. This helps avoid redundancy and promotes modularity. See a Terraform example using Terraform Data Sources.

Terraform Remote State

This approach allows you to store the state of one Terraform project in a remote location and then access it from other projects. This can be useful for sharing common infrastructure configurations across multiple environments or teams.

Terraform Modules

Modules provide a way to encapsulate reusable infrastructure code. You can create a module for your shared infrastructure and then import it into other projects. This helps promote code reuse and consistency.

Shared Backend Configuration

You can store common configuration values, such as VPC IDs, in a shared backend like Terraform Cloud or AWS SSM Parameter Store. This allows all projects to access the same configuration values without needing to hard-code them.

Infrastructure as Code (IaC) Tool Integration

Tools like Terraform Cloud, Packer, and Ansible can be integrated to create a complete IaC pipeline. This can help automate the process of building and deploying shared infrastructure across multiple projects. See How to Use Ansible and Terraform Together.

Terraform Workspaces

Workspaces provide a way to manage multiple environments or configurations within a single Terraform project. This can be useful if you have different sets of infrastructure for development, staging, and production.

Terraform Providers

Some cloud providers offer Terraform providers that allow you to manage shared resources across multiple projects. For example, the AWS provider has a data "aws_resourcegroup" resource that allows you to manage AWS Resource Groups, which can be used to group resources across multiple projects.

Using Terraform Data Sources

With this method, multiple projects can share resources (VPC, Subnets, Route Tables, Security Groups, EBS volumes, Customer Gateways... ) using a Terraform Data Source to find the resource ID in a different Terraform project.

Terraform Data Sources allow you to fetch and use external information within your configuration. This includes information from:

• Existing resources or systems
• Cloud providers like AWS
• Databases
• Other external services
• Data Sources are defined using the data block and can import data into your Terraform project without creating any actual resources.

By using Data Sources, separate projects can reference the IDs of these components created in another project and leverage them in their configurations.

Updates:

• 10/Dec/2023: Updated to Terraform > 1.5 and AWS provider v5.30.0

The following example shows a Terraform Base project that creates the basic AWS infrastructure and a Terraform Web Project that creates an EC2 instance on the subnet created by the Base project.

Base Project

The base project creates:

• A VPC named ditwl-vpc
• A Subnet named ditwl-sn-za-pro-pub-00.

Once created, other projects can use Terraform data sources to query AWS for the IDs of these elements and use them in other Terraform projects.

base/aws-base.tf

# Copyright (C) 2018 - 2023 IT Wonder Lab (https://www.itwonderlab.com)
#
# This software may be modified and distributed under the terms
# of the MIT license.  See the LICENSE file for details.
# -------------------------------- WARNING --------------------------------
# IT Wonder Lab's best practices for infrastructure include modularizing 
# Terraform/OpenTofu configuration. 
# In this example, we define everything in a single file. 
# See other tutorials for best practices at itwonderlab.com
# -------------------------------- WARNING --------------------------------

#Define Terrraform Providers and Backend
terraform {
  required_version = "> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"      
    }
  }
}

#-----------------------------------------
# Default provider: AWS
#-----------------------------------------
provider "aws" {
  shared_credentials_files = ["~/.aws/credentials"]
  profile                  = "ditwl_infradmin"
  region                   = "us-east-1" //See BUG https://github.com/hashicorp/terraform-provider-aws/issues/30488
}

# VPC with name ditwl-vpc
resource "aws_vpc" "ditwl-vpc" {
  cidr_block = "172.21.0.0/19" #172.21.0.0 - 172.21.31.254
  tags = {
    Name = "ditwl-vpc"
  }
}

# Subnet with name ditwl-sn-za-pro-pub-00
resource "aws_subnet" "ditwl-sn-za-pro-pub-00" {
  vpc_id     = aws_vpc.ditwl-vpc.id
  cidr_block = "172.21.0.0/23" #172.21.0.0 - 172.21.1.255
  tags = {
    Name = "ditwl-sn-za-pro-pub-00"
  }
}

Terraform Web Project

The sub-project creates only the needed infrastructure for a simple website. Since it needs a VPC and a Subnet, it uses Terraform data sources to find the IDs of the VPC and the Subnet created by the Base project.

Using Terraform Data Sources find:

• A VPC named ditwl-vpc.
• A Subnet named ditwl-sn-za-pro-pub-00 and located in the VPC named ditwl-vpc.

Use the value of the subnet resource to create an EC2 instance on it.

web/aws-web.tf

# Copyright (C) 2018 - 2023 IT Wonder Lab (https://www.itwonderlab.com)
#
# This software may be modified and distributed under the terms
# of the MIT license.  See the LICENSE file for details.
# -------------------------------- WARNING --------------------------------
# IT Wonder Lab's best practices for infrastructure include modularizing 
# Terraform/OpenTofu configuration. 
# In this example, we define everything in a single file. 
# See other tutorials for best practices at itwonderlab.com
# -------------------------------- WARNING --------------------------------

#Define Terrraform Providers and Backend
terraform {
  required_version = "> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"          
    }
  }
}

#-----------------------------------------
# Default provider: AWS
#-----------------------------------------
provider "aws" {
  shared_credentials_files = ["~/.aws/credentials"]
  profile                  = "ditwl_infradmin"
  region                   = "us-east-1" //See BUG https://github.com/hashicorp/terraform-provider-aws/issues/30488
}

#Find AMI focal-20.04-amd64
data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}


#Find a VPC named "ditwl-vpc"
data "aws_vpc" "ditwl-vpc" {  
  filter {
    name = "tag:Name"
    values = ["ditwl-vpc"]
  }  
}
#Find a Subnet located at the VPC named "ditwl-vpc" with tag Name="ditwl-sn-za-pro-pub-00"
data "aws_subnet" "ditwl-sn-za-pro-pub-00" {
  vpc_id = data.aws_vpc.ditwl-vpc.id
  tags = {
    Name = "ditwl-sn-za-pro-pub-00"
  }
}
# Create an AWS instance in the Subnet "ditwl-sn-za-pro-pub-00"
resource "aws_instance" "ditwl-web-01" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
  subnet_id     = data.aws_subnet.ditwl-sn-za-pro-pub-00.id
  tags = {
    Name = "HelloWorld"
  }
}

Next Steps

• How to Use Ansible and Terraform Together.
• AWS with Terraform Tutorial.