How to Share Infrastructure in Multiple Terraform Projects?

· Updated · 3 min read · Terraform Best Practices

Share and manage AWS infrastructure with Terraform #

Sharing infrastructure across multiple Terraform projects can help maintain consistency, reduce redundancy, and promote collaboration.

Terraform Data Sources #

This is the most common and recommended approach. Data Sources allow you to import information about existing resources from another project and use them in your configuration. This helps avoid redundancy and promotes modularity. See a Terraform example using Terraform Data Sources.

Terraform Remote State #

This approach stores the state of a Terraform project in a remote location (see Terraform Backends) and reads its outputs from other projects with the terraform_remote_state data source. It gives the other projects read access to the whole state, which can contain secrets, so prefer the data sources of the real resources when possible.

Terraform Modules #

Modules provide a way to encapsulate reusable infrastructure code (see Terraform Modules). You can create a module for your shared infrastructure and then import it into other projects. This helps promote code reuse and consistency.

Shared Backend Configuration #

You can store common configuration values, such as VPC IDs, in a shared service such as HCP Terraform (formerly Terraform Cloud) or the AWS Systems Manager Parameter Store. This allows all projects to access the same configuration values without needing to hard-code them.

Infrastructure as Code (IaC) Tool Integration #

Tools like HCP Terraform, Packer, and Ansible can be integrated to create a complete IaC pipeline. This can help automate the process of building and deploying shared infrastructure across multiple projects. See How to Use Ansible and Terraform Together.

Terraform Workspaces #

Workspaces provide a way to manage multiple environments or configurations within a single Terraform project. This can be useful if you have different sets of infrastructure for development, staging, and production.

Terraform Providers #

Some cloud providers offer Terraform providers that allow you to manage shared resources across multiple projects. For example, the AWS provider has the resource aws_resourcegroups_group, which manages AWS Resource Groups that can be used to group resources across multiple projects.

Using Terraform Data Sources #

With this method, multiple projects can share resources (VPC, Subnets, Route Tables, Security Groups, EBS volumes, Customer Gateways… ) using a Terraform Data Source to find the resource ID in a different Terraform project.

Terraform Data Sources allow you to fetch and use external information within your configuration. This includes information from:

  • Existing resources or systems
  • Cloud providers like AWS
  • Databases
  • Other external services
  • Data Sources are defined using the data block and can import data into your Terraform project without creating any actual resources.

By using Data Sources, separate projects can reference the IDs of these components created in another project and leverage them in their configurations.

Updates:

  • 10/Dec/2023: Updated to Terraform > 1.5 and AWS provider v5.30.0

The following example shows a Terraform Base project that creates the basic AWS infrastructure and a Terraform Web Project that creates an EC2 instance on the subnet created by the Base project.

Base Project #

The base project creates:

  • A VPC named ditwl-vpc
  • A Subnet named ditwl-sn-za-pro-pub-00.

Once created, other projects can use Terraform data sources to query AWS for the IDs of these elements and use them in other Terraform projects.

base/aws-base.tf #

base/aws-base.tf
# Copyright (C) 2018 - 2023 IT Wonder Lab (https://www.itwonderlab.com)
#
# This software may be modified and distributed under the terms
# of the MIT license.  See the LICENSE file for details.
# -------------------------------- WARNING --------------------------------
# IT Wonder Lab's best practices for infrastructure include modularizing 
# Terraform/OpenTofu configuration. 
# In this example, we define everything in a single file. 
# See other tutorials for best practices at itwonderlab.com
# -------------------------------- WARNING --------------------------------

#Define Terrraform Providers and Backend
terraform {
  required_version = "> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"      
    }
  }
}

#-----------------------------------------
# Default provider: AWS
#-----------------------------------------
provider "aws" {
  shared_credentials_files = ["~/.aws/credentials"]
  profile                  = "ditwl_infradmin"
  region                   = "us-east-1" //See BUG https://github.com/hashicorp/terraform-provider-aws/issues/30488
}

# VPC with name ditwl-vpc
resource "aws_vpc" "ditwl-vpc" {
  cidr_block = "172.21.0.0/19" #172.21.0.0 - 172.21.31.254
  tags = {
    Name = "ditwl-vpc"
  }
}

# Subnet with name ditwl-sn-za-pro-pub-00
resource "aws_subnet" "ditwl-sn-za-pro-pub-00" {
  vpc_id     = aws_vpc.ditwl-vpc.id
  cidr_block = "172.21.0.0/23" #172.21.0.0 - 172.21.1.255
  tags = {
    Name = "ditwl-sn-za-pro-pub-00"
  }
}

Terraform Web Project #

The sub-project creates only the needed infrastructure for a simple website. Since it needs a VPC and a Subnet, it uses Terraform data sources to find the IDs of the VPC and the Subnet created by the Base project.

Using Terraform Data Sources find:

  • A VPC named ditwl-vpc.
  • A Subnet named ditwl-sn-za-pro-pub-00 and located in the VPC named ditwl-vpc.

Use the value of the subnet resource to create an EC2 instance on it.

web/aws-web.tf
web/aws-web.tf
# Copyright (C) 2018 - 2023 IT Wonder Lab (https://www.itwonderlab.com)
#
# This software may be modified and distributed under the terms
# of the MIT license.  See the LICENSE file for details.
# -------------------------------- WARNING --------------------------------
# IT Wonder Lab's best practices for infrastructure include modularizing 
# Terraform/OpenTofu configuration. 
# In this example, we define everything in a single file. 
# See other tutorials for best practices at itwonderlab.com
# -------------------------------- WARNING --------------------------------

#Define Terrraform Providers and Backend
terraform {
  required_version = "> 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"          
    }
  }
}

#-----------------------------------------
# Default provider: AWS
#-----------------------------------------
provider "aws" {
  shared_credentials_files = ["~/.aws/credentials"]
  profile                  = "ditwl_infradmin"
  region                   = "us-east-1" //See BUG https://github.com/hashicorp/terraform-provider-aws/issues/30488
}

#Find AMI focal-20.04-amd64
data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

#Find a VPC named "ditwl-vpc"
data "aws_vpc" "ditwl-vpc" {  
  filter {
    name = "tag:Name"
    values = ["ditwl-vpc"]
  }  
}
#Find a Subnet located at the VPC named "ditwl-vpc" with tag Name="ditwl-sn-za-pro-pub-00"
data "aws_subnet" "ditwl-sn-za-pro-pub-00" {
  vpc_id = data.aws_vpc.ditwl-vpc.id
  tags = {
    Name = "ditwl-sn-za-pro-pub-00"
  }
}
# Create an AWS instance in the Subnet "ditwl-sn-za-pro-pub-00"
resource "aws_instance" "ditwl-web-01" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
  subnet_id     = data.aws_subnet.ditwl-sn-za-pro-pub-00.id
  tags = {
    Name = "HelloWorld"
  }
}

Next Steps #

#AWS #OpenTofu #Terraform