During development of Terraform plans there is a need to constantly create, destroy and recreate the infrastructure. This is a technique to prevent infrastructure destroy in Terraform.

Deleting infrastructure is perfectly normal during development, but once you are in production it is important to protect some instances and resources from being accidentally destroyed by Terraform.

Some might say that since we are using Terraform and infrastructure as code, the infrastructure can be recreated in minutes, and that is true, so it is not a problem to delete an instance, a database or a VPN, but I believe that once you are in production, it is not acceptable to have your systems and applications down.

Recreating an instance is really easy and a fast operation using AWS and Terraform, but then you have to install software, configure and deploy the application, maybe load data from backup.

In a world of micro-services and cloud native applications, adding instances and removing them from the pool is fast and desirable, but not everything is a micro service or has many clones for high availability and failover.

Think for example in a Cassandra node or an Elasticsearch node, or a traditional Database node.

Your responsibility is to protect those nodes and only in the case of a hard failure or a planned maintenance replace them.

How to prevent infrastructure destroy in Terraform

How do you protect AWS instances from being destroyed by Terraform, or by a user through the AWS control panel or other tools?

AWS instances have some configuration properties for this:

  • Preserve AWS EBS volumes on Termination
  • Enable AWS EC2 Instance Termination Protection
  • Enable AWS RDS instance Deletion protection (Available Sep 2018)

By default, all AWS EBS root device volumes are deleted when the instance terminates. EC2 and RDS instances can be terminated using the AWS API or the AWS control panel.

To change this behavior, in Terraform I like to include a global variable that indicates if the infrastructure is in production or not.

terraform.tfvars

The variable is_production will be false during development of Terraform plans and will be changed to true as soon as the infrastructure is in production or live.

Disable EC2 instance deletion

When creating instances, the value of is_production will be used to set the disable_api_termination and delete_on_termination arguments as shown on the example.

aws_ec2_pro_srv1234.tf

If this configuration is in place, the is_production variable is true and the infrastructure has been deployed, you will have to use the AWS console or edit your Terraform resources to change the values of instance_initiated_shutdown_behavior to be able to destroy the instances.

A third configuration property sets the behavior for an instance shutdown, the default behavior is to stop the instance but it can be changed to destroy if needed.

Disable RDS instance deletion

It is also possible to prevent deletion of a database instance by using de property deletion_protection. The default value of the property if unspecified is false.

aws_rds_pro_rds789.tf

The new functionality to prevent database deletion was included by AWS on Sep 26, 2018.

Controlling this variables can avoid many problems, like when having a Multi AZ RDS data base that has changed availability zone for maintenance or an upgrade from what is specified in Terraform local state. In this case Terraform thinks that the database needs to be recreated, in the “configured” availability zone and proceeds to destroy the data base if the plan is applied. Having deletion_protection enabled will prevent the deletion.


1
Leave a Reply

avatar
1 Comment threads
0 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
0 Comment authors
Recent comment authors

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
newest oldest most voted
Notify of
trackback

[…] : false}”. Uses the value from is_production variable to prevent EC2 instance destroy. See Avoiding AWS instance destroy with Terraform for an […]