AWS SQS and SNS with Terraform: Queues, Topics and Dead-Letter Queues

· 2 min read · Terraform & OpenTofu Tutorials

Decouple applications with queues and topics #

Amazon SQS is a message queue: a producer sends messages and a consumer reads them at its own pace. Amazon SNS is a publish and subscribe topic: one message is delivered to many subscribers (queues, Lambda functions, HTTP endpoints, email). Combined, they implement fan-out.

A queue with a dead-letter queue #

A dead-letter queue (DLQ) receives messages that fail to be processed several times, so that a "poison" message does not block the main queue.

sqs.tf
resource "aws_sqs_queue" "orders_dlq" {
  name                      = "ditwl-orders-dlq"
  message_retention_seconds = 1209600 # 14 days
  sqs_managed_sse_enabled   = true
}

resource "aws_sqs_queue" "orders" {
  name                       = "ditwl-orders"
  visibility_timeout_seconds = 60
  message_retention_seconds  = 345600 # 4 days
  receive_wait_time_seconds  = 20     # long polling, fewer empty requests
  sqs_managed_sse_enabled    = true

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.orders_dlq.arn
    maxReceiveCount     = 5
  })
}

resource "aws_sqs_queue_redrive_allow_policy" "orders_dlq" {
  queue_url = aws_sqs_queue.orders_dlq.id

  redrive_allow_policy = jsonencode({
    redrivePermission = "byQueue"
    sourceQueueArns   = [aws_sqs_queue.orders.arn]
  })
}

Notes:

  • The visibility timeout must be longer than the time a consumer needs to process one message. If you process with Lambda, AWS recommends at least six times the function timeout.
  • Long polling (receive_wait_time_seconds up to 20) reduces cost and empty responses.
  • For FIFO queues use fifo_queue = true, and a name that ends in .fifo.
  • To use your own KMS key instead of SQS-managed encryption, set kms_master_key_id.

An SNS topic #

sns.tf
resource "aws_sns_topic" "events" {
  name              = "ditwl-order-events"
  kms_master_key_id = "alias/aws/sns"
}

When the topic encrypts with a customer managed key, publishers and subscribers need permissions to that key. The AWS managed key (alias/aws/sns) cannot be used for deliveries from some services such as CloudWatch alarms, in which case use your own key.

Subscribe the queue to the topic #

SNS needs permission to send messages to the queue:

fanout.tf
resource "aws_sns_topic_subscription" "orders" {
  topic_arn            = aws_sns_topic.events.arn
  protocol             = "sqs"
  endpoint             = aws_sqs_queue.orders.arn
  raw_message_delivery = true
}

data "aws_iam_policy_document" "orders_queue" {
  statement {
    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.orders.arn]

    principals {
      type        = "Service"
      identifiers = ["sns.amazonaws.com"]
    }

    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_sns_topic.events.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "orders" {
  queue_url = aws_sqs_queue.orders.id
  policy    = data.aws_iam_policy_document.orders_queue.json
}

With raw_message_delivery = true the queue receives the original message and not the SNS JSON envelope.

Other subscriptions #

subscriptions.tf
resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.events.arn
  protocol  = "email"
  endpoint  = "ops@example.com"   # the recipient must confirm by clicking a link
}

Email subscriptions stay pending until confirmed, and Terraform cannot confirm them. Use them for alerts: see CloudWatch alarms.

Filter messages #

Each subscriber can receive only part of the events with a filter policy:

filter.tf
resource "aws_sns_topic_subscription" "eu_orders" {
  topic_arn = aws_sns_topic.events.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.orders.arn

  filter_policy = jsonencode({
    region = ["eu"]
  })
}

Test it #

$ aws sns publish --topic-arn "$(tofu output -raw topic_arn)" \
    --message '{"order":1}' \
    --message-attributes '{"region":{"DataType":"String","StringValue":"eu"}}'
$ aws sqs receive-message --queue-url "$(tofu output -raw queue_url)"

Cost #

Both services charge per million requests, with a monthly free tier. Messages up to 256 KB count as one request, and larger payloads are billed in 64 KB chunks. Always set a retention and monitor the DLQ with an alarm on ApproximateNumberOfMessagesVisible.

#AWS #Sqs #Sns #Terraform #OpenTofu