Reducing Terraform Drift in Production

Our Terraform runs were showing changes on every apply, even when nothing had actually changed. When the plan is always noisy, it’s easy to miss a change that actually matters.

I picked this up on my own during spare time. Originally thinking one drift fix per sprint. Seeing the reduced change set gave me a dopamine hit and I just kept going until there were none left.

Five sources of drift across CloudFront, ECS, Secrets Manager, and Terraform provider bugs. All cleaned up.

The fixes

Make origin_shield dynamic

• When origin_shield is disabled on a CloudFront distribution, AWS removes it from the refreshed state entirely, causing drift on every plan
• Made the block dynamic so it is only included when enabled
• There was a secondary provider bug when disabling via a dynamic block that needed its own workaround

Fix null_resources

• timestamp() in triggers caused Terraform to show changes every run since the value always differs
• Moved script execution to external data sources to remove drift while still running scripts every apply
• Split dependency installation into a separate null_resource so it only runs once

Fix task definitions

• Every apply created new ECS task definition revisions even with no real changes. Task definitions are tracked by AWS Config so this had a cost implication
• Identified mismatches in environment variable ordering and missing optional properties by comparing Terraform state with the AWS API response
• Standardized ordering and completed missing fields in the template
• Full write-up

Webhook secrets

• An AWS provider issue caused webhook secrets to always show as changed
• Added ignore_changes for the secret value

Resources not managed by Terraform

• Some resources existed in AWS but were intentionally not managed by Terraform, which resulted in drift
• Added the relevant attributes to ignore_changes

This has been shipped and running across all our environments for a couple of months, nothing broke.

This was also a good stress test of how well I understand our infrastructure.

It was a fun exercise.