DevSecOps & IaC

Detecting and Preventing Terraform Drift in Production

By ··9 min read

Terraform drift is the gap between what your state file believes exists and what actually exists in your cloud account. It accumulates quietly — and when it is finally discovered, it is often during an incident or a failed apply.

What drift is and why it happens

Drift occurs when the actual state of infrastructure diverges from what Terraform believes it should be. The causes are predictable: engineers make manual changes in the AWS console during incidents and intend to update the Terraform later but do not. Legacy resources existed before Terraform was adopted and were never imported. An on-call engineer added a security group rule at 3am and nobody remembered to remove it.

The consequences vary. In the best case, drift is cosmetic — a tag differs between what Terraform expects and what exists. In the worst case, a terraform apply removes a manually-added security group rule that was mitigating an active attack, or overwrites a manually-updated database parameter that was resolving a performance problem.

terraform plan as a drift detector

terraform plan compares the current state file against actual infrastructure and produces a diff. Running it on a schedule — without applying — is the simplest possible drift detection mechanism. A non-empty plan output means drift exists.

The key is to run plan against a refreshed state. Pass -refresh=true (which is the default) to ensure Terraform queries the actual provider for current state before computing the diff:

# Run plan with detailed exit codes for CI detection
terraform plan -detailed-exitcode -refresh=true

# Exit codes:
# 0 = no changes (no drift)
# 1 = error
# 2 = changes present (drift detected)

The drift detection pipeline

A practical drift detection pipeline runs terraform plan on a schedule, parses the output, and sends a notification when drift is detected. Here is a GitHub Actions workflow that does exactly this:

name: Terraform Drift Detection

on:
  schedule:
    - cron: '0 8 * * 1-5'  # Run at 08:00 UTC weekdays
  workflow_dispatch:

jobs:
  drift-check:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # For OIDC with AWS
      contents: read

    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/terraform-readonly
          aws-region: us-east-1

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.0

      - name: Terraform Init
        run: terraform init
        working-directory: environments/production

      - name: Detect Drift
        id: plan
        run: |
          terraform plan -detailed-exitcode -refresh=true \
            -out=tfplan 2>&1 | tee plan-output.txt
          echo "exitcode=$?" >> $GITHUB_OUTPUT
        working-directory: environments/production
        continue-on-error: true

      - name: Notify on drift
        if: steps.plan.outputs.exitcode == '2'
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "Terraform drift detected in production",
              "blocks": [{
                "type": "section",
                "text": {
                  "type": "mrkdwn",
                  "text": "*Drift detected in production environment.*"
                }
              }]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DRIFT_WEBHOOK }}

Warning: never run terraform apply automatically on drifted state

Automatically applying a plan that was generated from a drifted state risks undoing intentional manual changes — including emergency mitigations or data-preserving configuration changes made during an incident. Drift detection pipelines should notify, not auto-apply. Always review the plan output before deciding whether to apply.

Preventing drift

Detection tells you when drift has occurred. Prevention reduces how often it occurs in the first place. Two mechanisms are particularly effective:

First, enforce infrastructure-as-code-only changes via IAM and SCPs. Remove direct console write access from all engineers for production environments. If changes must go through Terraform, drift from console changes cannot happen. The friction of this policy is usually lower than it appears — most day-to-day work is read-only, and Terraform changes go through the same review process as code.

Second, tag manually-created resources so they are immediately identifiable in Cost Explorer and resource inventories. A tagging convention like managed-by: manual makes drift visible without requiring any tooling.

When drift is intentional: lifecycle ignore_changes

Some drift is intentional. Auto-scaling groups change instance counts at runtime. Databases get parameter changes applied outside of Terraform for performance tuning. Data sources in Terraform — resources whose values are managed externally — should be declared using lifecycle.ignore_changes to tell Terraform not to treat the divergence as drift.

resource "aws_autoscaling_group" "app" {
  name               = "app-asg"
  desired_capacity   = 3
  max_size           = 10
  min_size           = 1

  lifecycle {
    ignore_changes = [desired_capacity]
  }
}

Tooling: Driftctl and Terraformer

For environments where significant un-managed infrastructure exists, two tools are particularly useful. Driftctl scans your cloud account and reports on resources that exist in AWS but are not in any Terraform state file — the category of drift that terraform plan cannot detect because it only knows about what is in state.

Terraformer goes further — it can generate Terraform configuration and state imports for existing resources, which is useful when you want to bring previously-unmanaged infrastructure into Terraform control without re-creating it. Use it with care: the generated configuration is a starting point that typically requires manual refinement before it is production-ready.

Drift management is ultimately a discipline problem more than a tooling problem. Keeping drift low depends on a clear policy about where infrastructure changes can be made, a detection pipeline that makes drift visible quickly, and a regular process for reviewing and resolving it.

About the author

Written by Mahmoud Ali, founder of CoretiTech.

Get in touch →
Detecting and Preventing Terraform Drift in Production | CoretiTech