Infrastructure as Code was supposed to make infrastructure boring. Done poorly, it does the opposite: it adds a fragile state file, a new way to cause outages, and a false sense of safety. Here is how to keep IaC an asset rather than a liability.
The state file is the whole ballgame
Terraform’s state is a JSON map between your code and the real world. It is also the single most dangerous object in your infrastructure. Corrupt it, lose it, or let two people write it at once, and you can lose the ability to manage resources that are very much still running and billing you.
terraform {
backend "s3" {
bucket = "acme-tfstate"
key = "prod/network.tfstate"
region = "us-east-1"
dynamodb_table = "tf-locks" # prevents concurrent writes
encrypt = true
}
}The dynamodb_table line is the one that saves you. Without a lock, two applies racing against the same state will interleave and corrupt it.
Drift is inevitable, so plan for it
Someone will change something in the console. An autoscaler will resize a group. A hotfix will happen at 3am. The gap between your code and reality is called drift, and pretending it won’t happen is how IaC rots.
Detect drift on a schedule
Run terraform plan in CI nightly. A non-empty plan on an unchanged codebase is
drift, and it should page someone during business hours, not surprise you later.
Reconcile deliberately
Either import the manual change into code, or revert it. Never let drift accumulate; it compounds into a state nobody understands.
Make the console read-only where you can
The best drift is the drift that cannot happen. Lock down write access to resources that code owns.
Blast radius is a design decision
A single monolithic state file for your entire company is a single point of catastrophic failure. One bad apply can propose destroying the database and the network and the DNS in one plan. The mitigation is to split state along blast-radius boundaries.
Review the plan, not the code
The pull request diff shows what the code changed. It does not show what will happen to the world. Those are different, and the difference is where outages live. A one-line change can, through a chain of references, propose replacing a resource, which for a database means downtime and data loss.
| Symbol in plan | Meaning | Reviewer reaction |
|---|---|---|
+ create | New resource | Usually fine |
~ update in-place | Modify existing | Check the attribute |
-/+ replace | Destroy then recreate | Stop. Is downtime okay? |
- destroy | Remove resource | Stop. Is this intended? |
Automate this: post the plan output as a PR comment, and require a human to approve the plan, not just the code. A -/+ replace on a stateful resource should be impossible to merge without an explicit acknowledgement.
IaC does not remove the possibility of a catastrophic mistake. It makes the mistake reviewable before it happens, but only if someone actually reads the plan.
Keep it boring
The teams that run IaC well are not the ones with the cleverest modules. They are the ones with locked remote state, small blast radii, nightly drift detection, and a discipline of reviewing plans. None of it is exciting. All of it is what keeps infrastructure from becoming the scariest part of your stack.
