Who opened port 5432?
A security scan at a podcast-hosting company flagged the production database: its firewall accepted connections on port 5432, PostgreSQL's port, from any address on the internet. Closing the rule took a minute. Who had added it, and why, nobody could say: the activity log kept 90 days, and the rule was older. Was staging open too? It had been built by hand from a wiki page that never mentioned firewall rules.
Over three years, engineers had built the network, the Kubernetes cluster, the episodes database and the audio bucket by clicking through the cloud console, each change sensible at the time and written down nowhere. Nothing described the result, so it could be neither reviewed nor rebuilt.
Writing down what should exist
Infrastructure as code (IaC) replaces those clicks with files. The team describes each resource, its size, its settings, its firewall rules, in files a tool can read, and commits them to Git. Terraform, with its own configuration language, and Pulumi, with general-purpose languages such as TypeScript or Python, are two common tools. Both are declarative where it matters: the files say what should exist (a PostgreSQL database of this size, reachable only from the cluster's subnet), not the API calls that create it.
That gives the tool two things to compare. The desired state is what the files declare; the current state is what exists in the account. Applying the code computes the difference and makes the calls that close it: create what is missing, change what differs, delete what the files used to declare and no longer do. Run against an empty account, the same code builds the same resources again, though not the data inside them, and "what is in production?" has an answer in a pull request.
Read the plan before anything changes
With no plan or review step before apply, a typo in the config changes production directly, so the tools work in two steps. Plan compares the files with the account and prints what would happen; apply makes it happen. Terraform marks each resource in a plan + create, ~ change in place, - destroy, or -/+ destroy and replace.
The plan is what gets reviewed. A month after the move to code, an engineer renamed the episodes database in a one-line diff. The plan said -/+: the provider could not change that database's identifier in place, so Terraform would delete it and create an empty one. The reviewer read the plan, not just the diff, and the rename never shipped; a revert afterwards would have restored the declaration, not the episodes. Terraform's prevent_destroy setting is the backstop: a plan that would destroy a resource marked with it fails, as long as the setting stays in the files; deleting the resource's whole block deletes the guard too.
The team runs plan on every pull request. After the merge, the pipeline plans again against main and, once someone approves, applies that saved plan file, which Terraform refuses if the state has changed since it was made. Running both steps from the pipeline, not from laptops, gives infrastructure the same review and automation as application code (CI/CD pipelines: integrate, build once, promote).
The file that remembers what was built
To plan, Terraform has to know which real resource each block stands for: that aws_db_instance.episodes is the database with a particular ID. It records that mapping, with the attributes it last saw, in a state file, and Pulumi keeps a similar record. The state is how the tool notices a resource the files no longer declare, so it is not a cache: lose it, and the next plan proposes to create everything again. The company's resources already existed when it moved to code, so each one was imported into the state and its block adjusted until the plan proposed no changes.
So the state lives in a shared remote backend, usually a versioned object storage bucket, not on someone's laptop. It also holds secrets: Terraform writes attributes into the state as plain text, including a database password if the files set one. Limit who can read it, and keep secrets out of the files committed to Git (Secrets management & encryption).
A shared state has a race of its own. Two engineers merged minutes apart, and each merge started an apply, on a backend that simply overwrites the file and has no lock:
- Apply A, which adds a CDN in front of the audio bucket, reads state version 41.
- Apply B, which adds a queue for transcoding jobs, also reads version 41.
- Apply A creates the CDN and writes version 42, which records it.
- Apply B creates the queue and writes its own version 42, from its copy of 41, without the CDN.
- The next plan finds a CDN in the files and none in the state, and proposes to create one, while the real CDN keeps serving and billing, managed by nothing.
A state lock closes the gap: before reading the state, an apply takes a lock in the backend, and a second apply that finds it held waits or stops with an error. Most remote backends support one, through a record in a table or a lock file next to the state; a lock a crashed run left behind is released by hand, after checking that no run still holds it.
Only the pipeline applies. Terraform takes the lock and reads the state before it compares anything, and writes the new state back when it is done.
Changes nobody wrote down
Code does not stop people from using the console. During an outage at 2 a.m., an engineer raised the database's connection limit by hand, and the outage ended. That gap between what the code declares and what exists is drift. A week later an unrelated apply set the limit back to the value in the code, and the outage returned. A manual edit to an attribute that the code declares and the provider tracks, and that the code does not tell Terraform to ignore, is reverted by the next apply, or conflicts with it.
Drift in such an attribute shows up in a plan as a change nobody wrote, so a plan run on a schedule against an unchanged main catches it: with -detailed-exitcode, Terraform exits with code 2 when the plan has changes, and the job alerts. A right change goes into the code through a pull request; a wrong one is put back by an apply. Many teams also make the production console read-only, with a break-glass role whose every use ends in a pull request. That plan sees only the resources in its state, though: one created by hand outside the code, such as a firewall rule added as a resource of its own, appears nowhere until someone finds it in the account and imports it.
Staging that matches production
Code also answers the staging question. Staging and production now call the same module, a reusable set of resource definitions with inputs such as instance sizes and replica counts. Staging runs a small database and production a large one, but their firewall rules come from the same lines: at the same module version, and unless an input, a conditional or a lookup varies them per environment, both get the same rules. Modules are versioned like any code, so each environment pins the version it runs.
The opposite mistake is one giant module and one state for the whole system instead of small composable modules: a giant state makes every plan slow, because the tool refreshes every resource it tracks, and every apply risky, because a typo in the CDN settings goes out in the run that can touch the database. Splitting state by environment and layer keeps plans short and each apply's reach small, at the cost of passing outputs such as a subnet's ID between states.
The same idea reaches the machines inside. The immutable approach never changes a running server but replaces it from a new image, and in the cluster an orchestrator compares running containers with their declaration continuously, where Terraform compares only when run (Containers & orchestration). Changing existing servers in place is the job of tools such as Ansible, and the settings an application reads at runtime belong to Configuration management.
Sync a cluster from Git
In the Code Deployment Platform Challenge, desired deployment state is declared in Git and synced to the cluster automatically: desired against current state, compared without pause. Keep the repository as the only place that state lives, with Argo CD comparing it with the cluster and no edge from a person straight to the cluster. Then mark the path the 2 a.m. fix must take, and what the sync does to a change made on the cluster directly.