Two of everything, and none of it tried
A grocery delivery service takes orders for same-day slots from two cloud regions, east and west. On paper it is ready for trouble: three instances of the order service behind a load balancer in each region, a database that replicates between them, a health check on every instance, an 800 ms timeout on the call to the delivery-slot service, an alert that pages when checkout errors climb, and a runbook called "Evacuate a region". Every item on that list is a claim about how the system will behave during a failure that has not happened yet.
Claims like these go stale quietly. A new code path calls the slot service through a different HTTP client, one without the timeout. The health check answers from the process and never touches the database. West's autoscaling limit was set when east carried a third of the traffic, and nobody has checked whether west can carry all of it. Each piece looks right on its own, so the gaps can get past both the diagram and a code review. A real failure does, and if nothing looked first, that failure is an outage.
Chaos engineering is the practice of causing failures on purpose, under controlled conditions, to find those gaps first. The name suggests breaking things at random; the practice is closer to a lab experiment: say in advance what should happen, cause one fault, limit who can be hurt, and stop once the system crosses a line set in advance.
Start from what a shopper sees
An experiment needs a way to tell "fine" from "not fine" that does not depend on the fault. That is the steady state: the system's normal behaviour, measured from the user's side. For the grocery service it is completed checkouts per minute against the usual curve for that hour, plus the share of checkouts that succeed and how long the slowest of them take. CPU is a poor choice: a machine can run hot while every shopper is served. A service with an SLO already has this measure in its indicator (SLOs & error budgets).
Next comes the hypothesis, written down before anything is touched: "If calls from the order service to the slot service take 3 seconds longer for 5% of checkouts in the east region, the timeout fires, those checkouts continue without a reserved slot and ask the shopper to pick one at the next step, and checkout success, which counts an order accepted with its slot still to pick, stays above 99.5%." It names the fault, its size and the steady state that should survive it, and it can turn out wrong, which is the point: "let's kill something and see" produces a scare, not a conclusion.
One fault, a fence and a way out
The fault is real: the delay is added to actual calls between running services. Other common first faults are terminating an instance, cutting a dependency off, filling a disk, or taking out every instance in one availability zone. Two guardrails go in before any of them starts.
The blast radius is how much of the system, and how many users, the fault can reach. Here the fault is aimed at 5% of checkouts in one region. The team ran it in staging first and moved it to production only once it behaved there: staging has neither production's traffic nor all of its configuration, and some weaknesses only show under both.
The run is planned for ten minutes, after which the injector removes the delay on its own. The abort condition is the line at which it stops sooner, without a meeting: checkout success below 99% for two minutes, measured across the whole region, not just the 5%. A rule watches it and removes the fault on its own, and the engineer running the experiment holds a stop button that does the same. Both are tried before the fault goes in, because a fault nobody can switch off is an outage you started. The on-call engineer is told an experiment is running.
The injector delays the slot-service calls of 5% of checkouts. The abort rule reads checkout success for the whole region and removes the fault when it stays below 99% for two minutes.
The run that went wrong, usefully
- The injector adds 3 seconds to the slot-service calls of 5% of east-region checkouts.
- Checkouts from the basket page reach the 800 ms timeout and continue without a slot, as the hypothesis said.
- Checkouts from the newer "reorder last basket" button use a client with no timeout, and under that morning's load their stuck calls tie up the order service until checkouts that were never delayed queue behind them, the spread Cascading failures: how one dependency takes down its callers explains.
- Region-wide checkout success stays below 99% for two minutes, the abort rule fires, and the injector removes the delay.
- The queue drains within a minute and checkout success returns to its steady state.
The hypothesis was false, and that is the result worth having. The fence around the fault did not fence the damage: the delay reached shoppers it never touched, which is why the abort condition watched the whole region. The reorder path moved to the shared client, and the same experiment held the following week.
Game days put the people in the experiment
Some failures test the people as much as the system: deciding to evacuate a region, finding the runbook, having the access to change routing. A game day is a scheduled exercise in which a team runs one such failure, injected for real or simulated, with the people who would handle it, and measures how it goes. It is announced, has one person who makes the calls, and has an abort condition like any experiment. What it rehearses is the people and the runbook around the switch; the switch itself, and what a region lost without warning costs, is Failover.
The grocery team's first game day evacuated the east region on a weekday morning, before the lunch orders. The runbook said ten minutes. Draining east took 38, twenty of them spent finding someone allowed to change the routing weights, because the engineer holding the runbook was not. Two months later, with that permission granted in advance, the second game day drained east in 14 minutes, and that measured number, not the runbook's guess, is what the team now plans around.
What a green result means
A passing experiment says that this fault, at this size, under that day's traffic and with that version of the code, left the steady state intact. It does not cover the faults nobody injected: two at once, a slow disk rather than a dead one, the same delay at the evening peak. And it ages: the next deploy can add another path like the reorder button. So teams repeat experiments, automate the ones that pass, and widen them step by step, from staging to a small share of production to a whole zone. The stress presets in RodGrid's Simulate panel (Stress testing a design before it exists) can suggest where to aim, but they model faults on a diagram and cannot find a missing timeout in code the diagram does not show.
Rehearse an evacuation on the Grid
Open the Multi-Region Active-Active challenge and take one requirement from its brief: "Rehearse a full region evacuation and measure how long it takes". Geode teaches the build itself; here, write the drill onto it. Say what it measures (the time to drain the region and the success rate meanwhile), the abort condition that stops it, and who decides to start and to stop. A scheduled job may start the drill, but keep it off the failover path, which has to react to health rather than to a timer. For the restore side of the same habit, Backup & Point-in-Time Restore asks for a restore somebody has rehearsed.