RodGrid
Under pressure

Losing an availability zone

Three zones does not mean you can lose one for free. It means each surviving zone has to absorb half the traffic instead of a third — and the headroom for that is a number you either bought or did not.

Spreading across availability zones is the cheapest resilience most teams buy, and the easiest to misread. Three zones sounds like it means a zone can fail. What it means is that when one fails, the other two have to carry everything.

The number that decides it

With traffic split evenly across three zones, each one carries about 33%. Lose one and the survivors carry 50% each. That is a 50% increase on the machines that are still up.

So the question is not whether the design is multi-zone. It is whether each zone runs below roughly two thirds of its capacity in normal times. If your zones sit at 80% on a normal Tuesday, the failure of one does not degrade the service — it takes the other two down in sequence, which is the failure mode that reads as a total outage in the incident report.

  • N+1 — enough spare capacity to lose one unit. With three zones that means each running at ~66% or less.
  • N+2 — survives a zone loss during a deploy or a maintenance window, which is when zone losses are least convenient.
  • Autoscaling is not a substitute — it reacts in minutes, and the load step is instant. It shortens the outage; it does not prevent it.

The parts that do not simply move

Stateless services rebalance more or less on their own. State is where the interesting failures live.

  • Quorum systems — a three-node cluster with one node per zone survives losing one zone and cannot survive losing two. A five-node cluster across three zones is the usual answer, and it is a different bill.
  • Primary databases — if the primary was in the lost zone, the clock that matters is failover time, not zone count.
  • Cross-zone latency — traffic that used to stay inside a zone now crosses one, and a chatty service that makes twenty sequential calls per request feels that twenty times.

Watching it on a diagram

The AZ loss stress preset takes each infrastructure node you own down by the replicas it ran in the lost zone — all of them, for a node with a single replica — and adds a small latency penalty for the traffic that now crosses zones. Run it and read the diagram in the order the failure propagates: the first component to saturate is rarely the one anyone expected, because it is usually the one whose headroom was set by cost rather than by this calculation.

Note The preset models the steady state after the zone is gone, not the seconds during which connections drain and health checks flip. Failover behaviour is a property of your configuration, not of the drawing.
Tip If the design survives the preset comfortably, run it again with the traffic multiplier raised. Zone failures have an inconvenient habit of coinciding with the busy hour rather than with a quiet Sunday.