Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Failover

Move traffic from a failed primary to a standby without ending up with two primaries: fence the old one before promoting the new, and add up how long detection, promotion and redirect leave clients waiting.

Eighty-five seconds at the turnstiles

A football stadium admits 40,000 people through 120 turnstiles in the hour before kick-off. Each turnstile scans a ticket and asks a validation service whether it is genuine and unused; the service records the scan and the gate opens. The service runs in one zone of a cloud region, with a standby copy in a second zone that receives each scan shortly after it is recorded. Twenty minutes before kick-off, the primary's zone loses power. Whether the queues outside notice depends on one number, how long until the standby answers the turnstiles, and for this stadium it can come to about 85 seconds.

Failover is that switch: notice that the primary is gone, make sure it stays gone, promote the standby to take its place, and move the traffic to it. Each step takes time, and each can go wrong.

Probe, decide, switch

The router sends scans to the primary while the probe's verdict is healthy, and to the standby only after the primary has been fenced and the standby promoted.

Gone, or only out of sight?

Nothing announces that a primary has died. A machine that loses power sends no goodbye; the evidence is silence. A health probe asks the primary every few seconds whether it can serve, and the failover controller decides from the answers. Health Endpoint Monitoring covers what that probe should check. The same silence has other causes: a broken network path between the probe and the primary, a garbage-collection pause of a few seconds, a primary so busy that it answers late. Fail over on each of those and the switch itself becomes the outage.

So detection trades speed for certainty. The stadium's controller probes every 2 seconds with a 1-second timeout and declares the primary down after 5 consecutive failures, about 10 seconds of silence. Probing from more than one place, and requiring the probes to agree, helps tell "the primary is down" from "my path to it is down". A shorter window fails over sooner and more often by mistake; a longer one rides out a pause but keeps the gates waiting longer when the failure is real.

Fence first, then promote

The dangerous case is a primary that is alive and merely unreachable. Say the link between the zones fails instead of the power: the probe can no longer reach the primary, but turnstiles on the primary's side still can. Promote the standby now and two services accept scans at once, each recording tickets as used in its own copy, and a ticket photographed and shared between two friends opens a gate for each of them. Two primaries acting at the same time is split-brain.

A probe cannot rule it out, because a probe can declare a primary dead that is only unreachable. So the old primary is fenced before the standby is promoted, not only when it comes back: made unable to act, for example by cutting its power or network through the platform's management interface, or by revoking its access to storage. The other way is to require a quorum or a lease: the primary may accept scans only while it holds a lease it renews through a majority of nodes, so a primary cut off from that majority stops on its own, and the standby waits for the lease to run out before taking over. Distributed locks & leader election covers leases in depth, and the fencing tokens that refuse a stale primary's writes even when it wakes up late.

Promotion is more than a flag

Promotion turns the standby into the primary: it stops following the primary's stream of scans, opens its own copy for writes and starts accepting traffic. What it can serve depends on what it had received. With asynchronous replication, scans the primary recorded in its last moments may never have reached the standby, and those tickets read as unused after the switch. Replication & read replicas covers what each replication mode can lose.

How long promotion takes depends on how ready the standby is:

  • A hot standby runs at full size, applying the primary's changes as they arrive, and can take traffic within seconds.
  • A warm standby is running with current or nearly current data, often at a smaller size; it has to scale up or warm its caches before it can carry the full load.
  • A cold standby is little more than a plan and a backup: servers are started, sometimes provisioned, and the data restored when the failure happens, which takes minutes to hours.

Capacity is the trap. A standby sized at half the primary to save money receives the full stream of scans at the moment of the switch, with cold caches, and can fall over under it. The failover worked and the gates still stopped.

Moving the traffic

The last step points the turnstiles at the new primary, and its speed depends on how they find the service. A load balancer or a virtual IP in front of both copies can switch in about a second. A DNS record is slower: a client that already holds the old answer keeps using it until its cache entry expires, so a record with a 60-second TTL can leave turnstiles sending scans to the dead address for up to a minute, longer where a resolver or a client keeps answers past their TTL. Clients that hold pooled connections keep using them until each one errors or times out, so they need timeouts short enough to give up on the old primary. In a planned switch, the old primary's connections are drained: it finishes the requests in flight while new ones go to the standby.

Now the stadium's budget can be added up, as a plan rather than a worst case: about 10 seconds to detect, up to 10 more for the old primary's lease to run out, 5 to promote a hot standby and up to 60 for DNS, if clients honour the TTL. Roughly 85 seconds, while every turnstile's queue grows. The stadium's team put a load balancer in front of both zones, which took the redirect from a minute to about a second, and stopped there. A shorter detection window would save a few seconds but fail over on the network blips a 10-second window rides out, each one costing a full switch; a shorter lease would cut the wait before promotion, but the primary would stop serving on any blip that outlasted the lease. They accepted about 25 seconds instead, and settled what the gates do meanwhile, admitting tickets against a list held at each turnstile, as a Fallback, at the risk that a shared ticket opens two gates in that window.

When both copies serve

So far the standby took no traffic until the switch, an arrangement called active–passive. In active–active, both copies serve all the time, and failing over means the router stops sending to the dead one: there is no promotion, and if the survivor can keep scanning on its own and the redirect is quick, detection can be most of the delay. The price comes in two parts. With two equal copies splitting the load evenly, each must run below half its capacity, or the survivor drowns under the combined load. And both copies accept scans for the same tickets, so to stop one ticket opening two gates, the copies must agree before a gate opens, or each ticket must belong to one copy that alone decides it; settling the conflict afterwards cannot undo an entry. Either way, the design has to say what happens when one copy is gone: agreement that needs both copies stalls the gates, and the tickets the dead copy owned must be handed to the survivor safely.

The way back

When zone A's power returns, the old primary comes back stale: it missed every scan recorded during the outage. So failing back is a second failover, planned rather than forced: the old primary rejoins as a standby, catches up from the new primary, and the switch back waits for a quiet window, with the same fencing in the other direction. Automatic failback is often switched off for that reason, and because a primary that keeps crashing and recovering would otherwise pull the traffic back and forth.

Watch outFailover that is never rehearsed usually fails, at the step nobody tested, such as a fencing command that lacks permission. Test the switch regularly; Chaos engineering & game days covers rehearsing it on purpose.

Write the stopwatch on the Grid

The Failover & Request Hedging Challenge asks for a primary and a standby serving path, health monitoring that decides when the primary is unhealthy, and Failover redirecting traffic to the standby on outage, with Request Hedging beside it for slow requests. Focus on having the health probe detect the primary's failure and promote the standby cleanly, and write the budget on the edges: the probe interval and failure count, the promotion, the redirect. A good run fences the old primary before the standby is promoted, so a partitioned-but-alive primary cannot cause split-brain.

Practice this