Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Circuit Breaker

Stop calling a dependency that recent calls show is failing: how the breaker trips on a failure rate, fails fast while open, and lets a few trial calls decide when to close again.

A retry policy has no memory

An order service checks every order with a fraud-scoring service before accepting it, 30 orders a second at the evening peak. One evening the fraud service's database runs out of disk, and every call to it starts timing out after 1 second. The order service retries with backoff and jitter, up to three attempts, as Retry describes, but without the retry budget that article also recommends. So each order now waits more than 3 seconds to learn that scoring failed, and the order service keeps sending up to 90 calls a second to a service that cannot answer any of them, while every waiting order holds a thread that nothing else can use.

Each of those calls is reasonable on its own. What is missing is memory. Five minutes in, the 9,000th order to try is treated as a fresh question, although the 8,999 before it all got the same answer. A Circuit Breaker gives the caller that memory. It wraps the calls to one dependency, watches how they turn out, and when enough of them fail it stops making the call for a while and fails at once instead, protecting both the caller's threads and the struggling dependency.

Breaker in front of a dependency

Three states, one evening

The breaker is a small state machine. Its names come from electrical circuits, where a closed circuit lets current through and an open one does not. Here is what it does from the moment the disk fills:

  1. Closed is the normal state. Every call goes through, and the breaker records whether each one succeeded.
  2. The breaker keeps a sliding window of recent calls, say the last 50. Once at least half of them have failed, it trips to open. With every retry passing through the breaker too, at 90 calls a second, that happens within a second or two.
  3. While open, the breaker refuses every call at once for a cooldown, say 30 seconds, without touching the fraud service. The order service gets its answer in microseconds instead of 3 seconds.
  4. When the cooldown ends, the breaker goes half-open: it lets a few trial calls through, say five, and keeps refusing the rest.
  5. If the trial calls succeed, the breaker closes and traffic returns. If they fail, it opens again for another cooldown. In the simplest version a single failed trial is enough to reopen it.
Closed, open, half-open

There is no arrow from open straight back to closed. Only trial calls that succeed close the circuit.

Deciding what counts as a failure

The trip is only as good as the record behind it. A timeout must count, so pair the breaker with a Timeout on the call and count a timeout as a failure toward the trip threshold: a dependency that hangs instead of returning errors would otherwise never trip it. Errors that describe the request rather than the dependency should not count. A 404 for an order the fraud service has never seen, or a 400 for malformed input, says nothing about the service's health, and counting them can open the breaker in front of a perfectly healthy dependency. Some breakers can also count slow successes as failures, which catches a dependency that is degrading before it starts returning errors.

The rate also needs enough calls behind it. At 4 a.m., two failures out of three calls is a 67% failure rate and proves nothing, so breakers usually wait for a minimum number of calls in the window before they will trip.

A refusal is final

Speed is what the open state buys: an order that would have spent 3 seconds discovering the fraud service is down now finds out in microseconds, and its thread is free again. While the circuit is open, the breaker answers the caller at once with a fallback (a cached or default response) or an error, without calling the dependency. What an unscored order should get is a business question, and Fallback walks through how to answer it.

Retries need one rule here. Pair the breaker with retries with backoff, but do not retry calls the open breaker rejected. A refusal means the breaker already knows; retrying it either fails again at once, spending the caller's time on nothing, or lands just as the breaker half-opens and piles onto a dependency that is trying to recover. A retry loop wrapped around the breaker therefore has to stop at the first refusal; where each protection sits in the chain is laid out in Retry.

Where breakers go wrong: the way back

Later that evening the disk is cleared. The fraud service is up again, but its caches are cold, and for a few minutes it can handle perhaps a tenth of its normal load. A breaker that half-opens by letting all traffic through, instead of a few trial calls, sends it the full 30 orders a second, ten times what it can handle, plus their retries. That floods it straight back into failure, and the breaker reopens. The circuit flaps, open and half-open in turn every 30 seconds, and the service never gets the quiet moment it needs to warm up. A handful of trial calls tests whether the dependency is back without testing its limits. That also means passing trials do not show it can take the full load: requiring several successful probes before closing guards against closing on one lucky call, and letting traffic back in gradually, as some implementations do, helps a service that is still warming up.

Scope matters as well. A breaker guarding the whole fraud service blocks its working endpoints when one of them breaks, so give each breaker the scope of what fails together. Each instance of the order service usually keeps its own breaker, fed by its own calls, and a caller that makes one call a minute may never gather enough to trip. And a breaker reacts only to failures it has already seen: before it trips, or against a dependency that turns slow without failing, a Bulkhead is what keeps the slowness from taking every thread.

Draw the third state

A breaker drawn with two states has no honest way back: it either stays open forever or closes on a guess. In the Circuit Breaker challenge, give closed, open and half-open a State node each, write your numbers on the transitions (failure rate and window, cooldown, trial calls), and add the caller's fallback for the open period. Then trace one failed trial call. If it lands anywhere but open, or if any call reaches the fraud service while the circuit is open, the drawing has a bug.

Practice this