Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Cascading failures: how one dependency takes down its callers

How one slow dependency can exhaust the threads and connections its callers share, hop by hop, until features that never call it go down too, and which resilience pattern breaks each link of that chain.

Boarding passes stopped, and nothing was wrong with them

An airline's check-in service does three things for its mobile app: it checks passengers in, changes seats and issues boarding passes. Only check-in asks the loyalty service for the passenger's frequent-flyer tier, to decide whether a bag flies free. One morning a migration leaves the loyalty database without an index it relied on, and tier lookups that took 40 ms start taking 6 seconds. The loyalty service never goes down. It answers every request, correctly, slowly. Ten minutes later the app cannot issue boarding passes, which never touch loyalty, and the check-in service's CPU graph reads 12%.

Engineers call it a cascading failure: a failure in one dependency spreading to the services that call it, then to theirs, until parts of the system with nothing wrong in them are down too. It does not take a dramatic fault, and it does not respect which features depend on what. This article follows the chain from the slow database to the stuck boarding pass, one link at a time, because each of the resilience patterns exists to break one of those links.

Where the waiting piles up

Only check-in calls the loyalty service, but every route shares the gateway's connections to the check-in service and the check-in service's threads.

No answer is also an outcome

Across a network, silence does not tell the caller what happened. A request with no answer yet may never have started at the loyalty service, may still be running there, or may have finished with its reply lost on the way. A distributed system also has partial failure: one part fails while the others keep running, and the caller may be unable to tell a failure from a delay. The check-in service has a single piece of evidence, how long it has been waiting, and on that morning nothing in its code decided when a wait was too long.

Slow can do more damage than down

Had the loyalty service crashed outright, its callers would usually have found out fast. A connection to a port nobody listens on is refused within a network round trip; the check-in service would have logged errors and checked passengers in without a tier. A host that disappears without answering at all is the other case, and it behaves like a slow one, because then the caller waits until something gives up.

Slowness is worse because every call holds something while it waits. Little's law gives the size of the bill: on average, the number of requests in progress equals the arrival rate multiplied by the time each one spends in the system. At 150 check-ins a second and 40 ms per lookup, about 6 lookups are in flight at any moment. At 6 seconds per lookup, keeping up takes 900, and wherever there are fewer places to wait, the excess queues and waits longer still. Nothing failed; the same traffic simply needs 150 times as many places to wait.

The pools empty one hop at a time

Each of those 900 waits has to sit somewhere, and every place it can sit is a shared, limited resource. In order:

  1. The check-in service's HTTP client keeps a pool of 50 connections to the loyalty service. All 50 are busy within the first second, and later lookups queue for a connection.
  2. The check-in service handles each request on one of 200 threads, and a check-in waiting for a lookup keeps its thread. Within a couple of seconds all 200 are check-ins waiting, and seat changes and boarding passes queue behind them, often until the app has stopped waiting for the answer.
  3. The API gateway now sees every route to the check-in service as slow. Its connections to the check-in service fill with waiting requests, and if its routes to other services share the same workers, those routes start to queue as well.
  4. The app gives up after 10 seconds and shows an error, and passengers at the airport tap Retry.

This is why the CPU graph read 12%. A thread blocked on a socket does almost no work, so the service looks idle at the moment it can accept nothing, and an autoscaler that watches CPU sees no reason to act. If it had added instances, each would have opened its own 50 connections and sent more lookups to the slow database, the one component that could not take them. The signals that show this kind of failure are per-dependency latency, requests in flight and pool usage, not CPU; Observability: metrics, logs & alerting covers measuring them.

Retries turn waiting into load

Then the callers react, and their reactions land on the part that is already failing. Suppose each layer retries after its own timeout: the app makes up to 2 attempts on its own, the gateway up to 3 for each call it receives, and the check-in service up to 3 for each lookup. The layers multiply: one tap can become as many as 18 lookups against the loyalty database, before the passenger taps Retry by hand. Giving up does not cancel anything below, either, unless the cancellation is passed down, and nothing here passes it. When the gateway stops waiting, the lookup it abandoned is still running in the database, next to the retry it just sent.

The database was slow because it had more work than it could do, and the retries add more, so it slows further, which produces more timeouts and more retries. That loop can outlive its cause: the index is restored, and the backlog of retried lookups keeps the database saturated anyway, until something sheds that load.

One defence per link

Each resilience pattern breaks one link of that chain, and none breaks all of them, which is why they are used together:

  • The wait has no end: a Timeout turns "no answer yet" into an answer the caller can act on, and caps how long one call holds a thread, provided it also covers the wait for a connection. On its own it limits the damage without containing it: with a 2-second timeout, 150 lookups a second still keep 300 in flight against 50 connections.
  • One dependency drains a shared pool: a Bulkhead caps how many threads the loyalty calls can hold and refuses the excess at once, so they fill only their own compartment and seat changes and boarding passes keep their threads. The 50-connection pool was no such wall, because a lookup queuing for a connection still held a shared thread.
  • Callers keep calling what is failing: a Circuit Breaker notices that most recent lookups failed and refuses the next ones at once, which frees the caller and gives the loyalty service room to recover.

The other links have owners too. Retry keeps retries to one layer with backoff, jitter and a budget, and sets the order in which these protections nest around a call; a retried write also has to be safe to repeat (Idempotency & exactly-once illusions). Fallback chooses what the passenger gets instead, perhaps the standard bag allowance with the fee settled later. Load Shedding and Back-Pressure keep queues from growing past the point where anyone is still waiting.

Trace the chain before you fix it

Diagnose: Payment Flow With No Isolation From the Provider is this chain with a different dependency. A payment service calls an external payment gateway synchronously, on the request path, with nothing in between: no timeout budget, no queue, no fallback. The Challenge then makes the provider slow and flaky. Before adding anything, trace the links on the starter Grid: what each request holds while it waits for the provider, which resource runs out first, and which requests fail that never needed the provider. Then break the chain where the brief asks, by taking the provider call off the request path with a queue and a worker (Message queues & async workers), keeping the payment operations idempotent, and checking the tail-latency and availability SLO while the provider is degraded. For the protections around a single call, in order, Resilient Service Calls is the Challenge to try next.

Practice this