Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Fallback

Decide what to answer when a call fails: a cached value, a default, a reduced feature, another source or a clear error, and avoid the fallbacks that hide an outage or knock over something else.

Five boxes on one page

A product page in an online shop is assembled from five calls: the description, the price, the stock level, a delivery estimate and the reviews. One afternoon the reviews service stops answering. If the page treats any failed call as a failed page, a missing star rating takes the price, the photos and the Add to basket button down with it, and the shop sells nothing for an hour because of the least important box on the screen.

A Fallback is the answer a caller gives instead when a call fails, and it fires on any failure of the dependency: an error, a timeout, or a call rejected by an open Circuit Breaker. So the degraded experience is something to design deliberately, which in practice means deciding, for each call and before anything breaks, what the page shows when it does.

Degrade gracefully

One decision per box

There are four kinds of substitute answer, and this page uses all of them:

  • A cached copy. The reviews box shows the star rating and review count from the last successful call, kept for up to a day. Ratings move slowly, and a 4.6 from this morning is nearly as useful as a 4.6 from now.
  • A default. When the delivery-estimate service fails, the page shows the shop's standard promise, "Usually ships in 2–4 days", instead of a date it cannot compute.
  • A reduced feature. Without the stock service the page cannot say "3 left", so it drops the count and leaves checkout to confirm availability.
  • Another source. The description normally comes from the catalogue service; when that fails, it comes from a copy published to file storage every night.

The price gets no substitute. A cached price can be wrong right after a change, and an order taken at the wrong price becomes a refund or a dispute, which is worse than an error. When pricing fails, the page says the item cannot be bought right now. Some answers are worse than no answer, and for those the right fallback is a clear error. Anything that grants access belongs here too: falling back to allow when the permissions service is down turns an outage into a security hole.

Each decision weighs the same thing: how much harm a stale or approximate answer does here, compared with no answer at all. And whoever reads the page should be able to tell which they got. A rating marked "as of this morning", or a stock line that simply is not there, reads as a page with less on it, not as a broken one. Bolted on as an afterthought, a fallback tends to hand back stale or wrong-looking data with no sign that anything is off, and then neither the customer nor support can tell a degraded page from a broken one.

When the fallback is the problem

It hides the outage. The reviews fallback works so smoothly that nobody notices the reviews service has been down for hours, until the cached ratings expire one by one and review boxes start coming up empty. A fallback like this one is a failure the user did not see, and it is still a failure. Count the fallbacks served for each call, and alert on that rate as you would on errors.

It moves the load somewhere else. The stock box used to fall back from its cache to querying the inventory database directly. During a sale the cache cluster failed, every product page fell back at once, and the database, sized for the cache's occasional misses, received the whole site's traffic, went down, and took checkout with it. A fallback that calls something else runs exactly when the primary has failed for everyone, so check it against that load, not against a single request. The safest fallbacks do no remote work at all: a value already in memory, a default, a box left out.

A fallback runs only when something has broken, which makes it some of the least exercised code on the page. Break the dependency on purpose in a test environment and watch what the page does, or the first real outage will also be the fallback's first run.

The last link in the chain

In Resilient Service Calls the Fallback is one of four protections around a flaky dependency; the order they nest in is set out in Retry. Its own job there is the one this page has been doing box by box: decide what the client gets once everything inside has given up, whether the retries ran out, a timeout fired or the breaker refused the call. Write that answer on the Fallback node, a cached value, a default or a clear error, and make it look different from a normal success.

Practice this