Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Retry

Make a failed call again, automatically: which failures are worth another attempt, how long to wait between attempts (exponential backoff with jitter), how many to allow, and how retries at several layers multiply into a retry storm.

Many failures go away if you ask again

A warehouse service prints shipping labels by calling a carrier's API, about 40,000 times a day. Roughly one call in 200 fails: a connection reset, a 503 Service Unavailable from a carrier server that is restarting, a request dropped by a load balancer on the way. Sent again a moment later, nearly every one of those calls succeeds. That is a transient failure: one caused by a passing condition, not by anything wrong with the request.

The Retry pattern makes that second attempt automatically, so most of the 200 or so calls a day that hit one of those failures still end in a label instead of an error. In its first version it is a loop: make the call, and if it fails, make it again, up to a limit. Each version of that loop below is one that someone shipped, and each was replaced for a reason.

A retry between the caller and the carrier

Version one: again, at once

The first loop retried immediately, up to three times. For months it looked perfect, because a dropped connection is a one-off. Then the carrier had a ten-minute outage. Every failed call became four calls within a few milliseconds, so a carrier that was already struggling received up to four times its usual traffic from this warehouse, and the same from every other customer whose client did the same thing. An immediate retry assumes the failure is already over. When the failure is overload, the retry is more of it.

Version two: only the failures worth repeating

The first loop also retried calls that could never succeed. The fix was to sort failures by what they say about the next attempt:

  • Worth retrying: a refused or reset connection, 503 Service Unavailable, 429 Too Many Requests, and a timeout. Each describes the carrier's condition at that moment, and the condition may pass.
  • Not worth retrying: 400 Bad Request for a postcode the carrier rejects, 401 for a revoked credential, 404, 422. The same request will get the same answer, and four copies of it only fill the logs.

A 429 or a 503 may carry a Retry-After header, the server's own estimate of when to come back, as a number of seconds or a date. When it is there, wait at least that long; how a server decides to send one is in Throttling.

Creating a label is a write, and a write changes what a retry means. After a timeout, or a connection lost once the request was sent, the outcome is unknown: the carrier may have created the label and only the reply was lost, so a blind retry can create a second label and pay for postage twice. Giving up cannot tell you what happened, as the payment story in Timeout shows. A retried write is safe when the server can recognise the repeat, which is what an Idempotency Key gives it, provided every attempt carries the same key. Without one, retry a write only on failures that prove the request never arrived, such as a refused connection.

Version three: wait longer each time

Between attempts the loop now waits, and the wait doubles: 200 ms before the second attempt, 400 ms before the third, 800 ms before the fourth. That is exponential backoff. A blip is usually over by the first retry; a failure that lasts gets fewer and fewer calls instead of the same pressure every few milliseconds. Doubling grows quickly, so implementations also cap the wait at a maximum, say 10 seconds, and the attempts and waits together have to fit inside the caller's own deadline; that sum is worked through in Timeout.

Version four: stop retrying in step

Backoff still left a pattern in the carrier's graphs: spikes. When the carrier wobbled for a few seconds, thousands of clients failed in the same second, waited exactly 200 ms, retried together and failed together, waited exactly 400 ms, and arrived together again. Identical schedules turn a crowd of independent clients into one synchronised wave.

Jitter breaks the step by randomising each wait. A common form, often called full jitter, picks a random wait between zero and the backoff value: anywhere up to 200 ms, then up to 400 ms, then up to 800 ms. The retries spread across each window instead of landing together at its end, and the carrier sees a smooth trickle where it saw a wall.

Counting attempts across the whole path

The label service makes up to four attempts. It is not the only layer that retries. The packer's handheld app calls a gateway, the gateway calls the label service, and each was written by someone who added a retry. If the app and the gateway also make up to four attempts each, one tap on Print can become 4 × 4 × 4 = 64 calls to the carrier while it is down. Retries at different layers multiply, and the outer layers keep retrying after the person holding the scanner has given up. That multiplied load, arriving just as a dependency is weakest, is a retry storm.

Two rules keep it in check. Retry at one layer, and let the others pass the failure up. And give retries a retry budget: a cap on retries as a share of ordinary traffic, for example at most one retry for every ten original requests over the last minute. During a blip the budget is barely touched. During an outage it runs out within seconds, and the extra load retries can add is held near 10% instead of the 300% that four attempts per failing call allow. A budget slows the retries against a dependency that is down; it does not stop the calls, and why a call the breaker refused must not be retried is a lesson from Circuit Breaker.

Where the loop sits in the chain

Around a flaky dependency the four protections usually nest in this order. The Fallback is outermost, so it can answer whatever went wrong inside. The retry wraps the circuit breaker, so the breaker counts every attempt and can stop the loop by refusing. And a timeout bounds each attempt inside the retry, next to the dependency, so no single attempt can hang. That nesting is what Resilient Service Calls grades; put this article's policy on its Retry node, the failures it retries, the attempt cap, backoff with jitter, and check that the loop stops once the breaker opens. Version one at full scale is waiting in Diagnose: Webhook Delivery Retry Storm: a delivery service that re-sends failures immediately, with no backoff, for you to find the amplification and redesign it.

Practice this