Retry storms, or the fix that becomes the failure

Aug 22, 2026

Retrying is the standard answer to a transient error. When the error is saturation rather than bad luck, the retry adds load at the exact moment the system needs less of it.

Retry is the first thing anyone adds to a client, and usually the right thing. A packet was dropped, a node was replaced mid-request, a connection was reset — trying again costs almost nothing and turns a visible error into a hiccup nobody notices.

That reasoning holds for one failure mode only: the transient one, where the second attempt is likely to land on a healthy system. When the error is saturation instead, the second attempt lands on the same overloaded component — and it arrives as extra work.

The multiplication nobody planned

A client configured for three attempts turns one request into up to three. When a dependency starts failing under load, that is not a 3% increase in traffic — it is 3x, applied precisely while the dependency is least able to absorb it.

It gets worse with depth. Retries at more than one layer multiply rather than add: a gateway that retries three times in front of a service that also retries three times can put nine attempts on the database behind them. Nobody wrote 9 anywhere. It is the product of two independent, individually reasonable settings.

Retry at one layer, usually the outermost one that can still be useful.
A retry budget — cap retries at a small fraction of total requests (a token bucket works) so the multiplier has a ceiling no matter how bad things get.
A circuit breaker so a dependency that is failing consistently stops being called at all, instead of being asked again by everyone.

Backoff is necessary and not sufficient

Exponential backoff spreads attempts over time, which is real relief. But it does not reduce the total number of attempts — it delays them. If the dependency stays down for the length of the backoff window, the same volume arrives slightly later, and it arrives synchronised: every client that failed at the same moment waits the same interval and returns together.

Tip Jitter is what breaks the synchronisation — a random spread on each wait, so the retries of a thousand clients do not land in the same second. Backoff without jitter converts one spike into a series of spikes.

The part that outlives the outage

The reason a retry storm is worth naming separately is that it does not end when the original problem does. The dependency recovers, finds the accumulated retries of everyone who failed while it was down, saturates again, and produces the next round. The trigger is gone and the system is still on the floor.

This is the shape to look for in a postmortem: an incident whose duration has nothing to do with the duration of its cause.

Watching it on a diagram

On a RodGrid canvas the Retry storm stress preset doubles the traffic and adds error rate to the service nodes at the same time — the two halves that make the failure, applied together. What you are looking for is not whether a component turns red, but which one turns red first, and whether the components behind it were sized for the multiplied number or the original one.

Note The model is steady state: it shows the system under sustained multiplied load, not the oscillation of successive retry waves. Real storms are spikier than this, so treat the result as the calm version of a worse event.