The cache stampede hides behind your hit rate

Aug 23, 2026

A 99% cache hit rate means the database behind it is sized for 1% of your traffic. Here is what happens the moment that assumption stops holding, and how to see it on a diagram before it happens.

A cache with a 99% hit rate is usually reported as a success. It is also a statement about the database behind it: that database is sized for 1% of your traffic. Nobody wrote that down, and it is the most load-bearing number in the design.

The arithmetic

Take 10,000 requests per second and a 99% hit rate. The origin sees 100 requests per second. It is comfortable there — plenty of headroom, low latency, nothing to page anyone about.

Now the cache stops answering. Not slowly: the hit rate goes to zero and the origin sees 10,000 requests per second. That is a 100x step change, arriving in one tick, at a database provisioned for the small number.

What follows is not a gentle degradation. The origin saturates, latency climbs, callers time out and retry, and the retries add load to a system that is already past capacity. The cache being empty is the trigger; the retry loop is what keeps it down after the trigger is gone.

Why it happens all at once

The failure mode that surprises people is the synchronised one. If a fleet of servers warms its cache at the same moment — a deploy, a restart, a scheduled refresh — then every entry gets the same TTL and every entry expires in the same second. The design has an invisible alarm clock in it, and it is set for all replicas at once.

A deploy rolls the fleet and every instance starts with an empty cache.
A cache node restart drops its share of the keyspace in one go.
A uniform TTL written at the same time expires at the same time.
A hot key — one entry serving a large share of traffic — turns a single expiry into a fleet-wide miss.

The three usual answers

Single-flight (request coalescing) — on a miss, one caller recomputes and the rest wait for that result instead of each hitting the origin. This is the one that directly bounds the origin's load, and it is the one most often missing.
TTL jitter — spread expiry over a window instead of a single instant, so the fleet's misses arrive spread out rather than together.
Stale-while-revalidate — serve the expired value while one caller refreshes it in the background. Callers see slightly old data instead of a queue.

They compose, and they solve different halves. Jitter spreads the misses out; single-flight bounds what a burst of misses can do to the origin. A design with jitter alone still has a hot key that can take the origin down on its own.

Seeing it before it happens

The reason this is hard to reason about on a whiteboard is that the interesting number — what the origin receives — is a product of two others that live in different boxes. Change the hit rate and the load on a component three hops away moves by two orders of magnitude.

On a RodGrid canvas the components are typed, so the simulator knows which box is a cache and which one is behind it. The Cache stampede stress preset sets the cache hit factor to zero and re-runs the model, which makes the origin's new arrival rate a number on the diagram instead of an argument in a meeting.

Note This is a steady-state model: it answers what the system looks like while the cache is cold, not how tall the instantaneous spike is at the moment of expiry. The transient peak is higher than the steady state it settles into. Use the result as a floor on the damage, not a ceiling.

Tip Draw the cache you rely on most, run the stampede preset, and look at what the component behind it is being asked to absorb. If that number is a surprise, the hit rate was doing more work than the design admitted.