Same call, two numbers
A weather app reads forecast documents from a store with three replicas. Measured over one morning, the median read took 9 ms and the p99 took 420 ms. Repeat one of the slow reads a moment later and it usually comes back in about 9 ms.
Each page view makes four of these reads, for the hourly forecast, the ten-day outlook, the alerts and the radar frames, and the page is not finished until the slowest returns. If slow reads strike independently, 1 − 0.99⁴ of page views, about 4%, wait for at least one read of 420 ms or worse. This is tail latency, the slowest few percent of responses, and Latency percentiles & tail latency explains why it gets worse the more calls a page makes.
The slow reads were not harder than the fast ones. The team traced them to the replica's moment: a garbage-collection pause, a cold cache, a noisy neighbour on the same machine. While replica A is stuck, B and C are usually fine, and they hold the same document. So why keep waiting for A?
A second copy, while the first is still out
Request Hedging waits a short threshold (say, the p95 latency), then sends the same request to a second replica and uses whichever response returns first. The p95 here is 25 ms, so a read that lands on a paused replica goes like this:
- The Hedger sends the read for Bergen's hourly forecast to replica A and starts a 25 ms timer.
- The timer expires with no answer from A.
- The Hedger sends the same read to replica B and leaves A's copy running.
- Replica B answers 9 ms later.
- The Hedger returns that answer to the caller at 34 ms and cancels A's copy.
About 95% of reads answer before the timer expires and never reach step 2. The second copy has to go to a different replica, because a copy sent to A would wait behind the same pause.
Replica A still holds the first copy when the Hedger sends the second to replica B. The first good answer goes back to the caller, and the other copy is cancelled.
One number, and why it is a high percentile
A high percentile is what makes this cheap. Roughly 5% of reads outlast the p95, so roughly 5% send a second copy: 105 requests for every 100 reads. Set the threshold at the median and half the reads hedge, 150 requests per 100. Set it at the p99 and only 1 read in 100 hedges, but no read gets any help until it has already taken 420 ms.
What the 5% buys is easiest to see at 50 ms. If A's and B's slow moments are unrelated, and the 5% extra load leaves their latencies about where they were, a hedged read takes longer than 50 ms only when A has not answered by then, under 5% of the time, and B has not answered within its own 25 ms, 5% of the time. That is under 0.05 × 0.05, or 0.25% of reads, so the p99 falls from 420 ms to under 50 ms. The word to watch is "unrelated"; a later section comes back to it.
This is the setting described in The Tail at Scale (Dean & Barroso, 2013): hold the second request until the first has been out longer than the 95th-percentile latency expected for that kind of request, which limits the extra load to about 5%. The alerts document and the ten-day outlook have different latencies, so each needs its own threshold. And the threshold is not a Timeout. Nothing is given up when it passes, and the caller's deadline still bounds the whole call.
Racing is not retrying
A Retry starts a new attempt after the caller has seen a failure or a timeout, even if the first attempt is still running on the server; a hedge sends a copy because the first is slow, before anything has failed, and keeps waiting for both. That makes it a race, and "first answer" has to mean first good answer. A replica returning errors in 2 ms would otherwise beat every healthy one, so an error does not win the race: if the other copy is still out, the Hedger waits for it until the deadline.
Then cancel the loser, since its work no longer has a reader. Cancelling is best effort. A paused replica sees the cancellation only when it wakes, and a finished read has already been paid for, so plan for the full 5% and treat what cancelling saves as a bonus. Skip it, and a replica already behind goes on working through requests nobody is waiting for.
When every replica is slow
On the hour, a new forecast run is published and the replicas' caches go cold together. Reads now take 60 ms on all three. Hedging fails here in two ways at once.
It stops helping, because the second copy goes to a replica just as slow as the first. And it adds load at the worst time: nearly every request crosses the old p95 threshold, so even one hedge each pushes load toward 2x exactly when you can least afford it, and the extra work can slow the replicas further. Slowness that belongs to the request is shared in the same way: an outsized document that takes 300 ms to read everywhere gains nothing from a second copy.
Capping hedges (one per request is plenty) only sets that 2x ceiling. To prevent the storm, also give hedges a global budget, for example at most ~5% extra requests. The Hedger counts reads and hedges over the last ten seconds, and once hedges pass 5% of reads, a slow read just waits for its first copy. Also stop hedging when the backends are overloaded, because a copy adds work a replica has no room for, the problem Load Shedding handles from the replica's side.
A dead replica is an outage, not a tail. Hedging can rescue a read that landed on it only while the budget allows a copy; any other read depends on the first attempt, which may end in an error or a timeout and at worst waits up to the caller's deadline. Taking it out of service is the work of Failover. Scatter-Gather also sends one question to several places, but at once, and chooses among the answers or merges them; a hedge's copies are interchangeable and the second is sent only when the first is slow.
Hedging beside a failover
The Failover & Request Hedging Challenge pairs the hedge with a failover, in up to eight nodes. Draw the hedge from a Hedger to a second Service that is also serving, a hot standby that takes reads or an active peer, because a replica that cannot take this read now cannot answer a copy. Write the rules on its edges: reads only, one hedge, the threshold, cancel the loser, a small global budget. A reader should be able to tell which path answers a slow read and which answers a dead primary.
One percent of page views first
Only your traffic can say how related your replicas' slow moments are. Turn it on for 1% of page views, leave it on across several hourly publishes, and compare that group with the rest:
- The p50 and p99 of the whole page, measured at the client. The p99 should fall and the p50 should barely move.
- Extra requests as a share of reads, which should sit near 5%, and how often the budget refused a hedge.
If the p99 fell and the extra requests stayed near 5%, widen the share. If hedges fired and the p99 did not move, check whether the budget refused the copies or another component sets the page's p99. If neither does, your tail is probably shared across replicas, and no threshold will fix that.