Eighty milliseconds, and almost nobody got it
A job-search site splits its index of postings across 100 shards. Every search goes to all of them, and the page shows results when the last one has answered. For months the dashboard has shown an average shard response time of 80 ms, while support tickets keep describing searches that hang for a second or two.
Group one shard's last 1,000 response times. 800 were answered from memory in about 20 ms. 185 needed a disk read and took about 200 ms. 15 took about 1.8 s, stuck behind a garbage-collection pause. Add them up, 800 × 20 + 185 × 200 + 15 × 1,800 = 16,000 + 37,000 + 27,000 = 80,000 ms, divide by 1,000, and there is the dashboard's 80 ms: a time that not one of the thousand requests took.
Sort them, then read off a rank
Sort the same thousand from fastest to slowest. The 500th value is 20 ms: half the requests took that long or less. That is the median, or p50. The 900th is 200 ms, the p90. The 990th is about 1.8 s, the p99. A percentile is the latency that a given share of requests comes in at or under, so p99 is the time at least 99% of requests beat or match, and the slowest 1% take at least that long. The high end of the list, p99 and beyond, is the tail latency those tickets describe.
The sorted list also shows what the average hid. This shard serves two kinds of request, from memory and from disk, and the mean of 80 ms falls in the gap between them, where no request is. It also mixes a few very slow requests into many fast ones without saying which is which: the 15 slowest, 1.5% of the requests, supply a third of the average (27,000 of the 80,000 ms), and the dashboard cannot tell that apart from every request taking 80 ms.
One shard's slowest 1% is most searches
A slow 1% sounds harmless until a request waits for many calls. Suppose each shard lands in its own slowest 1% with probability 0.01, independently of the others. A search avoids that on all 100 shards with probability 0.99¹⁰⁰, about 0.366. In the other 63% or so, at least one shard is in its slowest 1%, and the search waits for it:
- 10 calls: 1 − 0.99¹⁰, about 9.6%.
- 40 calls: 1 − 0.99⁴⁰, about 33%.
- 100 calls: 1 − 0.99¹⁰⁰, about 63%.
So more than half of all searches take at least as long as one shard's p99: the shard's tail has become the search's median. This shard does worse than the formula assumed, because its slow group is 1.5%, not 1%: 1 − 0.985¹⁰⁰ is about 78% of searches waiting roughly 1.8 s.
Diagram components: Web App, Application Service, Elasticsearch, Elasticsearch, Elasticsearch, Elasticsearch.
The API merges results when the last shard replies, so this search takes about 1.8 s, whatever the other 99 shards did.
The formula assumes shards are slow independently. At the same per-shard rate, shards that share a cause, a disk or a pause schedule, hit fewer searches, each with many slow shards at once. Either way, the number to improve is the shard's tail, not its average.
Where a slow 1% comes from
Many tails are bad moments rather than bad requests: the same query a second later is often fast. Here it was a GC pause; elsewhere it is a cold cache after a deploy, a retry, a replica on a failing disk, a noisy neighbour, or a queue that builds near capacity, which Servers, not utilization, decide the queue works through. Finding where the time went inside one slow request is a job for Distributed tracing.
Numbers that do not add up
Switching the dashboard to p99 invites a second mistake: averaging each shard's p99 into one line. Percentiles do not average. Take two minutes on one shard, a quiet one with 1,000 requests and a p99 of 50 ms, then a busy one with 9,000 requests and a p99 of 900 ms. The mean of the two p99s is (50 + 900) ÷ 2 = 475 ms. The p99 of both minutes together is the 9,900th of the 10,000 sorted requests. It lands somewhere between 50 and 900 ms, and the two p99s alone do not say where: if every quiet request took 50 ms and the busy minute held 8,909 requests at 10 ms and 91 at 900 ms, it is 50 ms. Averaging across hosts fails the same way.
What does add up is counts. Each host counts its requests into fixed latency buckets, such as under 10 ms, 10 to 50 ms, 50 to 250 ms and so on. Two hosts' counts for the same bucket are simply added, and so are two minutes', so a latency histogram can be summed across every host and window, and the percentile read from the summed counts: adding counts keeps everything the buckets recorded, while a percentile keeps one value and throws away how the requests on either side of it were spread. The answer is only as precise as its bucket, so keep buckets narrow around the values you care about. Prometheus histograms and HdrHistogram work this way, while a Prometheus summary computes its quantiles on each host and cannot be meaningfully combined. Collecting and storing these counts is covered in Observability: metrics, logs & alerting.
Three more traps distort the reading. A p99.9 needs thousands of requests behind it: in a minute of 500 it is simply the slowest one. A timer that starts with the handler never sees time spent queued before it, so measure as close to the client as you can. And a load generator that waits for each response before sending the next stops sending during a stall, so the stall counts as one slow sample instead of every request real users would have sent into it; this is coordinated omission, and it makes the measured tail look better than the real one.
Shortening the tail
- Fewer legs. 20 larger shards instead of 100 means 1 − 0.99²⁰, about 18% of searches meeting a slow shard instead of 63%, provided a bigger shard's tail does not grow with it.
- Not waiting for the straggler. Send a second copy to another replica after a delay such as the shard's p95 and take the first answer (Request Hedging), or, where an incomplete answer is acceptable, merge what arrived by a deadline, the gather timeout of Scatter-Gather, with the number chosen as Timeout explains.
- Fewer slow moments. Cache what the slow path reads, shorten GC pauses, and keep enough headroom that queues stay short.
A latency objective such as "99% of searches under 500 ms over 28 days" is written against a percentile, and with a bucket boundary at 500 ms its share of good requests comes straight from the histogram. The budget it implies, and how to alert on it, belong to SLOs & error budgets.
A tail target with no list behind it
Diagnose: URL Shortener Under Launch Traffic sets a 120 ms tail target for one API instance, every redirect reading the database, and a 10x flash crowd. That tail is a modelled estimate, not a percentile of sampled requests (Servers, not utilization, decide the queue explains how the simulator derives it), so it shows neither a hit-and-miss split nor the fan-out arithmetic above. It does show the queueing source: the node it flags passes 100% utilization and the tail reads unbounded, and the tail comes back down as scaling and a cache on the redirect path move each node away from capacity. What it leaves out is the part that decides a real p99 for this service: it uses the cache's hit rate only to take load off the database, so the redirects that miss never become slow requests in its tail, however long each miss takes.