Every server busy, almost nobody served
The graphs from that Friday evening look as if they contradict each other. A food-delivery app's API was receiving more requests than at any other time that week, every instance's CPU sat at 100%, and completed orders fell close to zero. Between them the instances could finish about 4,000 requests a second. A television advert brought 6,000. Requests queued, each one waited longer than the one before, and the app, which gives up on a request after 3 seconds, abandoned most of them while they were still waiting. The servers kept working flat out on answers nobody was there to receive, and the app's retries added more requests on top.
What fell was not throughput, the requests finished per second. It was goodput: the requests finished while someone was still waiting for the answer. Backpressure & load shedding describes that collapse in general; this article is about the defence that sits inside the server. Load Shedding means that once the server is at its limit, it refuses the excess at once, fast and cheap, before that work consumes real resources, so the requests it does accept still finish in time. Turning away 2,000 requests a second sounds like failing. The alternative was failing nearly all 6,000.
The other 2,000 requests a second get an immediate refusal from the shedder, decided before any authentication or database work.
A shedder makes three decisions on every request: whether the server is full, which requests go if it is, and how to say no. Each one can quietly undo the other two.
Knowing the server is full
CPU is a weak signal. It reads 100% on a server that is busy and keeping up as well as on one drowning in queued work, and it says nothing about how long requests are waiting. Two signals sit closer to what the app experiences. One is concurrency, the number of requests admitted and not yet answered: each API instance did its best work with about 200 requests in flight, so its shedder admits up to 200 and refuses the next one at once. The other is queueing time, how long admitted requests wait before a worker picks them up: once the oldest request in the queue has waited past a set share of the app's 3 seconds, say 500 ms, new arrivals are refused, leaving the rest of the budget for the work itself. A fixed limit like 200 has to be re-measured whenever the code changes, which is why some servers adjust it continuously from the latency they observe.
Choosing what to drop
Refusing 2,000 requests a second at random would refuse a third of all orders along with a third of everything else. Requests are not equally valuable, so the shedder sorts them into classes and drops from the bottom up. Drop prefetches and background traffic first, checkouts last. For the delivery app:
- First to go: prefetches of restaurant photos for screens the user has not opened, recommendation refreshes and analytics events, which the app can send later or not at all.
- Next: browsing restaurants and menus. A customer who is refused will try again in a moment.
- Last: placing and paying for an order, and tracking updates for orders already placed.
- Not shed at all: the load balancer's health checks. An instance that keeps failing its health check is taken out of rotation, its traffic moves to the other instances, which are just as overloaded, and shedding turns into a cascade.
Read the class from something cheap, the path or a header the server can trust, because deciding must cost less than serving. And whatever its class, a request that arrives with its caller's deadline already passed, when requests carry one as in Timeout, is dropped before any work starts.
A refusal that costs less than an answer
If the API authenticates the user, parses the body and reads the database before deciding it is too busy, each refusal costs nearly as much as an answer, and the server is still overloaded, now by refusals. Shed at the first point that can decide: the load balancer or gateway in front, or the first lines of the request handler, before authentication and before any database work.
The usual refusal is 503 Service Unavailable: the server cannot take the request right now. 429 Too Many Requests means this particular client has sent too many, the answer of a Rate Limiter, which caps each client by policy, whereas shedding drops work because the server itself is at its limit, whoever sent it; some services send 429 when shedding all the same. Refused clients come back, and whether they return as a smooth trickle or as a second wave depends on how they wait, which is Retry's side of the story.
Why not queue the surplus?
Queueing the extra requests feels kinder than refusing them, and for work that is still useful later it is: a receipt email can wait a minute, and absorbing bursts in a durable queue is Queue-Based Load Leveling. For a person waiting on a screen it usually is not. With 6,000 arriving and 4,000 served each second, the queue grows by 2,000 requests every second, and a new arrival's wait grows by half a second every second. Six seconds into the surge, a request that joins the queue will wait longer than the app's 3-second limit; three seconds later the server reaches those requests, and from then on it spends its whole capacity on requests whose callers have left. The queue saved none of them. It only postponed the moment they failed, and made the requests behind them fail too.
So keep the queue short and measure it in time rather than slots: a request that has already waited longer than it can be useful is dropped, not served.
Drop on purpose, on a Grid
Slowing down and dropping share one Challenge, Backpressure & Load Shedding, and its last requirement, when to do which, has a short answer: a producer that can wait is slowed, which is Back-Pressure; a request someone is waiting on right now is admitted while it can still finish in time, or refused at once. For the shedding half, put the Load Shedding node in front of the Service, ahead of anything expensive, and write on it the signal it watches and the order in which classes go. The refused requests should leave from that node with a cheap 503, never from the Service after it has done the work.