Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Backpressure & load shedding

Why a system a little over capacity can end up serving almost nobody, and how it stays useful: a bound on every buffer, producers slowed where they can wait, work refused at the edge where they cannot, deadlines that travel with the request and retries kept on a budget.

5,400 in, 4,000 possible, almost none done

Two lines on one graph. Uploads arriving: 5,400 a second from midnight, 35% more than the servers could finish. Uploads completed: close to zero for twenty-five minutes, though the surge lasted ten. "Why did almost every upload fail when we were only a third over?" was the first question in the review. The app was photo sharing, on its first New Year in a new region. Its stateless servers had been scaled ahead of time, but the metadata database that records every photo could not be scaled in minutes, and it held the upload servers to about 4,000 uploads a second.

A service in that position does not run 35% slower. Each upload server handed requests to a worker pool whose queue had no limit, so the surplus, 1,400 uploads a second, waited in line. About thirty seconds in, the line held some 40,000 uploads, ten seconds of work, longer than the app waits before giving up and trying again, up to three attempts in all. Ten seconds later the workers were finishing uploads whose phones had already given up and sent a fresh attempt to the back of the line. Arrivals climbed towards three times the real demand, and the servers spent their whole capacity on answers nobody was waiting for.

So overload is a state rather than a spike. Past saturation, the goodput of an unprotected system, the work finished while someone still wants it, drops towards zero, and retries hold it there: at ten past midnight real demand fell back below capacity, yet the backlog and its retries kept the queue full for another quarter of an hour. A failure that keeps itself going after its trigger has gone is called metastable.

Nothing about this crowd was abusive: every phone stayed within any per-user limit, so the per-client limits of Rate limiting, built against one client taking more than its share, had nothing to refuse. Against legitimate demand you cannot serve there are two moves: slow whoever produces the work, which is backpressure, or refuse some of it on purpose, which is load shedding. Here they are placed and combined; Back-Pressure and Load Shedding take each mechanism apart.

Every unbounded buffer is a promise

The worker pool's queue was not the only hiding place: the load balancer's connection backlog, the wait for a database connection and the database's lock queues all had room. Every unbounded buffer is a promise you cannot keep: it turns "we are overloaded" into "we are overloaded and every request now waits 40 seconds". A bound forces the decision to the front, where refusing is cheap and the caller learns the truth now rather than after a timeout.

Size each bound in time, not in slots: at 4,000 uploads a second, a queue of 2,000 is half a second of waiting, and one of 100,000 is 25 seconds that store nothing but failures. Why waiting climbs so steeply near 100% busy is explained in Servers, not utilization, decide the queue.

Wait inside, refuse at the door

What happens when a bounded buffer fills depends on who is on the other side. Inside the system the producer is code you own, and code can be made to wait:

  • Upload servers to the database. Cap the queries in flight at what the database can hold, say 64 in all, split between the upload servers, with a short queue in front of the cap, and fail a request at once if it cannot get a slot within 200 ms. Between your own tiers a concurrency limit fits better than a rate limit, because when queries slow down it stops more of them piling up in progress; a fixed number has to be sized, and adaptive limiters (AIMD, gradient) adjust it from signals such as latency.
  • Upload servers to the thumbnail workers. Nobody watches thumbnails being made. The upload server puts a job on a bounded queue and the workers pull the next one when they have room, as Kafka consumers do. When the queue is full, the enqueue waits up to a short limit or fails, and that tells the upload server to slow down or refuse new work. Back-Pressure follows that signal across each kind of hop, TCP and reactive streams included, and the queue itself is Message queues & async workers.
  • Work that can wait for hours. The "your year in photos" collage can be built tomorrow, from a durable queue drained at the workers' pace: Queue-Based Load Leveling, also a Challenge of that name. It helps only while the workers catch up once the peak passes; a queue that keeps growing and never drains is an unbounded buffer with a disk behind it.

At the edge the producer is a person holding a phone, and a phone will not wait for long: hold its upload and it times out and sends it again. So the system sheds, refusing a share of requests immediately so that the rest finish within their deadline. Refusing pays only if it is cheap: an upload refused after authentication, a 2 MB body read and a database call cost almost as much as one served; refused at the load balancer, from a header and a counter, it costs next to nothing. Refusing at random would drop a quarter of the uploads people are watching, so the edge ranks the work: feed prefetches go first, uploads last, health checks never, though with uploads alone above what the database takes, some uploads are still refused. The signals that say a server is full, the order in which classes go and what the refusal should tell the phone are all in Load Shedding.

The two halves meet at the bound. When the 64 slots and their short queue are full, an upload server fails fast, and the edge, with a cap on uploads in flight per server sized from that server's share of the 64 slots, stops admitting before that point: pressure inside becomes a refusal at the door, and an upload that gets in finishes in time as long as what is left of its processing fits in its deadline.

A bound on every hop

The refusals are the design working: the edge turns the surplus away, upload servers wait for one of 64 database slots, and thumbnail workers take a job only when they have room.

For a crowd of people arriving at a known minute for something scarce, a line in front of the site usually beats refusals they would only retry: Virtual waiting rooms: admitting a crowd fairly.

Give the request its deadline

An upload that reaches the database 9.5 seconds after the phone sent it has half a second left: unless the database and the reply fit in it, the phone gives up and the work is wasted. If the request carries its deadline, every hop can see that and drop work nobody will wait for, which matters most in overload, when the queues are full of such requests; how the remaining time is passed down a chain is in Timeout. A deadline also exposes retries that pretend to be short. The desktop uploader, which syncs whole folders, tries three times with a 30-second timeout each: that is a 90-second request in disguise, not counting any pause between tries, and every server it reaches works as though it had 30 seconds. Give the whole operation one deadline and let every hop honour it.

Retries are load

The midnight loop ran on retries, which arrive exactly when capacity is shortest. Retry at one layer only, since attempts at several layers multiply, and give retries a retry budget, say at most one retry for every ten original requests over the same minute, so an outage adds at most about 10% more attempts instead of the 200% three attempts allow; backoff, jitter and the rest are in Retry. A Circuit Breaker in each caller is the other half of the defence: it protects the caller from a sick dependency, while shedding protects a server from too many healthy callers, and neither replaces the other. How the loop spreads from a service to its callers is Cascading failures: how one dependency takes down its callers, and Diagnose: Webhook Delivery Retry Storm puts it inside a running design.

The midnight graph, run again

A design that answers the Backpressure & Load Shedding Challenge should replay midnight with different lines. Push the producer past the consumer's pace, and the bounded queue fills to its stated size and stops there, while the producer that can wait slows to match. Push the entry past capacity, and the lowest-priority requests are refused at once, at almost no cost, while the high-priority ones, as long as they alone fit, still finish inside their deadline. Those two runs answer its last requirement, when to slow down and when to drop: slow down what you own, refuse what you cannot hold. The Challenge judges what each pattern node does rather than replaying traffic, so trace both runs yourself before you submit.

Practice this