Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Queue-Based Load Leveling

Put a queue between bursty producers and a steady consumer, so a spike becomes a backlog drained over time instead of an overload: latency traded for stability.

Monday, 09:00: 300 receipts a second

An expense app lets people photograph their receipts. A text-recognition service reads each photo, extracts the merchant and the amount, and writes the result to the database. It can read 50 receipts a second, plenty for most of the day, when uploads trickle in at about 20 a second. Then, at 09:00 on the first Monday of the month, a reminder goes out and uploads jump to 300 a second for ten minutes.

If the upload API calls the recognition service directly, the service receives six times what it can handle. Requests wait, time out and are retried, the retries add to the pile, and the service spends the peak on work for people who have already given up. Keeping six times the capacity running would cover those ten minutes and sit idle for the rest of the month.

A queue in the middle

Queue-Based Load Leveling puts a queue between the two sides. The upload API stores the photo, writes a message, "read receipt 88123", to the queue, and answers the user at once: received, processing. The recognition workers take messages off the queue at their own 50 a second, whatever the arrival rate is. The queue's depth, not the service, absorbs the burst; the upload API and the queue still have to accept all 300 a second, but storing a message is cheap next to reading a receipt.

Bursts in, steady drain out

The upload API writes as fast as photos arrive. The recognition worker reads at its own rate, and so does the database behind it.

The database behind the workers now sees at most 50 writes a second as well, however hard the burst, and that is often the real point: the queue protects everything downstream of the consumer, not just the consumer. Inside a single process, the same idea with a bounded buffer between threads is Producer-Consumer. Here the queue is a broker between services, so a backlog survives a restart of either side, and of the broker too if it stores its messages durably.

It works only because the user does not need the result in the upload's response. Answering "received" now and delivering the outcome later, by polling or a notification, is Asynchronous Request-Reply. A request whose caller must have the answer before it can go on, such as a card payment at a till, does not belong behind this kind of queue.

Where the burst goes

The burst does not disappear. It becomes a backlog, and the arithmetic fits on a napkin:

  1. From 09:00 to 09:10, 300 messages arrive each second and 50 leave, so the queue grows by 250 a second, to 150,000 messages at 09:10.
  2. From 09:10, arrivals fall back to 20 a second while the workers still take 50, so the queue shrinks by 30 a second.
  3. 150,000 messages at 30 a second take 5,000 seconds, about 83 minutes: the queue is empty again at about 10:33.
  4. A receipt uploaded at 09:10 has 150,000 messages ahead of it, so at 50 a second it is read about 50 minutes later.

The last line is the price. Nothing was overloaded and nothing was dropped, but someone who uploaded at 09:10 waits 50 minutes for a result that normally takes a second. Load leveling trades latency for stability, and whether that latency is acceptable is a product decision. For an expense report, an hour is usually acceptable. If it is not fine, the queue alone is the wrong answer, and the consumers need more capacity.

Depth says how much, age says how late

Two numbers tell you whether the queue is doing its job. Queue depth is how many messages are waiting. Message age is how long the oldest one has waited, and it is the one that tracks what users feel. A depth of 10,000 is ten seconds of work for consumers that take 1,000 a second, and more than half an hour for consumers that take 5. Alert on age, against the delay the product accepts.

The shape of the depth over days matters as much as its size. A sawtooth that rises with each burst and falls back to zero is leveling at work. A line whose lowest point climbs a little every day means the consumers handle less than the average arrival rate, and no queue absorbs that: the backlog has no steady state and grows until something gives. Often that is the broker's retention limit, after which the oldest messages are deleted unprocessed. Treating the queue as infinite headroom instead of watching its depth hides exactly this: leveling has turned into silently deferred failure.

When the queue only grows

Leveling absorbs bursts above the average. It cannot absorb an average above capacity, and when the queue only ever grows, the fix is no longer the queue. You can add workers on the same queue, Competing Consumers, which also raises the load on the database the queue was protecting. You can refuse the least important work past a limit, Load Shedding. Or you can cap the queue and make producers wait, so the pressure reaches the source, Back-Pressure. Backpressure & load shedding shows how the last two combine, and Message queues & async workers covers the broker itself.

Draw the burst, not just the box

The Queue-Based Load Leveling challenge asks for this system: bursty Clients writing tasks to a Queue, a Service consuming at its steady rate, and a drawing that shows bursts lengthening the queue instead of overloading the consumer. Put the rates in the titles, such as "Uploads, 300/s peak" and "Worker, 50/s", so the mismatch is visible at a glance, and mark the Queue as the buffer. The point of the drawing is the steady side: the consumer drains at a sustainable pace whatever arrives, and the depth rises with the burst and comes back down rather than climbing forever. Competing Consumers and Load Shedding are in the palette too; they belong in the drawing only as the answer to a queue that never empties, or to a wait the product cannot accept.

Practice this