Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Message queues & async workers

Moving slow work out of the request into a queue that workers drain, what the caller gives up for it, why a job can run twice, and when a queue is the wrong tool.

Three minutes of CPU per upload

A print bureau lets customers upload a 3D model and get back a price and a print time. Both come from the slicer, the program that cuts a model into the thin layers a printer lays down and plans every move of the nozzle. A keychain slices in about five seconds; a full-size helmet at fine layer height takes three minutes of one CPU core.

The first version sliced inside the upload request, holding a web server process for the whole slice. When a design school's class of 40 uploaded their coursework within ten minutes, every web server process was busy slicing, the login page stopped answering, and the load balancer cut the longest uploads at its 60-second timeout. Students pressed Upload again, which started a second slice of the same model.

Taking the slice out of the request

The upload request now stores the model file, writes a short message, slice model 4471, to a message queue, and answers the browser: received, slicing. Separate processes, the workers, take messages from the queue and run the slicer. No web server process waits on the slicer any more, and the slow work runs at whatever pace the workers keep.

Upload, queue, slicers

The upload API hands each job to the queue and returns. Each job goes to one slicer, which records the result.

Look at what the request gave up. Its response used to carry the outcome, a price or an error such as "this mesh has holes"; now, when it returns, nobody knows whether the slice will succeed or when. In exchange it answers as soon as the file is stored and the message accepted, and keeps answering while the workers are slow, busy or restarting, because messages wait in the queue until a worker is free. The customer's page has to learn the outcome another way, by asking again or by being told, and Asynchronous Request-Reply works through that handshake.

The class of 40 becomes 40 waiting messages, which four workers clear in about fifteen minutes at 90 seconds a model while the rest of the site carries on. Queue-Based Load Leveling takes that burst-to-backlog arithmetic further; adding workers to the same queue, and what it does to ordering, belongs to Competing Consumers.

"Received" has to mean stored

The upload's answer promises that the job has been accepted and will be attempted. If the API replies before the broker has the message, or the broker holds it only in memory and restarts, the job page says "slicing" forever and no worker ever sees the job. So the producer returns only after the broker has durably accepted the message and confirmed it. RabbitMQ, for example, sends that confirmation, a publisher confirm, once the producer turns confirms on, for a persistent message on a durable queue once the message is written to disk, but it also confirms a message that reached no queue, which the producer learns about only by publishing with the mandatory flag. Guaranteed Delivery follows that durability hop by hop. The upload also makes two writes, the job's row in the database and the message, and a crash can fall between them, the gap Transactional Outbox closes.

Done is when a worker says so

What decides the system's behaviour is when the broker forgets a message. This order keeps jobs from disappearing:

  1. The broker hands message 4471 to Slicer 2 and keeps it, marked as taken.
  2. Slicer 2 slices the model and writes the print file and the price to the jobs database.
  3. Slicer 2 acknowledges the message, and only then does the broker delete it.
  4. If Slicer 2 dies, or its connection drops, before acknowledging, the broker makes 4471 available again and another worker takes it.

Acknowledge before the work instead, and a crash halfway through loses the job: the message is already deleted and nothing will try it again. The safe order is not always the default. Celery, a common Python task queue, acknowledges a task just before running it unless the task is configured with acks_late; even with it, a task whose worker process dies abruptly is acknowledged, not redelivered, unless task_reject_on_worker_lost is set too.

Why model 4471 can be sliced twice

Acknowledging after the work closes that hole and opens a smaller one. Slicer 2 writes the print file and the price, and its machine loses power before the acknowledgement leaves. The broker cannot tell "died before slicing" from "died after slicing". Delivering again is the only choice that does not risk dropping the job, so Slicer 1 slices 4471 a second time.

This is at-least-once delivery: a message keeps being delivered until a worker acknowledges it or a retry limit sets it aside, so the same job can be processed more than once. A worker slower than the broker's redelivery deadline, or a broker that occasionally delivers twice, does the same. For the slice, as long as the same model and settings produce the same file and each write replaces the print file whole, the second run only wastes CPU, overwriting it with an identical one. The next step is less forgiving: the worker books the model a printer slot, and a second run books a second slot that nothing will fill. The fix is a handler that records what it has already applied, as in Idempotent Consumer; why a queue plus a database still does not make processing happen exactly once is the subject of Idempotency & exactly-once illusions.

Redelivery has a darker case: a job that fails every time. A model whose faces intersect crashes the slicer, and each worker that takes the message back crashes in turn. With no retry limit, this poison message loops forever, costing a worker on every delivery, and on a queue that releases the next message only once this one is acknowledged it blocks every job behind it. With a limit and nowhere to put it, it is dropped after the last attempt with nothing kept to inspect. Setting it aside instead, in a dead-letter queue with the reason it failed, is what Dead Letter Channel is about.

What the queue should not be asked to do

The queue is a hand-off, not the system of record. Where model 4471 is comes from its row in the database, which the workers update; most queues cannot look up one message, and an acknowledged one is gone. Nor is the queue bottomless: with no workers added as it grows and no backpressure on the producers, messages pile up without bound until the broker's limits decide what happens to them.

One queue for every kind of job is another trap: the bureau's ten-second printability check, which customers wait for on screen, would sit behind helmet slices for minutes. Jobs with different latency needs or priorities get queues of their own, with workers sized for each.

And some work should stay in the request. Checking that the file is a readable model takes milliseconds, and its refusal belongs in the upload's response, not on a job page later. When the work is quick and cheap and the caller needs its result in the same response, doing it in the request is usually simplest: with no spike to absorb and no retry that has to outlive the request, a queue adds a broker to run, a status to track and one more way to fail, for little in return.

If pricing, the print-farm scheduler and analytics each had to react to the same upload, each would need its own copy, which takes the topics of Pub/sub: topics, subscriptions & replay rather than a work queue; the Distributed Message Queue Challenge builds that Kafka-style kind of broker, a partitioned log with ordering per partition and offsets per consumer group, not the work queue described here.

Submissions waiting for a verdict

The Online Code Judge Challenge has the slicer's problem with code in place of models: a submission runs against hidden tests in a sandbox for longer than a request should wait, and the brief asks for submissions queued across a worker pool with Kafka as the broker on its palette and Queue Workers draining it. On Kafka a worker usually acknowledges by committing its consumer group's offset, and the record stays in the log. Beside the boxes, answer in writing:

  • When the submit request returns, what has the broker already done with the submission?
  • When does a worker acknowledge, and what happens to a run cut short before that?
  • If a submission is judged twice, which write is safe to repeat: saving its verdict, or adding one to the contestant's attempt count?
  • Where does a submission that crashes every sandbox end up after its last attempt?

One question the queue cannot answer is the contestant's own: is it judged yet? That lives in the submission's row, which the workers update, not in the queue.

Practice this