Three workers, one queue, one log
An order service puts a message on a queue for every order placed: charge the card, email the receipt. Three worker processes, on three machines, take messages from that queue. A slice of their combined log:
10:02:01.120 worker-a took order-510110:02:01.124 worker-b took order-510210:02:01.131 worker-c took order-510310:02:01.480 worker-b done order-510210:02:01.482 worker-b took order-510410:02:01.910 worker-a done order-5101
Each order appears once, handled by whichever worker was free. Nobody assigned order-5104 to worker-b: it asked for the next message when it finished 5102, and got it. That is Competing Consumers: several consumers on the same queue, competing for its messages, and the broker hands each message to one of them at a time, not to all of them. Within a single process, the same shape, workers pulling from a shared queue, is Thread Pool. Here the workers are separate processes on separate machines, any of which can die at any moment, and that is what the rest of this article is about. At the level of a whole system design, the same setup appears in Message queues & async workers.
Each message goes to one worker, whichever asks first. Adding a fourth worker adds capacity without changing the order service.
The gain is plain. One worker handles 40 orders a second, so a peak of 100 a second builds a backlog; three workers handle 120. Add workers to go faster; lose one and the rest carry on. That holds until the thing every worker writes to, usually the database, runs out of capacity: ten workers send it ten times the writes, and past its limit more workers only make each one slower. Sizing against what sits downstream is the lesson of Object Pool, and draining at a deliberately steady rate is Queue-Based Load Leveling. The rest of this article is what changes the moment there is more than one worker.
Handed out is not finished
"Each message to one worker" sounds easy until a worker dies holding one. If the broker deleted a message the moment it handed it out, a worker that crashed halfway through would lose that order for good. So brokers separate handing a message out from finishing it. The worker receives the message, does the work, and then acknowledges it (in some APIs, deletes it). Until then the broker keeps the message and normally does not offer it to the other workers. If the worker's connection drops, or it does not acknowledge in time, the broker makes the message available again and another worker takes it.
A message is done only when a worker says so. On Amazon SQS, for instance, the "in time" is a visibility timeout, 30 seconds by default: a received message stays hidden from other workers that long, then reappears unless the worker has deleted it.
The order that was charged twice
Back to the log. worker-c took order-5103, charged the card, and was stopped by a deploy before it acknowledged. When the timeout expired the message was available again; worker-a took it, charged the card a second time, and emailed the receipt.
The broker behaved correctly. It cannot tell "crashed before charging" from "crashed after charging", and the choice that never loses an order is to deliver it again. A slow worker causes the same thing without crashing: if processing takes longer than the visibility timeout, the message reappears while the first worker is still on it, and two workers process it at once, which is why the timeout is set above the longest processing time, or extended while the work runs. Some brokers, Amazon SQS's standard queues among them, can also occasionally deliver a message twice with no crash and no slow worker. So any message can arrive more than once, and the handlers have to be idempotent: processing the same message twice must leave the same result as processing it once. The usual guards each protect one kind of effect: an idempotency key the payment provider recognises stops the second charge, and recording the processed message id in the same database transaction as your own write stops a duplicate write, but not a charge made outside that transaction. Idempotent Consumer covers both.
Each worker in order, the queue out of order
With one worker, messages are processed in the order the queue hands them out. With three, they are not. The order service publishes "order-5110 created" and, 40 ms later, "order-5110 cancelled". worker-a takes the first and worker-b the second. The cancellation is quicker to process, so it runs against an order that does not exist yet, and then the creation brings the cancelled order back to life. Each worker handled its own messages in order; across the workers, the order was lost.
When some messages must stay in sequence, usually the ones about the same order, account or device, they have to go through one ordered lane while unrelated messages still run in parallel; Sequential Convoy shows how, and letting urgent messages overtake routine ones on purpose is a different pattern, Priority Queue. Without such a lane, do not assume message order is preserved across competing consumers: once several workers pull concurrently, per-consumer processing order is no longer global order.
The message that takes every worker down
One order carries a malformed address that makes the email library throw. worker-a takes it and fails, and the message comes back; worker-b takes it and fails; then worker-c. A poison message like this is delivered again and again, and every delivery costs a worker its time, or its process if the failure is a crash. The defence, which most brokers support directly, is to count deliveries and, after a few, move the message to a Dead Letter Channel instead of offering it again. There someone can read the address, fix the cause and replay the order, while the three workers go back to orders that can succeed.
Scale one queue on a Grid
The Competing Consumers challenge asks for producers sending to one shared queue, several consumers pulling from it, each message going to one consumer rather than to all of them, and throughput rising as consumers are added. Draw the Client as the producer, a single Queue and three Competing Consumers nodes, label the edges so that each message visibly reaches one worker, not every worker, and put a rate in each worker's title, such as "40/s". Then add a fourth worker: throughput rising with workers, while the order service stays untouched, is the claim the drawing has to make. A short note that handlers are idempotent says what that claim costs.