Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Sequential Convoy

Process related messages in order while still processing unrelated ones in parallel: choosing the group key, hashed partitions or session-enabled lanes, what one worker per group caps, and what stays unordered between groups.

Two messages about one contractor's office badge, and two workers at the access-badge system:

  1. At 17:02:11, worker-a takes BadgeIssued for employee 3317 and starts loading the badge onto the door controllers, about six seconds of work.
  2. At 17:02:14, worker-b takes BadgeRevoked for employee 3317, because reception has noticed the contract ended yesterday.
  3. At 17:02:15, worker-b has removed the badge from the doors loaded so far, marked it revoked and acknowledged the message.
  4. At 17:02:17, worker-a has loaded the remaining doors, marked the badge active and acknowledged its message.

Neither worker logged an error. The badge was revoked four seconds after it was issued, and the record says active. Three weeks later it opened the loading dock, which is how security found out.

One worker is too few, and most order does not matter

Competing Consumers explains why this happens: workers sharing a queue keep no order between them. The blunt fix is one worker. But Monday mornings bring 2,400 badge changes an hour, and at six seconds a message one worker manages 600. The system runs eight workers (4,800 an hour) for that reason.

Employee 3317's BadgeIssued has to come before 3317's BadgeRevoked. Whether it comes before employee 5120's anything is irrelevant. A Sequential Convoy keeps the order within a group of related messages, here everything about one employee, while different groups run concurrently. Global order is what limits you to one worker; order per group lets you use all eight.

Two ways to get a lane

The usual way is to hash a group key to an ordered partition. The producer sends the employee id as the key, the hash picks one of eight partitions, and each partition is read by one worker at a time. Kafka works like this: messages with the same key go to the same partition as long as the partition count does not change (Sharding & partitioning shows why changing it moves keys), and a consumer group gives each partition to one of its members. All of 3317's messages queue up behind each other.

Order within a group, parallel across groups

Numbers on the edges are employee ids. The hash sends 3317 and 5120 to the same partition, so they share a worker. 7044 runs alongside them.

Many groups may share one partition. With 12,000 employees and eight partitions, about 1,500 people share each lane, so 3317's message can wait behind 5120's for no reason of its own.

The other way is a session-enabled queue, which gives each group its own ordered lane. On Azure Service Bus, the sender sets a session id on each message, and a receiver locks one session and gets that session's messages in order while other receivers work on other sessions. Amazon SQS FIFO queues have a close relative in the message group id: while any message of a group is in flight, SQS holds back the rest of that group, and other consumers take other groups. A slow employee then holds up only that employee's messages and the one worker on them. One receive can return several of a group's messages, so that worker has to handle them in order, stopping at a failure.

Either kind of lane keeps the order in which messages reached it. If two services publish about 3317 without coordinating, the lane preserves whichever arrived first. Putting messages back in sequence at the receiver, using a number assigned at the source, is Resequencer.

Grouping by what, exactly?

One question picks the key: which messages, if swapped, would leave a wrong result? Those need the same key, and the others should not share one if you can help it. The badge team got this wrong three ways.

  • Too coarse. The first version keyed by building, because door controllers belong to buildings. In move week head office got 1,000 changes an hour against a lane that manages 600, so after one hour it was 400 messages behind, and a new revocation waited 40 minutes. One giant partition serializes everything in it.
  • Too fine. Keying by badge number looked tidy until someone lost a badge. The old badge's BadgeRevoked and the replacement's BadgeIssued then had different keys, and when the second ran first, the rule of one active badge per employee refused the new badge.
  • Random. A random key, or none on a broker that then spreads messages around, puts one employee's messages on different partitions. Every worker is busy, and the key has destroyed the order the pattern exists to preserve.

One worker per group, and what that caps

Order holds because no two workers are on one group at the same time and the one that has it handles the group's messages one after another, so a group's speed is one worker's speed, however large the pool. Then reception began issuing every visitor badge under a single account, visitors, at nine o'clock twice as fast as one worker can load them. That is a hot group, and adding workers does nothing, because a ninth worker finds nothing it is allowed to take. On a shared partition it also delays the other people hashed to it. The fix is a finer key, since visitors need no order among themselves, or a faster handler. Partitions add a second cap of their own: in a consumer group, eight partitions keep at most eight workers busy.

A retry has to keep its place

One morning a door controller timed out on BadgeIssued for another employee. The team's shared retry helper put the message on a delayed retry queue and acknowledged the original, so the lane moved on. BadgeRevoked ran straight away. Thirty seconds later the retry came back and the badge went active, with only one worker involved.

So the retry stays in the lane, and the lane waits. The consumer stops at the failed message, handling nothing later from its group, even what it already holds, and retries it in process or gets it redelivered: on a session or a FIFO message group it comes back ahead of the rest of its group. On Kafka, the consumer seeks back to that offset and drops what it fetched after it. That means a stuck message holds its group, and on a partition every group sharing it. A message that will never succeed would hold them until something diverts it, and recognising one is what Poison Pill teaches. Diverting it frees the lane and leaves a gap: the next message may be a BadgeRevoked for a badge that was never loaded, and the handler has to cope with that. Letting revocations overtake to escape the wait would be Priority Queue, which pulls the other way: inside one employee's group it would run the revocation ahead of the issue.

When a lane changes hands

Workers restart and get deployed, so lanes move. In a Kafka rebalance the new owner resumes from the last committed offset, which can repeat messages already handled; Idempotent Consumer deals with those. In-flight work is harder. A worker stalled past the group's timeout loses its partition and may wake up halfway through BadgeIssued, while the new owner is already on that message and the BadgeRevoked after it, so for those seconds two workers are on one group. An expired session lock or SQS visibility timeout can do the same. So "one worker per group" holds while ownership holds, and a handler whose writes must not go backwards needs its own check: each message carries an increasing version per employee, assigned by one source, and a write older than the stored version is ignored, with the comparison and the write done as one atomic step. That protects the record; a stale worker can still load a revoked badge unless the same check runs at the door controller as it applies the change.

Drawing a lane in the Challenge

The Event-Driven Orders on Azure Challenge asks you to preserve per-customer order using queue sessions, so put Azure Service Bus in front of the Azure Functions that process orders and label that edge with the group key, such as "session: customer id". Then trace two orders from one customer and one from another: which may run at once, and which must wait?

No promise between two employees

The convoy says nothing about order across groups. A contractor's badge needs an active sponsor. At 17:00:00 the system publishes BadgeRevoked for manager 7044, and at 17:00:10 BadgeIssued for contractor 3317, naming 7044 as sponsor. If 7044's lane is behind, the handler for 3317 checks the sponsor, finds 7044 still active and issues the badge. Had the lanes been idle, it would have refused. Same two messages, two outcomes, decided by which lane happened to be busier. A decision that spans two groups either gets a single key, at the price of a coarser lane, or is made where both facts are known in order, here in the system that publishes the messages.

Practice this