Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Pub/sub: topics, subscriptions & replay

One event, every subscriber: topics and subscriptions, consumers sharing one subscription's work, retention and replay for a subscriber that was down or arrives late, ordering per key, at-least-once delivery, and why one slow subscriber need not slow the others.

"Can we start with last month's scans?"

A parcel carrier scans every parcel at every step: picked up, sorted at a hub, loaded on a van, delivered. Two teams already live on those scans. Notifications texts the recipient when the parcel is out for delivery, and the ETA model updates the predicted delivery time. A third team is building lost-parcel detection, which flags a parcel that has gone quiet for two days, and it asks the question that decides the design: can it start with the last thirty days of scans?

If the scan service called each team's API in turn, the answer would be no: the history would be whatever each team happened to keep. The scan service would also have to know every consumer, so the new team would need a change in a service it does not own, and a slow ETA model would hold up every scan.

Publish once, subscribe as often as you like

Pub/sub (publish/subscribe) turns those calls around. The scan service publishes each scan once, to a topic called parcel-scans, and does not know who reads it. Each team creates a subscription to the topic, and every subscription receives every event. The new team is one more subscription; the scan service does not change. If instead only one kind of consumer ever needs each event, you want a work queue, the territory of Message queues & async workers; and one personal copy per follower, a feed, is the push-or-pull decision of Fan-out: push vs pull.

Inside one subscription the work can still be shared. Notifications runs two workers, and the recipient should get one text, not two. The workers share the subscription, and each event goes to one of them: that is Competing Consumers, applied to a single subscription. The topic copies events across subscriptions; a subscription divides them among its consumers. Kafka calls a subscription a consumer group: every group reads the whole topic, and within a group each partition is read by one member at a time.

One topic, three subscriptions

Each subscription (a Kafka consumer group) reads the whole topic at its own offset. The two notification workers share one subscription by splitting its partitions; the new lost-parcel group starts from the oldest retained scan.

TipThis is the infrastructure side of two Design Pattern articles. For the modelling view, publishers and subscribers that never reference each other, see Publish / Subscribe and Event Bus.

Where the past is kept

Whether the new team gets its thirty days depends on what the broker does with an event after delivering it:

  • Kafka appends every event to a log, split into partitions, and keeps it for a retention period whatever the subscribers have read: seven days by default, configurable per topic by time or by size. For each partition, a subscription is little more than a position in that log, its offset, which the consumers commit as they go. Reading deletes nothing, so a new subscription can start from the oldest retained event. In Kafka's consumer, for a group with no committed offset, that is auto.offset.reset set to earliest; the default, latest, starts at the end and sees only new scans.
  • A broker that pushes into a queue per subscription, such as a RabbitMQ exchange fanning out to one queue per team, copies each event into every subscription's queue and removes it from that queue once that team's consumer acknowledges it. A subscriber that was down finds its queue waiting. A new subscription starts empty: events are copied only into queues already bound to the exchange, so nothing published before the new queue was bound is in it.
  • Redis Pub/Sub keeps nothing. It delivers to whoever is connected at that instant, and a subscriber that drops for two seconds misses those two seconds for good.

So the lost-parcel team needs a log with enough retention. With parcel-scans kept for 35 days, it starts a new consumer group from the earliest offset and reads the history as fast as it can; it catches up with the live scans only if it reads faster than new ones arrive. The same offsets cover a subscriber that was down: notifications shipped a crashing release and stopped for an hour, its committed offset stayed where it was, and when it came back it read the hour it had missed. Retention does not wait for subscribers, though: the broker deletes old log segments by time or by size, whether anyone read them or not, and a size limit can delete them before the 35 days are up. A subscriber away for longer, or a replay that reads more slowly than events arrive, loses the oldest part of what it has not read, and all of it if its consumer's auto.offset.reset is the default latest. Committed offsets expire too: by default Kafka deletes a group's offsets seven days after its last member leaves, so a team down for ten days restarts wherever auto.offset.reset says, however long the topic keeps its scans. And a month of history is also a month of older event shapes that today's code must still read, the problem taken up in Schema evolution: event formats & compatibility.

In order, but only per parcel

parcel-scans has 12 partitions so that 12 consumers per group can read in parallel, and Kafka keeps order only inside a partition. If the scans were spread across partitions at random, parcel 7731's "delivered" could sit in partition 3 and be read before its "out for delivery" in partition 8, and the tracking page would go backwards. So the scan service gives every event a key, the parcel id, and the producer sends every event with the same key to the same partition (Kafka's default partitioner hashes the key). All of parcel 7731's scans are read in the order they were written to that partition. Across parcels there is no order at all, and for tracking none is needed.

The key is whatever must stay in order together, and nothing broader. Key by depot instead, and every parcel through the busiest hub lands in one partition, read by one consumer per group while the others have little to do. Key by scan id and the per-parcel order is gone again. The partition count is the other half of the choice: a consumer group has at most one busy member per partition, and adding partitions later sends many keys' new events to a different partition while their old events stay where they were, so one parcel's new scans can be read before its older ones while those still wait unread in the old partition.

The text that arrives twice

Notify worker 1 reads "out for delivery" for parcel 7731, sends the text, and is stopped by a deploy before it commits its offset. Its partitions move to worker 2, which resumes from the last committed offset and sends the text again. Committing first and processing afterwards only swaps the failure: a crash between the two loses the scan instead. So most subscribers get at least once delivery and must make a repeated event harmless, which is the job of Idempotent Consumer. Repeats also start at the publisher when it retries after a timeout, so the event id should come from the scan itself, not from the broker; a publisher that must commit a database change together with the event needs a Transactional Outbox. Kafka's transactions can make a loop that reads and writes only Kafka exactly once (Stream processing: windows, watermarks & state explains how), but they cannot take back a text already sent to a phone.

One slow subscriber

The lost-parcel group spends its first hours a month behind, reading history. On a log, nobody else waits for it: each group reads at its own offset, notifications and the ETA model stay current, and the scan service does not wait for any group. They all still share the brokers' capacity: the history read loads their disks and network, so a big replay may need a Kafka quota. The number to watch is each group's consumer lag, the distance between the newest event in a partition and the group's committed offset. Alert on it, and on the age of the group's oldest unread event, with a margin before whichever retention limit, time or size, would delete that event first.

With a queue per subscription the isolation is weaker: the slow team's queue grows on the broker until a memory or disk alarm makes RabbitMQ block publishing connections, and then the scan service stalls and no team receives new scans, so bound each queue (a full one drops its oldest messages or refuses new ones), alert on its depth, and park events that keep failing in a Dead Letter Channel.

Build the log yourself

The Distributed Message Queue challenge asks you to design the broker this article has been leaning on, so a single Kafka box is not an answer there: it is the thing to draw. Draw brokers that own partitions and append each log to their own disk, consumer groups with committed offsets, and enough retention to replay from a past offset. Then test the drawing on the parcels: every scan of parcel 7731 in one partition, a new group reading a month of history from the oldest offset, and a worker that dies before committing, with the redelivery the brief asks you to handle. For the modelling side alone, Publish / Subscribe is the shorter challenge.

Practice this