Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Message Broker

A separate process that accepts, stores and routes messages between applications, so each one knows only the broker and never another application's address: what that takes off the senders, and what it costs to run the one piece every flow now passes through.

Eighteen links and counting

A regional news group publishes from three places: the editors' content system, the ingest that pulls agency wire copy all day, and the photo desk. Six systems need to hear when something is published or corrected: the website, the mobile app's push service, the search index, the newsletter builder, the archive and the analytics pipeline. The first integrations were direct calls. When an editor pressed Publish, the content system called the website, then push, then search, then the archive, each with its own address, credentials, timeout and payload.

Three senders and six receivers make up to eighteen links, each written, deployed and broken on its own. The content system's publish code knew where every receiver ran. When the search team moved its indexer to a new cluster, the content system needed a release. When the archive went down for its Sunday maintenance, publishing a story failed or hung until a timeout, because the call to the archive was part of publishing. And when the podcast team asked for every published story too, that was one more link, in each sender that had stories to give.

One address for everyone

A Message Broker is a separate process that sits between the applications and holds their messages on the way. The content system hands each story to the broker as a message, and its part is done; the broker passes the message on to every receiver that has asked for stories. Each application knows two things: where the broker is, and what the messages look like. None of them knows where another application runs, or even that it exists. Eighteen links become nine connections, one per application, and the podcast service is a tenth, with no change to any sender.

Every application talks to the broker

Senders hand their messages to the broker, and each receiver reads from its own queue there. No edge joins a sender to a receiver.

Two kinds of decoupling came with it. The first is in location: the search team can move its indexer again, and only the indexer's own connection settings change. The second is in time: when the archive goes down on Sunday, publishing carries on, because the content system's work ends once the broker has accepted the story. The stories wait on the broker, as long as it has room for them, and the archive works through them when it comes back. Publishing no longer depends on the slowest receiver being up.

What the broker does not remove is the receivers' dependency on the message. If each message carried only a story id, and every receiver called the content system back for the text, the content system would be in six receivers' critical path again. A message that carries what its receivers need keeps the senders out of their way.

The work moves with the links

Moving every link into one process moves their work there too: the broker accepts each message and confirms it to the sender, stores it, routes it, delivers it, and tracks what each receiver has acknowledged, message by message or as a position in a log, so that it can deliver the rest again. How it hands messages out depends on the channel. A queue gives each message to one of the workers reading it, the subject of Message queues & async workers; a topic gives every subscriber its own copy, as Pub/sub: topics, subscriptions & replay explains; and what an acknowledgement actually promises is the question Guaranteed Delivery answers. RabbitMQ, Kafka and the managed services of the large cloud providers all fill the role, on different models underneath: some remove a message once it is acknowledged, others, Kafka the best known, keep it in a log for a retention period while each consumer tracks its own position.

Election night, one disk

For its first year the broker ran on one virtual machine. On the Friday before the regional election, the analytics team stopped its consumer for a migration that overran. Every published story and correction kept arriving for analytics, nobody read them, and its queue grew all weekend. At 22:40 on election night the broker's disk filled. A broker that cannot store a message cannot accept it, so it refused new ones from every sender at once: editors could not publish results, and wire copy stopped coming in. In the old design, a stopped analytics pipeline could fail the calls to analytics, but it could not fill a disk that every other link shared.

That is the price of the hub. The broker is a central dependency, and when it stops, every flow through it stops together. So it is run as a service in its own right:

  • Clustered, across machines, so that losing one node need not stop the broker, using each product's own way of replicating messages between nodes.
  • Sized for the backlog, not just the normal traffic: the disk has to hold what accumulates while a receiver is away, for as long as you are willing to let it be away.
  • Monitored on the numbers that warn early: queue depth and the age of the oldest message per receiver, disk and memory use, connections. A queue that only grows means a receiver has stopped or cannot keep up, and that is cheaper to see on Friday than at 22:40 on Sunday.
  • Limited per queue, with a maximum length in messages or bytes, sized so that one stopped receiver's backlog fits on the disk; a message time-to-live can sit beside it, but it caps age, not size. Depending on the product and its configuration, reaching the limit drops the oldest messages, rejects new publishes, or moves messages to a Dead Letter Channel, which needs a limit of its own. Choose the behaviour for each queue.

Rules that nobody owned

The second cost arrives slowly. Many brokers can route on a message's headers, and the push team asked for a rule there: push only stories tagged breaking, none between 23:00 and 07:00 unless also tagged urgent, never sport on weekdays. Each new exception went into the broker's routing configuration, because the broker was the one place every story passed. Two years later the newsroom's push policy lived in configuration that no editor could read, changed by the platform team, with no tests. When a weekday story about a stadium roof collapsing sent no push, it took an afternoon to find out that a sport tag had suppressed it.

A broker that holds business rules has become an application that every team depends on and whose rules no team owns. Let the broker route on what a message is: its channel, its type, a header the sender sets for routing. Keep the decision about what to do with it in the service that owns that decision; here, the push service reads every published story and applies the newsroom's policy in code that it tests and deploys. When messages do need routing by their content, a Content-Based Router does it: a step of its own that reads each message and sends it to the channel its content calls for, with an owner and tests, not a line buried in the broker's configuration.

Take the broker out and look

The Messaging Endpoints & Broker Challenge asks for this separation: a Message Broker routing between senders and receivers so that they never link directly, alongside a gateway, endpoints, a channel adapter and a wire tap. Before you submit, remove the broker node from the drawing in your head. If any sender is still connected to any receiver, that edge is a link the broker was meant to replace. Then read the labels on the broker's edges: they should say what the broker does on each, such as publish and route, and never carry a business rule, which belongs to a service.

Practice this