Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Notifications & transactional messaging

Feature code says what happened; one notification service decides who hears about it, on which channel, and whether they have heard enough: preferences and opt-outs, templates, deduplication and per-user caps, a queue per channel so a slow provider holds up no other, retries with a dead-letter path, and delivery status.

Why did the gate change come by email?

A passenger on flight 412 to Lisbon writes to an airline's support team: "I turned on push notifications for flight changes. Why did the gate change reach me by email, twenty minutes after boarding closed, while the app said nothing?"

The answer was in the code. Flight operations sent its own emails when a gate moved. The bookings team had built SMS for delays, through a different provider account. The app team handled push, but only for check-in reminders. The setting the passenger had switched on was read by one of those three, and it was not the one that knew about gates. Every team that wanted to tell a passenger something had written its own sending code, with its own idea of preferences, wording and retries.

The airline's fix was to stop feature code from sending anything. Flight operations now emits an intent, a small message saying what happened and to whom: flight.gate_changed, passenger 2 on booking QX7P2L, flight 412, gate B12 to B31. It names no channel and holds no wording. One notification service takes it from there: which channels this passenger wants for this kind of event, the text in her language, whether she has already been told, and what became of each attempt.

A gate change concerns all 180 passengers on the flight, so it becomes 180 intents, one per passenger. That is cheap at this size; spreading one event to millions of recipients is a different design question, the one in Fan-out: push vs pull.

The intent goes on a queue

Flight operations should not wait for any of this. When a gate changes, it puts the intents on a queue and carries on, and the notification service consumes them at its own pace. If the SMS provider takes eight seconds to answer, the gate change is still saved and shown on the airport screens on time; if the notification service is restarting, intents wait in the queue instead of being dropped. The queue itself is the subject of Message queues & async workers.

Saving the gate and enqueuing the intents are writes to two systems, though, and a crash between them leaves a gate that moved with nobody told. Writing the intents in the same database transaction as the gate change and relaying them afterwards removes that window: Transactional Outbox.

One intent in, a queue per channel out

Flight operations only enqueues the intent. The notification service checks preferences and duplicates, then hands each message to its channel's own queue, so a slow SMS provider backs up the SMS queue alone.

Deciding who hears what

For each intent the service reads the passenger's preferences, which channels she wants per kind of message: push for gate changes and boarding, SMS for delays when she is abroad without data, email for receipts. They are read for every intent, so they are served from a fast store, with the durable copy in a database. Now the setting she switched on is read by the one place that sends.

Some preferences carry legal weight. A passenger who unsubscribes from offers, or replies STOP to a promotional SMS, has to stop getting them, and many countries set how quickly. Operational messages such as a cancellation are usually kept outside those opt-outs, because the passenger needs them; where that line sits is a product and legal decision, enforced once, in this service. The wording lives in templates, one per message type, channel and language, so rewording the gate-change text is an edit to a template, not a release of flight operations.

Told once, and not too often

Queues like this one usually deliver at least once. If the service sends the gate-change push and crashes before acknowledging the intent, the intent comes back and the push goes out again. So each intent carries the id of the event behind it, and for each channel the service records a key of event id, passenger and channel in a fast store, with an expiry longer than any delay after which a copy can still turn up, and skips that channel's message when its key is already there. The key has to be set in one atomic step, only if absent, or two copies handled at the same moment both find it missing. Where a crash lands still matters: between setting the key and queueing the message, the message is lost; after a provider accepted a message but before its answer came back, a retry sends it again. Idempotency & exactly-once illusions explains why a key alone cannot rule both out.

Volume is the other way to flood someone. On a bad evening, flight 412's estimated departure moves eight times in an hour, and eight pushes in an hour train passengers to ignore the ninth, which might be the one that matters. A per-user cap limits how many messages of one kind the service sends one passenger in a window, while boarding calls and cancellations stay uncapped. Folding is a separate choice: an update held back by the cap can be replaced by a newer one while it waits, so the next push carries the latest estimate, not a stale one. The counting behind such a cap is the same as for any per-client limit (Rate limiting).

The evening the boarding calls waited

The first version of the service put every outgoing message on one send queue, worked by one pool of workers. One evening the SMS provider slowed from 200 milliseconds per request to 20 seconds, SMS sends took up every worker, and the boarding-call pushes behind them, which their provider would have taken in milliseconds, went out after the gates had closed. A queue per channel, each with its own workers and its own timeout, keeps a slow SMS provider's backlog in the SMS queue. Webhooks & third-party integration works through the same trap with one slow receiver among thousands, and Cascading failures: how one dependency takes down its callers names the general shape. The Diagnose: Notifications Behind a Slow Provider challenge starts from a design with neither the intent queue nor the split.

Within each channel, a failed send goes back on its queue with growing, randomised waits (Retry). A permanent error, such as a number that does not exist, stops at once, and a message that runs out of attempts lands in a dead-letter queue, where an alert brings in a person (Dead Letter Channel). A boarding call also carries an expiry: once the gate closes, the service stops retrying it and records it as failed, though a copy a provider accepted before then can still arrive late. Switching it to SMS after a failed push can beat retrying the push.

What the record says

Each message keeps a delivery status: queued, sent, delivered, failed. Sent means only that the provider accepted it. Later states, where the channel reports them, arrive from the provider as callbacks to an endpoint the airline exposes, which are webhooks; a push often gets no report after acceptance. With that record, the support team answers the Lisbon passenger with one lookup: which channel each message went to, when the provider accepted it, and whether a later status came back. For email, what happens after the provider accepts a message has its own article, Email delivery: authentication, bounces & sender reputation.

Break it on purpose

Build the Multi-Channel Notification Service challenge, then break your design twice on paper, following one gate-change intent through it. Suppose the SMS provider slows down to 20 seconds: the gate-change push must still reach the passenger without waiting for any SMS. Then suppose, with no crash and no lost provider answer, the same gate-change intent reaches the queue twice: the passenger must get one message on each channel she has enabled, not two. If the first test fails, look for a queue the channels still share; if the second fails, check the duplicate key, whether it is set in one atomic step, and whether it is checked before messages reach the channel queues.

Practice this