Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Guaranteed Delivery

Each hop acknowledges a message only after it is stored durably, by that hop or the next, and the hop before keeps its copy until then, so a crash does not lose it. It is only as durable as each hop's storage and replication, it arrives at least once rather than exactly once, and each send pays for it in latency.

Four handovers between Submit and an adjuster

A claim filed from a car insurer's mobile app changes hands four times before an adjuster can open it. A driver photographs a dented door, fills in the form, taps Submit, and then:

  1. The app sends the claim to the claims API over the phone's connection.
  2. The claims API publishes it to the message broker, on the channel claims.submitted.
  3. The broker delivers it to an instance of the claims-intake service.
  4. Claims-intake writes it to the claims database, where it joins an adjuster's work queue.

Each handover has a moment when the sender has the claim and the receiver does not yet have it safely. A crash in that moment is ordinary on any day with enough claims: a phone going into a tunnel, a broker restarted for a patch, an intake instance stopped by a deploy. Whether the claim survives comes down to one question: who still holds a copy when the crash happens?

Nobody lets go first

Guaranteed Delivery answers it with an old mechanism, store-and-forward. Each hop tells the previous hop "got it" only once the message is in storage that survives a restart, its own or, for a hop that only passes it on, the next hop's, and a hop that holds a copy deletes it only after hearing "got it" from the next one. When every hop follows the rule, a crash of any one hop leaves the claim in at least one hop's storage, and the hop before the broken handover sends it again. For the claim, hop by hop:

  • The phone. The app saves the claim in its local database before trying to send it, and shows "Saved, it will be sent when you have signal" rather than "Submitted". It deletes the local copy only when the API answers 202 Accepted. A tunnel, a flat battery or the operating system killing the app leaves the claim on the phone, and the app tries again later.
  • The claims API. It answers 202 only after the broker confirms it has stored the message. If no confirmation arrives in time, the API answers with an error, and the phone keeps its copy.
  • The broker. Before it sends that confirmation, it writes the message to disk, to replicas on other machines, or both, depending on how it is configured. It keeps the message until a consumer acknowledges it; a log such as Kafka instead keeps it until its retention limits, by time and by size, delete it, and the consumer's acknowledgement is a committed offset, so those limits have to keep a message longer than any consumer can fall behind.
  • Claims-intake. It acknowledges the message to the broker only after the claim row is committed in the database. If the intake instance dies first, the broker delivers the message again, to another instance or to the same one after it restarts.
A copy is held at every hop

Each label is what a hop waits for before it lets go of the claim. Until then, a crash leaves a stored copy upstream, and that copy is sent again.

The acknowledgement that came too early

The rule is easy to break by accident, most often at the last hop. Many client libraries can acknowledge a message automatically, as soon as it is received. With that setting, claims-intake receives the claim, the broker deletes its copy, and the deploy stops the intake instance before the database commit. The broker has no copy, intake has no copy, and the claim is gone, from a channel everyone called guaranteed.

The same gap opens at any hop that says "got it" before the message is stored: an API that answers 202 while the publish is still in flight, a broker that confirms before the message reaches the storage it is configured to use. The acknowledgement is the moment the responsibility moves to the next hop, so it has to come after the store.

How durable is durable?

"Stored" is only as strong as the storage behind it, and that is a matter of configuration, not of the pattern. Some of what decides it:

  • In a RabbitMQ classic queue, a message survives a broker restart only if it was published as persistent to a durable queue; a quorum queue stores every message on disk. The publisher confirm for such a message is sent once it has been written to disk or, for a quorum queue, once a majority of the queue's replicas have accepted it.
  • In Kafka, the producer's acks setting chooses what it waits for. With acks=1, the partition leader confirms once the message is in its own log, and if that broker fails before a follower has copied it, a confirmed message can be lost. With acks=all on a topic with three replicas and min.insync.replicas set to 2, a confirmed message is on at least two brokers.

So "guaranteed" means this: once a hop has confirmed a message, the message survives the failures that hop's storage and replication were set up to survive, for as long as the hop keeps it: a message TTL or a retention limit discards what nobody consumed in time. A broker writing to a single disk survives a restart, not the loss of that disk, and a phone holding the only copy survives a tunnel, not being lost or wiped.

At least once, sometimes twice

The rule that prevents losses also creates duplicates. Claims-intake commits claim CL-20931 and crashes before its acknowledgement reaches the broker. The broker cannot tell "crashed before the commit" from "crashed after it", and the choice that does not lose the claim is to deliver it again, so another instance commits it a second time. The phone has the same problem one hop earlier: if the 202 is lost on the way back, the app sends the claim again and the API publishes it twice. Guaranteed Delivery is at-least-once delivery, not exactly-once: a confirmed message is not lost to a crash the storage was set up to survive, and some messages arrive more than once.

Duplicates are dealt with at the receiver. Claims-intake records each claim's id in the same transaction as the claim row, under a unique constraint so two deliveries running at once cannot both commit it, and skips an id it has already seen, the technique of Idempotent Consumer, with Message Deduplication as its broker-side companion. For the phone's retries to be recognised, the id has to be created on the phone when the claim is saved, so a resent claim carries the same one.

Delivered is not processed either: the broker's part ends when intake acknowledges, and what intake then does with the claim is beyond its sight. If the claims API also recorded the claim in its own database before publishing, a crash between the commit and the publish would leave a row with no message, the gap Transactional Outbox closes. A claim that fails on every attempt should go to a Dead Letter Channel after a bounded number of tries.

When losing a message is the better deal

Every confirmation waits for the store, a write to the broker's log, a copy on replicas or both, so each send is slower, more so with replicas in other zones, and brokers and producers batch to keep throughput up. The claims app pays that gladly: a lost claim means a customer phoning to ask where it went. The same app also shows the driver where the tow truck is, from a position the truck sends every five seconds. If one position is lost, the next replaces it, and a position redelivered after a broker restart would be out of date anyway. A Message Channel does not have to be persistent, and that one can be kept in memory, with no confirms: a crash costs a few seconds of a dot on a map. When the next message makes a lost one irrelevant, and nobody keeps the old ones as a record, persisting it buys nothing.

Who holds the copy?

In the Idempotent Processing Challenge, which is graded on duplicates doing no harm, this pattern is the at-least-once queue it requires: put the Guaranteed Delivery node between the Service and an Idempotent Consumer, on an edge labelled "redeliver until ack". Then go over each arrow and ask which box still holds a copy if the box it points to dies at that moment. An arrow where the answer is "none" is where the design can lose a message.

Practice this