Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Dead Letter Channel

After a bounded number of attempts, move a message that cannot be delivered or processed to a separate channel, with the reason it failed, so the queue keeps moving and the message can be fixed and replayed.

Three things to do with a message that fails

Most consumers that read from a queue eventually take a message they cannot process. What it can do next comes down to three choices: try again, throw the message away, or put it somewhere. A utility's meter-reading pipeline shows where the first two lead.

Smart meters send a reading every 30 minutes. The readings are published to a queue partitioned by meter, so each meter's readings stay in order, and a billing-ingest service consumes them into the usage tables that bills are built from. Then a firmware update reaches 2,400 meters, and their readings start to include a register, R3, that billing ingest has no mapping for. The readings are well-formed. Ingest throws unknown register R3 on every one of them, about 115,000 a day.

  • Try again, forever. The consumer takes the reading, fails, and takes it again. On an ordered queue or partition, every reading behind it waits, including those of untouched meters that share the queue or partition. Consumer capacity goes to a message that will fail on every attempt, and a side effect the handler commits before the failure, such as a write to the usage table, is repeated on each one unless that write is idempotent.
  • Throw it away. Someone catches the exception, logs it and acknowledges the message so the queue moves. The queue does move. Three days later a billing analyst asks why 2,400 meters have shown no consumption since the update, and the only record of those readings is a log line per reading.

Bounded attempts, then a channel of its own

A Dead Letter Channel is the third choice: a separate channel where a message goes once the system has stopped trying to deliver or process it, instead of being lost or left blocking the queue. Each R3 reading gets a fixed number of attempts, say five, spaced out so that a brief database outage has time to pass. After the fifth failure the reading moves to the dead-letter channel and ingest takes the next message. Billing for every other meter carries on, and the R3 readings wait, intact, to be inspected, fixed and replayed.

Failed readings go aside

Ingest gives up after five attempts and moves the reading aside; an operator inspects the dead letters and replays them once the cause is fixed.

NoteThe Dead Letter Queue node in the async category implements this channel.

Retry decides how the attempts are spaced and which errors deserve another one at all. Five spaced attempts can ride out a passing fault; they do nothing for R3, which will fail until someone ships a mapping. Spotting a message that will always fail, and diverting it before its fifth attempt, is a separate decision, made in Poison Pill.

Most brokers do the moving for you: an Amazon SQS redrive policy moves a message once it has been received maxReceiveCount times without being deleted, RabbitMQ republishes a message rejected without requeueing to the dead-letter exchange configured for the queue (with none configured, it is dropped), and Azure Service Bus keeps a dead-letter subqueue on every queue and subscription. Kafka has no built-in dead-letter topic, so the consumer, or a framework such as Kafka Connect, writes the failing record to a topic you choose and moves on. A consumer that moves messages itself acknowledges the original only after the broker confirms that the dead-letter channel has stored the copy durably, so a crash between the two can leave a duplicate, not a lost message.

Other ways in, and one that stays out

Running out of attempts is the common way in, not the only one. A message can also be dead-lettered because it expired before anyone consumed it, or because it could not be delivered at all, for example to a queue that hit its length limit and is configured to divert the overflow. Expiry only leads here on a broker configured to divert expired messages; elsewhere an expired message is simply deleted.

A reading that is not valid JSON, or has no meter id, belongs somewhere else. Retrying will not make it parse, so it goes to an Invalid Message Channel, where a producer bug is not hidden among messages that a fix and a replay will clear.

A dead letter has to say why

A week later someone opens the dead-letter channel and finds hundreds of thousands of readings. Next to the original message, unchanged, a dead letter needs to carry the reason it was diverted (here, unknown register R3), how many attempts it had, the channel it came from and when it failed. The original channel is what lets a replay send it back to the right place. The reason is what lets an operator see one problem instead of reading the messages one at a time.

Some brokers record part of this themselves: RabbitMQ adds an x-death header with the queue, the reason and a count, and Azure Service Bus sets a dead-letter reason and description on the message. When the consumer moves the message itself, writing these fields is the consumer's job.

Nobody reads an unwatched queue

The R3 readings can still be lost with a dead-letter channel in place, just more slowly: messages sitting there are kept but not billed, and a channel nobody watches only postpones the analyst's question. It needs an owner, the team that is paged, usually the one that owns the consumer, and an alarm on its depth or on the age of its oldest message. One dead-lettered reading is worth a look during working hours; 115,000 in a day is an incident. Depending on the broker and its retention settings, a backlog nobody reviews is eventually deleted or simply keeps growing; either way, nobody bills it.

Sending them back

Billing ships a mapping for R3, and the readings can be replayed: moved from the dead-letter channel back to the channel they came from, to be consumed again. Replay is not free, and three things decide whether it goes well.

  • Fix the root cause first, or the replayed messages simply fail again. Replayed before the mapping ships, every reading takes its five attempts again and lands back where it started.
  • A message may have partly applied its side effects before it failed. Ingest might have written the reading to the usage table and then thrown while updating the daily total, so the consumer must be idempotent to process it a second time safely. Idempotent Consumer shows how.
  • Replayed messages arrive out of their original order: last week's readings land after this morning's. A meter reports a running total, and if ingest computes consumption as the difference from the last reading it received, an old reading arriving late produces negative usage. Ordering by the reading's own timestamp, not by arrival, avoids it, though a total already computed from a later reading still has to be corrected.

Some brokers have replay built in, such as SQS's redrive from a dead-letter queue back to its source; elsewhere it is a small tool that reads each dead letter, republishes it and only then deletes it. Replay issued as an operational command on a channel of its own, rather than as a script someone runs, is what a Control Bus provides.

Answering the analyst

The Dead Letter & Invalid Messages challenge is this article's two failure channels plus the way back out of one of them. Build it around two messages from the meters: an R3 reading that a mapping will fix, and a reading with no meter id that no consumer fix will. Each should end in its own channel, with the edge into it saying why, and the operator's replay should lead back to the main channel, drawn as a Message Channel because the Challenge offers no Queue node. The design is done when it answers the billing analyst from the first section: where the readings went, why they went there, and how they come back.

Practice this