Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Poison Pill

A message that is well-formed but makes its consumer fail every time it is handled, and how to recognise it, count its attempts somewhere a crash cannot erase, and divert it before it freezes everything queued behind it.

Partition 17

At 06:40 on a Monday, 187 trucks stopped moving on the dispatch map. Their drivers were on the road and their engines were reporting, but on screen every one of them stayed where it had been at 06:40. The other 8,800 trucks in the fleet kept moving, apart from a pause of a few seconds every few minutes that nobody had noticed yet.

The trucks send GPS positions and engine events to a Kafka topic partitioned by truck id. Each of the 48 partitions carries the events of about 190 trucks, in the order they were written. Kafka gives each partition to one instance of the map updater at a time, which reads it in order, one message after another, and the 187 frozen trucks all hash to partition 17.

At the head of partition 17 sat a trip_summary from truck 4471, which had just spent forty minutes in a valley with no signal. Its new firmware, version 5.8, batches every position recorded while offline into one event, and this one held 4,100 points. The message was valid JSON, with the right schema and a known event type. The map updater's route-smoothing step builds a table of distances from every point to every other, about 17 million entries here against 40,000 for the 200-point summaries older firmware sends. The container hit its memory limit and was killed. It restarted, read the same message, and was killed again.

That is a poison pill: a message that is well-formed but makes its handler fail every time it is handled, so trying again changes nothing. It is not the same as a message the consumer cannot read at all. A trip_summary that is not valid JSON, or has no truck id, never reaches the smoothing step and belongs on an Invalid Message Channel; this one parses, passes every check, and then fails. In concurrent programming the name also means a sentinel put on a queue on purpose to tell a worker to shut down; this article is about the accidental kind.

Same message, same failure

Most failures are not poison. At 05:10 the same morning, the database behind the map failed over to its replica, and for forty seconds every write timed out; a minute later every one of those messages went through. That is a transient failure: it depends on the moment, not on the message. A deterministic failure depends on the message: the same bytes, given to the same code, fail the same way at 06:40 or at noon.

The consumer has to tell them apart without knowing the cause. A message that fails alone, while the messages around it and in other partitions succeed, is a poison suspect; when everything fails at once, as at 05:10, suspect a shared cause first, such as a dependency, because diverting messages then could pour healthy traffic into a side channel. An exception your own code raises about this input points the same way, but some failures raise nothing: the process killed at 06:40 never did. How to space the attempts for failures that can pass is the subject of Retry.

A counter that dies with the process

On paper, the map updater was protected: its handler wrapped each message in a try/catch, counted failures per message in memory, and sent a message aside after five. The trip_summary went round like this instead:

  1. The instance that owns partition 17 reads the trip_summary at offset 88,412 and starts smoothing it.
  2. Memory runs out, and the operating system kills the process with a signal no code can catch. The catch block never runs and the counter is never incremented. The offset is never committed either, because the map updater commits only after a message is handled.
  3. Once Kafka notices the instance has gone, it hands partition 17 to another instance, which resumes where the last commit left it: offset 88,412, the next message to read.
  4. That instance reads the same message and dies the same way, its own counter still at zero.
  5. The first instance comes back with empty memory, and the cycle goes on.

Each crash also stalls the other partitions the dying instance was reading until Kafka hands them out again: the few-second pause the rest of the fleet showed. Competing Consumers describes the same walk of one message through a pool of workers.

The fix is to count somewhere that survives the failure being counted, and to count before handling, not after. Some brokers keep the count for you: Amazon SQS raises a message's ApproximateReceiveCount every time it is received, so a consumer killed halfway through still leaves the count one higher (the name warns that it is approximate, so treat it as a threshold, not an exact tally). A Kafka consumer group keeps no count per message (Kafka's newer share groups do, but they give up order within a partition), so a consumer in a group that needs one keeps it itself, in a small store outside the process, keyed by topic, partition and offset, and by consumer group if groups share the store. The map updater now records the attempt for offset 88,412, and waits for the store to confirm the write, before it starts smoothing, and checks the count first: on the third delivery, it does not try again. That is one extra write per message, the price of counting something that can kill the counter.

Counted before it is handled

The map updater records each delivery before handling it, so the count survives the crash it is counting. On the third delivery the trip summary goes aside and partition 17 moves again.

Do not wait for the fifth attempt

A partition is ordered, which is why one bad message hurts so much: the map updater cannot set the trip_summary aside and carry on without giving up that order, so the 190 trucks behind it wait. This is head-of-line blocking: the message at the front holds back everything behind it, including the events of trucks with nothing wrong with them. Keeping each truck's events in order while the rest of the fleet moves on is the problem Sequential Convoy takes up.

So every attempt spent on a message that will fail anyway is paid for by 190 trucks. The attempt cap of a Dead Letter Channel ends an endless loop, and five spaced attempts suit a failure that might pass. For a failure already recognisable as deterministic they are pure delay: waits of 1, 2, 4 and 8 seconds between five attempts hold the partition for at least 15 seconds, and with firmware 5.8 heading for 3,000 trucks, every truck coming back from a dead zone with a summary large enough to fail the same way would cost its partition at least those 15 seconds again.

Diverting early takes one of two forms, depending on what the consumer can know.

  • When the failure has a shape you can recognise, divert on the first one. The team put a check in front of the smoothing step: a trip_summary with more than 1,000 points raises a TripTooLarge error, which the handler treats as deterministic and sends aside at once, with no retry. The check also turns the crash into an ordinary error the handler can act on.
  • When it does not, keep the cap low. A crash carries no error type, and the delivery count records attempts started, not why they ended. But when the restarts land on this one message while the other partitions stay healthy, two attempts are enough to justify a conservative cap, so the third delivery goes straight aside.

The diverted message goes to the Dead Letter Channel, which owns what happens next: the reason recorded with it, who is alerted, and how it is replayed once the smoothing step can cope with 4,100 points.

Drawing truck 4471's trip summary

In the Dead Letter & Invalid Messages challenge, which Invalid Message Channel walks through, this article adds two things: the Poison Pill node for the trip_summary, on the path from the Handler to the Dead Letter Channel, and a label on that edge naming the rule that sends a message there, such as "3rd delivery, counted before handling". Then test the label with one question: if the map updater is killed halfway through the trip_summary, is that count still there afterwards? A count in the handler's memory fails; one kept by the broker, or written to a store before handling, passes.

Practice this