Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Invalid Message Channel

A separate channel where a consumer puts every message it cannot accept as valid (unparseable, wrong version, unknown type, missing field), kept with its raw bytes, the reason and its source, so the producer's owner can fix it while the main flow keeps moving.

40 buses, 31,000 taps

A city transit operator's fare service received 31,000 card taps it could not read in less than a day. All of them came from the same 40 of its 620 buses, about 5.6 per cent of all bus taps in that time.

Every metro gate and every bus validator sends one message per tap: the card number, the validator, the stop, the time, the fare product. The fare service reads them from one channel, works out each journey and charges the card. On Tuesday night, 40 buses received validator firmware 3.1, and from then on their taps carried the card number under cardId instead of card_id. The fare service needs a card number before it can do anything else, and it threw an exception on every one of those taps.

What happened next was decided by code nobody had read in years: the handler caught the exception, wrote a log line and acknowledged the message, so the channel kept moving and 31,000 journeys were never charged. Nobody noticed until a revenue analyst asked why 40 buses had carried almost no paying passengers. Retrying would not have helped either: the next attempt would have found the same cardId where card_id should be, as would the hundredth.

A check at the door

An Invalid Message Channel is where a consumer puts a message it cannot accept as valid at all, instead of dropping it or trying it again. The decision is made at the receiving endpoint, before any business logic runs: the fare service checks each message on the way in, and one that fails the check is copied to the invalid channel, acknowledged on the main channel only once the invalid channel confirms it has stored that copy, and the next tap is read.

Fails the check, parked apart

Taps that fail the check at the door go to their own channel, and the alert goes to the team that owns the validators. A valid tap that keeps failing goes to the dead letters instead.

What counts as invalid is set by what this consumer needs before it can start work. For the fare service, four kinds of message fail at the door:

  • Bytes that do not parse, such as a body cut off halfway by a bug in a validator's encoder.
  • A version it does not read: schema_version 4, when the fare service understands up to 3.
  • A type it does not know, such as a tap_reversal from newer gates, on a channel where the fare service is expected to handle every type it receives.
  • A required field that is missing or unusable, like Wednesday's missing card_id.

The checks at the door look only at the message itself. Whether a card has been blocked needs the account database, so that check belongs to the business logic, and a blocked card is a business outcome, not an invalid message.

A tap that is valid and still fails does not belong here either. If the card-account database is down, a perfectly good tap fails and will succeed later; that calls for retries and, once they run out, a Dead Letter Channel. If a valid tap for one rare fare product crashes the fare calculation every time, that is a Poison Pill. And a valid tap the fare service has no use for, such as one from a staff test card at the depot, is not invalid: dropping it on purpose is the job of a Message Filter.

Only a fix changes the answer

Because the checks at the door look only at the message, retrying a tap without card_id gives the same verdict every time. Taps already sent are recovered by changing the consumer, or by repairing them and sending them again; fixing the producer stops new ones. If the fare service is taught to read cardId, the 31,000 parked taps become valid, and sending them through again is a deliberate replay after the fix, which the Dead Letter Channel explains how to do safely, not a retry. Messages that parse and pass the checks yet still fail on every attempt are a different case, covered in Poison Pill.

What a parked tap carries

On Thursday morning, an engineer from the validator team opens the invalid channel to find out what went wrong. Three things make that quick, and the first is the easiest to lose.

  • The raw bytes, exactly as received. Not the parsed object, which may not exist, and not a cleaned-up version: a parser that quietly drops fields it does not know would have thrown away cardId, the one clue that points to a rename.
  • The reason, specific enough to group on: missing required field card_id, not validation failed. Thirty-one thousand messages with one reason can be investigated as one group.
  • The source, taken from the envelope rather than the body, with the channel it came from and when it arrived. When the body cannot be parsed, the validator id inside it cannot be read either, so the source has to come from what travels around the message: its headers, or the channel and connection it arrived on. Here each validator sends its id and firmware version as message headers, and all 31,000 taps say firmware 3.1.

Page the team that can fix it

The fare service's on-call engineer can see this problem but cannot fix its cause: the bug is in validator firmware, owned by another team and installed on 40 buses. An alarm that wakes the consumer's on-call at 02:00 produces a ticket forwarded in the morning; routed by the source on each message to the team that owns the producer, it reaches people who can roll back firmware 3.1. With the channel in place, the first invalid tap arrives minutes after the first updated bus picks up a passenger, and that alert can go out before the 40 buses have finished their first morning. The fare team still wants a count, since those taps are its revenue, but not a 02:00 page.

A check stricter than the consumer needs fills the channel with false alarms. Suppose firmware 3.1 had kept card_id and only added a new field, occupancy. A fare service that rejects any field it does not recognise would have parked all 31,000 taps anyway, though it could have charged every one. Check what the consumer needs and tolerate what it does not use. Which changes a producer can make without breaking its readers, and how a schema registry can catch the rest before they ship, is covered in Schema evolution: event formats & compatibility.

Validity is each consumer's own judgement: a tap the fare service rejects may be perfectly usable by the ridership-statistics service, which never reads the card number, so one invalid channel per consumer is the natural fit. A shared one can work if every entry records which consumer rejected it, so alerts can still be routed per producer.

Two channels, told apart

The Dead Letter & Invalid Messages challenge asks for both failure channels and for the difference between them to be explicit. Draw a Client for the validators, a Message Channel for the taps and a Handler for the fare service, then two edges out of the Handler: one to the Invalid Message Channel, labelled with the check that failed, such as "missing card_id", and one to the Dead Letter Channel, labelled with what ran out, such as "retries exhausted". Give each channel its own reader: the operator Actor the Challenge asks for, who inspects the dead letters and replays them to the Message Channel, and a second Actor for the validator team, alerted by the invalid channel. Labelled that way, the two channels say which messages are retried and who hears about each, and they cannot be mistaken for one another.

Practice this