Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Claim-Check

A payload too large or too sensitive for the channel waits in a store, and the message carries only a reference that each receiver redeems. The hard questions then move to the store: which write comes first, who may read it, and how long it keeps each payload.

38 megabytes on a one-megabyte channel

When an applicant presses Submit on a mortgage lender's portal, the portal publishes one ApplicationSubmitted message and two services act on it: underwriting, which checks income against the loan, and fraud check, which compares the identity document with the details the applicant typed. The publish for application 58114 came back with this:

publish rejected: message of 38412906 bytes exceeds the channel limit of 1048576 bytes

The applicant had attached what the lender asks for: three months of payslips, six months of bank statements and a passport scan, roughly 29 MB of scanned PDFs. The portal had put all of it in the message, base64-encoded next to the loan amount and the applicant's name, and base64 makes binary data about a third larger.

The last number is this broker's configured ceiling. Brokers cap message size, many at a limit measured in kilobytes or a few megabytes, because they hold each message in memory or on disk, often copy it to replicas, and hand it whole to every consumer on every delivery.

Why raising the limit fixes the wrong thing

On many brokers the limit can be raised and the error would go away. Then the 38 MB is written to the broker's storage, replicated, delivered to underwriting and again to fraud check, and moved once more on every redelivery, while the messages behind it wait. And the passport scan and bank statements would live in the broker for as long as it keeps messages, open to anyone who can read the channel, from an operator browsing the queue to a Wire Tap copying traffic for inspection.

Leave the documents at the counter

The fix keeps the documents out of the channel altogether. The portal stores each document in a document store, an object store, under a key of its own, and only then publishes a small message that carries the keys instead of the bytes. That move is the Claim-Check pattern. The keys are the claim check, named after the ticket a cloakroom hands you for your coat, and a consumer that needs a document redeems the check: it reads the document from the store by its key.

Keys on the channel, scans in the store

The documents go into the store before the message is published. Each consumer fetches only the documents it reads, by key.

Application 58114's message is now a few kilobytes: the application id, the applicant, the loan amount and term, and one entry per document with its type, key, size and a checksum. One key per document lets fraud check open only the passport and underwriting only the payslips and statements. The fields consumers decide on stay in the message, so a step that sends large loans to senior underwriters reads the amount without touching the store. And a key is never reused: a replaced payslip gets a new key and a new message, because a consumer that redeems an overwritten key late reads a document the message never described. The checksum lets it confirm it got the bytes the message meant.

Two writes, and the crash between them

Checking in is two writes to two systems, the store and the broker, with no transaction spanning both, so the order decides what a failure leaves behind. The portal does it this way:

  1. The portal writes each document to the store and waits until the store confirms the write.
  2. The portal publishes ApplicationSubmitted with the keys.
  3. The broker confirms the publish, and the portal tells the applicant the application was received.

A failure between the first step and the second, a crash or a publish the broker rejects, leaves documents that no message points to. These orphans are somebody's passport and bank statements, a liability as well as a storage cost. A periodic sweep deletes the documents of applications never submitted, after a grace period longer than the portal's own retries. A publish whose confirmation was lost may still have gone out, so the portal retries it with the same application id, and the sweep must treat a missing confirmation as unknown, not as proof of an orphan.

The opposite order is worse. Publish first, and a consumer can receive keys that resolve to nothing yet; if the portal crashes before storing, they never will, and an application sits stuck with two services retrying it.

When the key opens nothing

A claim check can fail in two ways that look alike to the consumer and are not. When the store times out or answers with a server error, the document is most likely still there and it is the store that is unavailable, so retrying later is the right move. When the store answers that the key does not exist, waiting will not bring the document back, beyond a brief retry for a store that can serve a read slightly behind a write it just confirmed. That message has to leave the queue with the key and the store's answer attached; how it leaves, and who reads it afterwards, is what Dead Letter Channel explains.

Who holds the documents, and for how long

The access question moves with the data. The portal may write documents, underwriting and fraud check may read them, and nobody else may do either, so a key seen by the queue browser or the wire tap opens nothing. A reference that is a presigned URL behaves differently: anyone holding the message can open the document until the link expires, so the channel's readers become the documents' readers again. What such a link pins, and for how long, is explained in Valet Key.

Deleting is where the cloakroom stops being a good picture. A cloakroom gets its ticket back and returns the coat once. Here the claim check is never handed back: two consumers redeem it on their own schedules and nothing tells the portal when both are done, so deletion follows a rule set in advance; a flow whose consumers report completion could delete on those reports instead, once no redelivery can still need the documents. For the lender the documents belong to the loan file and are kept as long as its retention rules require. Where a payload exists only to cross the channel, its time to live has to outlast everything that can still deliver the message: the deepest backlog, the retries, a dead-lettered message replayed a week later. Lifecycle rules that delete on a schedule, and the rest of the store's design, are in Object storage & blobs.

Brokers redeliver, so the same reference can reach underwriting twice, the second time perhaps after its documents were deleted; recognising the repeat before redeeming it is the job of an Idempotent Consumer.

What the small message costs

Every consumer now makes one more read per document it opens, and the store has become a dependency of every consumer: when it is down, the broker can be healthy and applications still stop moving. A fraction of a second is negligible next to underwriting a mortgage, but on a channel carrying thousands of small messages a second a store round trip for each adds latency and load while saving little, since those messages already fit, so the pattern usually pays off for payloads that are large or sensitive. Some producers check in only above a size threshold, and their consumers then handle two shapes of message. A consumer that cannot reach the store at all, such as a partner outside the lender's network, can have a Content Enricher step redeem the check for it, which brings the large message back for that one hop.

Application 58114 on a Grid

The Claim-Check challenge asks for this path with one sender and one receiver, and for the keep-messages-small intent made clear. Draw the loan portal as the client, a Claim-Check step, a data store for the documents, a queue for applications and underwriting as the receiver; fraud check is optional, on a queue of its own, since two handlers on one queue would split the applications between them. Put sizes on the edges, tens of megabytes into the store and a few kilobytes onto the queue, and label the receiver's edge to the store with the documents it fetches. Then read your diagram in the order the portal acts, and check that the store's edge comes before the queue's.

Practice this