Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Aggregator

A stateful step that collects related messages by a correlation id and emits one combined message when a completion condition, chosen before the first part arrives, says the group is done, including the group whose last part never comes.

Waiting for one more tote

At 15:40 the warehouse's packing dashboard shows a line that has not changed all afternoon: order 50716 held: 2 of 3 picks received, waiting since 09:14. The order was booked on the 14:00 truck. The truck has left.

The warehouse stores goods in zones, and an order whose items sit in several zones is picked in parallel, one pick per zone. Order 50716 has frozen peas from the chilled zone, a garden chair from the bulky zone and a pack of batteries from small parts. Each zone's picker fills a tote and scans it out, and the scan publishes a TotePicked message carrying the order number, the pick's number and the order's total, pick 2 of 3. The packing service listens to those messages and releases an order to a packing station when all of its totes are in, so it leaves as one parcel with one label.

The bulky-zone picker found the chair's shelf empty and moved on. No scan, no message. The packing service's rule was "wait until all picks are in", and nothing would ever make that true for 50716, so the order would wait until someone noticed, and its two totes with it.

A step that remembers

The packing service is an Aggregator, whether or not anyone designed it as one: a stateful filter, in the Pipes and Filters sense, that collects related messages, holds them until a completion condition says the set is done, and then emits one combined message. Most steps handle a message and forget it; an aggregator cannot, because what the third tote means depends on the two before it.

Three zones, one parcel

Each zone's tote message carries the order number. The aggregator releases the order when all three picks are in, or sends it to the short-pick desk at the deadline.

Which totes belong together

The order number is the correlation id this aggregator groups by: each arriving tote is filed under its order number, in a group of its own. Correlation Identifier explains what makes a good id. The pick number and the total were stamped on each pick task before the work split, the same values a Splitter puts on the parts it emits, so every message already says which group it belongs to and how big that group is.

Done means something you chose in advance

"Wait until all picks are in" is a completion condition, but it only covers the happy case. The condition worth running is decided before the first tote arrives, and it is one of these, or a combination:

  • A count: release when 3 picks are in. Exact, because the total travels on every message. On its own it waits forever for a pick that never comes, which is order 50716, and an Aggregator that waits forever leaks memory, or rows in its store: every group that will never complete stays open, a few more each day.
  • A timeout: release whatever has arrived by a deadline. It never waits forever, but on its own it also holds every complete order until the deadline.
  • Count or timeout, whichever comes first: an order leaves as soon as its last tote arrives, and an order missing a tote stops waiting at the deadline. This is the usual choice.
  • A rule over the contents: release when every item on the order is accounted for, picked or reported short. It fits when the number of messages is not known in advance but the expected contents are.

The warehouse now uses count or timeout, with the deadline taken from the truck: an order booked on the 14:00 truck stops waiting at 13:30, which leaves half an hour to pack or to act. It also has the scanner send a PickShort message when a shelf is empty, a third trigger that closes the group at once, so most incomplete orders reach the short-pick desk well before 13:30; the deadline remains for the pick that sends nothing at all.

What leaves at 13:30 with a tote missing

A timeout ends the wait. It does not decide what an incomplete group means, and that decision belongs to the business, made before the first deadline fires. There are two outputs worth having:

  • What has arrived, marked as partial. The packing station ships the peas and the batteries with a note that the chair follows, and the combined message says 2 of 3, pick 2 missing, so neither the packer nor the customer email treats it as the whole order.
  • A failure for the group instead of a result. The order goes to the short-pick desk, where someone finds the stock elsewhere or calls the customer.

The warehouse chose the second: it charges one delivery fee per order, and a chair sent on its own would travel at the warehouse's cost. Dropping the group when the timer fires is not on the list: the two totes are standing at a packing station, and someone has to know why.

Groups that survive a restart

Between the first tote and the last, each group is state: which picks have arrived, what the totes hold, when the deadline falls. Kept only in memory, it is gone after a deploy, and the last tote of 50716 would open a new group of one and wait for two picks that already came. So the groups live in a durable store, a table keyed by order number; the service acknowledges a tote message only after writing it there, and stores each group's deadline with it, because a timer held only in memory vanishes in the same restart, so a periodic sweep closes the groups past their deadline. With several instances of the service, two totes for one order can arrive at once and each overwrite the other's update, so every change to a group (adding a pick, finding it complete, closing it at the deadline) must go through the same coordination: one atomic operation on the group, or a single instance that handles an order's messages and its deadline one at a time, for example by partitioning the channel by order number. Closing a group and storing the message it will emit go in the same write, so a crash before the send means a resend after the restart, not a lost order, and whoever receives it has to cope with getting it twice.

Late totes and totes counted twice

At 13:52 the bulky-zone tote for 50716 is scanned after all: a picker found a chair on another shelf. Its group closed at 13:30, and treated as a first message the tote would open a new group that waits for picks that will never be scanned again. So the aggregator remembers closed groups for a while, the order number and how it closed, longer than a tote could plausibly arrive late, and sends a tote for a closed group to the short-pick desk, which is already handling that order.

Counting one tote twice is the opposite mistake. The broker delivers at least once, so a consumer that crashes after storing a tote and before acknowledging it receives the same pick 1 of 3 again, and an aggregator that counts messages would release an order whose chair is still on a shelf. Counting distinct pick numbers, the sequence numbers the Splitter stamped, makes the repeat harmless: the order completes when the set holds 1, 2 and 3. Making a whole consumer safe against repeated messages is a wider problem, which Idempotent Consumer takes on.

Order 50716 on the Grid

The Splitter & Aggregator challenge ends with the half this article is about: an Aggregator that collects related parts back into one result, with the correlation shown and the completion condition visible before it emits. Build it around order 50716: the sender, the order, the Splitter, two zone pickers as item processors, one of them standing for two zones (a third would exceed the Challenge's eight-node limit), the Aggregator, the packing station and the short-pick desk. Label each picker's edge with the order number and its pick numbers, and the Aggregator's two ways out with the condition that sends a group down each, such as all picks in, or 13:30 with a pick missing.

When the parts are replies from several services to one request, rather than pieces of one order, the same gathering sits inside Scatter-Gather, which adds the question of how long the caller can afford to wait.

Practice this