One message, one fate
A messaging system works in whole messages. A message is delivered whole, acknowledged whole, retried whole and, when it keeps failing, set aside whole. For a message that means one thing, that is exactly right. For a message that carries many things, it ties every one of them to the fate of the worst.
A payroll provider receives one file per employer for each pay run. A mid-sized employer's file holds 4,200 payments, one per employee: an amount, a bank account and a reference. The file arrives as one message on the payroll-files channel, and the first version of the payment service consumed it as one: open the file, send each payment to the bank in a loop, acknowledge the message when the loop ends.
Payment 3,118 goes to an account the bank reports as closed. The handler throws, the file is not acknowledged, and the broker delivers it again. The second attempt starts at payment 1, so unless the service kept a record of what it had already sent, 3,117 payments go to the bank a second time before the loop reaches the same closed account and fails the same way. Meanwhile the other instances of the payment service sit idle: a queue hands one message to one consumer at a time, so the whole file is one consumer's work, however many are running.
Cut along the work, not along the file
The payroll provider puts a Splitter in front of the payment service: a step that takes one composite message and emits one message per part, so each part can be processed independently. It reads the file once, publishes one payment message per line to a payments channel, and then acknowledges the file. From there on, the unit of work is a single payment: it is sent to the bank, acknowledged, retried or set aside on its own, and the payment service's instances take payments from the channel in parallel, which is Competing Consumers at work.
The splitter turns the file into 4,200 payment messages. A payment the bank refuses goes aside by itself, and the other 4,199 carry on.
Where to cut is a design decision, not a property of the file. Payroll cuts per payment because a payment is what succeeds or fails at the bank. A file of invoices might be cut per invoice rather than per invoice line, if the lines of one invoice only make sense together.
What a payment needs to carry
A payment cut out of the file loses everything the file said once, at the top: the employer, the pay run, the date the money must arrive, the account it is paid from. The payment sender handles one message and cannot go back to the file to look, so the splitter copies that context onto every part.
The splitter also stamps three values on every part, so that the parts can be treated as a set again later:
- a correlation id naming the message the part came from, the same on all 4,200 payments, such as
run-1031-acme; - a sequence number, the part's position in the file, from 1 to 4,200;
- the total count, 4,200, on every part.
Without them, nothing downstream can tell which payments belonged to the same file, or whether it has seen all of them. The payroll provider needs both answers, because the employer gets one report per pay run saying which payments went through and which did not. Building that report means collecting the parts back into one message with an Aggregator, and deciding when the set is complete is its problem, not the splitter's. Putting parts back in their original order, if a later step needs that, falls to a Resequencer.
Payment 3,118, by itself
The closed account now costs one payment. The payment sender tries payment 3,118, the bank refuses it, and the message is parked on a Dead Letter Channel, carrying the bank's refusal, while the sender moves on to the next payment. The other 4,199 are paid, each acknowledged as it goes. Because the failed message carries its correlation id and sequence number, the dead letter says exactly which line of which file failed, and payroll staff can correct the account and send that one payment again.
Splitting changes what finished means. Before, a file counted as processed or not. Now a pay run can end with 4,199 payments made and one not, and that is the true state of the world: the bank really did pay 4,199 people. The per-run report has to show it, so the refusal must reach that report, not only the dead letters. Splitting fits because these payments are independent. If parts had to succeed or fail together, a transfer and the fee that pays for it, splitting would give each its own fate, which is the opposite of what they need; undoing the parts that already happened needs a Compensating Action.
When the splitter itself stops halfway
The splitter publishes the payments, waits until the broker confirms it has stored every one, and only then acknowledges the file. If it crashes after publishing 2,000 of them, the broker delivers the file again, and those 2,000 are published a second time. Acknowledging the file first would trade that for something worse: a crash would lose the payments not yet published, with nothing left to bring them back. Unless the broker can commit the publishing and the acknowledgement as one transaction, duplicates are the price, and the receiving side pays it.
The ids the splitter stamped make that affordable. The correlation id and the sequence number together name one payment uniquely, and the splitter derives both from the file, so a republished payment carries the same pair: the payment sender can record each pair it has paid and skip a repeat instead of paying it twice. A payment sender built that way is an Idempotent Consumer, and keeping its record correct across crashes and instances is the hard part: a crash after the bank pays and before the pair is written lets the repeat pay again.
Payroll in the Challenge
The Splitter & Aggregator challenge asks for the whole round trip: a composite message from a sender, a Splitter that emits one message per item, processors that handle the parts independently, an Aggregator that collects them, and the correlation that ties them together. For the Splitter half, send the payroll file through it. Label the edge out of the Splitter with what every part carries, the file's id, its sequence number and the total count, and draw two item processors, so the diagram shows the parts being handled independently. Then test your design against payment 3,118: when one part fails, is there anything in your diagram that holds up the others?