One delivery scan, three operations
A parcel carrier's 1,200 couriers collect cash on delivery for small online shops. When a courier scans a parcel as delivered, the handheld puts the scan on a queue, deliveries.in. A settlement router, the kind of endpoint Service Activator describes, takes each scan and does three things with the broker: it sends an instruction to cod.settlement to pay the shop the €42 collected, sends a text request to recipient.sms ("Your parcel was left with your neighbour at no. 14"), and acknowledges the scan so the broker removes it. If the router dies between any two of those, what is left on the broker?
With the client library's usual settings, each operation completes on its own, so each gap has its own answer:
- Dies after the settlement instruction, before the acknowledgement. The broker delivers the scan again, to this router after a restart or to another instance, which sends a second instruction. The shop is paid €84 for a €42 parcel, and the carrier finds out at month-end reconciliation.
- Acknowledges first, to avoid that. Now a crash between the acknowledgement and the sends loses the scan: the broker has removed it, and the shop is never paid.
- Dies between the two sends. If the scan was already acknowledged, the shop is paid but the recipient never gets the text; if not, the redelivery sends the settlement instruction a second time.
Reordering only chooses which half-finished state a crash leaves. When the work happens elsewhere, acknowledging last and accepting the redelivery is the safe default, for the reasons Guaranteed Delivery gives. Here all three operations are on the same broker, so there is a better option: ask the broker to treat them as one.
Ask the broker to hold all three
That option is a Transactional Client: the router opens a transaction in the messaging system itself and does the receive and both sends inside it. Until the router commits, the broker stores the two new messages but no receiver can see them, and the scan stays with the router instead of being removed. On commit, both messages become visible and the scan is removed, together. On a rollback, or when the router's connection drops before a commit, the two sends are discarded and the scan becomes available for delivery again.
In JMS this is a transacted session, created with Session.SESSION_TRANSACTED. Every receive and send on it since the last commit belongs to the current transaction; there is no begin call, because the next transaction starts when the previous one ends. The router receives the scan, sends the two messages and calls session.commit(), or session.rollback() when anything throws. It works the same with a polling loop or a listener, the choice Message Endpoint weighs.
The rollback is what makes the retry clean: the redelivered scan meets a broker with no instruction and no text from the failed attempt, so once an attempt commits, the queues hold one of each. That claim holds inside the broker, for the messages and acknowledgements that were part of the transaction. A scan that can never succeed, with a barcode the router cannot parse, rolls back and comes back on every attempt, until a redelivery limit, where one is configured, moves it to a Dead Letter Channel.
Kafka: the offset rides along
A Kafka consumer does not remove a message; it commits an offset, its position in the partition. Kafka's transactions put that commit inside the producer's transaction: a producer configured with a transactional.id calls beginTransaction(), writes its output records (to any number of topics and partitions), adds the consumer's next offsets with sendOffsetsToTransaction() (with enable.auto.commit=false and no manual commits, so the offsets the group resumes from are committed only through the transaction), and calls commitTransaction(), or abortTransaction() on failure. Outputs and position commit together.
Aborted records stay in the log, marked as aborted, and whether a reader sees them is the reader's setting: a consumer with isolation.level=read_committed skips them, while the default, read_uncommitted, returns everything, including the settlement instruction from an attempt that rolled back. The name is borrowed from Transactions & isolation levels, but here it decides only whether a consumer sees records from aborted or still-open transactions, and every reader that acts on the topic, the payment service included, must set it.
The transactional.id also guards against a router that froze rather than died and wakes up after a replacement took over: when a producer starts with the same transactional.id and calls initTransactions(), the broker rejects the older producer's later writes and commits.
The record the broker cannot see
Six months in, the shops' contracts require a proof-of-delivery record for every settled parcel, kept in the router's PostgreSQL database for disputes. The obvious change puts an INSERT before session.commit(), but that commit reaches only what the broker holds; the database has a transaction of its own. Commit the database first, and a crash before the broker commit leaves a record for a scan that is then delivered and processed again, giving a second record, or a unique-key error on every redelivery. Commit the broker first, and a crash leaves the shop paid and no record.
That is the dual write, and the broker's transaction does not remove it. Transactional Outbox does, by writing the outgoing messages as rows in the same database transaction as the record; making database and broker commit together through XA, where the broker supports it, is the ground of Two-phase commit. The outbox does not cover the input: a scan redelivered after the database commit is processed again unless the router recognises it, as an Idempotent Consumer does. The SMS worker meets the same boundary: its HTTP call to the SMS provider cannot be rolled back, so a worker that sends the text and then rolls back sends it again on the redelivery.
The receive and both sends commit or roll back together, inside the broker. The record insert has its own database transaction, which this commit does not include.
When the answer to commit is lost
If the connection drops after the broker commits but before the router hears so, a scan from deliveries.in is safe, because the broker holds all three operations or none of them. Partner couriers that report by HTTP are not: their scans arrive in a transaction with no receive in it, a lost commit answer turns into an error, the partner retries, and a second transaction sends a second settlement instruction, which the payment service must recognise by parcel id with an Idempotent Consumer (why exactly-once ends at the edge of one system is the question Idempotency & exactly-once illusions answers).
What each commit costs
A commit is a round trip to the broker, and for persistent messages the broker writes to disk or to replicas before it answers. At 40 scans a minute that is barely noticeable; at the evening peak of around 10 a second, it is added to every one. Committing 20 scans per transaction spreads the cost, but one bad scan then rolls back the other 19.
A transaction held open hurts in another way. If the router checks the shop's account with a billing system that takes 4 seconds, inside the transaction, the scan and its messages wait for those 4 seconds. In Kafka it spreads further: a read_committed consumer cannot read past the first record of a transaction still open in its partition, so every payment worker reading that partition waits behind the slowest router. Do the slow work before the first send where the client allows it (in Kafka, beginTransaction() can come after the lookup), and keep the time from the first send to the commit short.
Drawing the boundary
The Transactional Outbox & Inbox Challenge is where to practise placing a transaction boundary around a send and a receive. Its design keeps each transaction inside one database, the delivery-record problem above solved the other way, so there is no Transactional Client node to place. Model the receiving side instead: a Service that records each incoming message in the Transactional Inbox and applies its effect in one local transaction, and acknowledges to the Message Broker only after that commit. Label the edges inside that transaction "same tx" and the acknowledgement "after commit". Leaving out the Two-Phase Commit node the palette offers is how the design meets its last requirement, avoiding distributed transactions.