Sixty-four points per pair of shoes
A sports retailer's loyalty programme gives one point per whole euro spent, credited when an order ships. The order service publishes an order.shipped event for each shipment, and the loyalty service takes them from a queue. For order o-88213, €64 of running shoes, the handler reads the event, adds a +64 row to the customer's points ledger, commits, and acknowledges the message so the broker can drop it.
The acknowledgement comes last on purpose: acknowledging first and crashing before the commit would lose the 64 points for good, for the reasons Guaranteed Delivery gives. The price is the opposite gap. If the handler commits and dies before its acknowledgement reaches the broker, the broker delivers the o-88213 event again, to this handler after a restart or to another instance, and the customer ends up with 128 points. With a broker that delivers at least once, that second copy is routine; why a broker's delivery alone cannot be exactly once is explained in Idempotency & exactly-once illusions.
The way out is to make the handler an Idempotent Consumer: running it twice on the same message leaves the ledger as running it once did. A handler that only sets a value, a tier to gold, is already there, unless a late copy can land after another write to the tier and undo it; adding 64 points is not, and making it so takes two decisions, where the handler records what it has applied and which id it records.
The id goes in with the points
The loyalty database gets a second table, processed_messages, whose one important column is the message id, under a unique constraint. The handler now writes both in one transaction: insert the id, insert the ledger row, commit, then acknowledge. When the copy of the o-88213 event arrives, its insert hits the unique constraint, the handler rolls back, acknowledges, and moves on. The ledger still says +64. Only that violation earns an acknowledgement: a rollback for any other reason, a lost connection or a timeout, leaves the message unacknowledged so it can be delivered again.
Both halves of that transaction matter. Record the id in a separate step after the points, and a crash between the two leaves points with no record, so the redelivery credits them again. Record it first, in its own transaction, and a crash between the two leaves an id that claims points nobody credited, so the redelivery is skipped and the customer never gets them. In one transaction, a crash keeps both or neither.
The unique constraint also catches a case a lookup would miss. With several instances taking messages from the same queue (Competing Consumers), a copy can reach a second instance while the first is still inside its transaction. A check for the id would find nothing in both; the database lets only one insert of that id commit.
The handler writes the processed id and the ledger row in one transaction and acknowledges only after it commits. A redelivered copy fails the unique constraint and changes nothing.
Which id: the message's or the shipment's
The table needs an id that is the same on every copy of a message and different for every message that should earn points. There are two candidates, and each fails in its own way.
The message id. The producer assigns one when it publishes, say evt_4f1b, and a broker redelivery carries the same one. Not every id on a delivery qualifies: a RabbitMQ delivery tag or an SQS receipt handle names one delivery, not the message, so the copy can arrive with a different one and walk past the table, and a delivery tag, numbered per channel, can even match an unrelated message's. The message id fails when the producer sends the same fact twice. If the order service publishes order.shipped, gets no confirmation from the broker in time and publishes again, and builds a new event for that retry, the new event gets a new id, evt_9a07. The loyalty service sees two different messages for one shipment.
A business key. The handler can dedupe on what the points are for instead: the shipment. Put a unique constraint on (order_id, shipment_id) on the ledger row itself, and no event for that shipment can credit it twice, whatever id each copy carries. A business key fails when it is coarser than the effect. Dedupe on order_id alone, and an order that ships in two parcels, €40 of shoes on Monday and €24 of socks on Wednesday, earns points for only the first parcel to be processed; the second event is a different fact that looks like a repeat.
Hashing the event's content is the same mistake, harder to spot. If the payload says only which customer earned how many points, two separate €64 orders by the same customer produce identical events, and the second purchase earns nothing.
So the producer decides which choice is safe. When it assigns the message id once per fact, for example a relay that resends the same stored row with the same id, the message id is enough. When it may re-create events, a business key that names exactly one effect is the safer record, and it lives on the ledger row, so it is kept exactly as long as the points are.
How long to remember
A table of message ids grows by a few million rows a month here, so it gets cleaned up, but an id can go only when no copy of its message can still arrive. A queue that keeps unacknowledged messages for 4 days can hand one over on day 4. A log such as Kafka keeps messages for its retention period, 7 days by default, and a consumer group reset to an older offset to recover from a bug reads them all again. Deleting ids after 2 days turns a replay from day 3 into double points.
Effects the transaction cannot hold
The processed-id row protects only what commits with it. The handler's other effects, a "You earned 64 points" email or a transfer to a partner airline's miles account, are calls to other systems, and why no ordering around the commit makes such a call safe is shown in Idempotency & exactly-once illusions. Two tools remain. A call that changes something on the other side carries a key the other side recognises, derived from the shipment, so a repeat gets the first result back (Idempotency Key). A message to publish, such as points.credited for the email service, is written as a row in the same transaction as the points and sent by a separate relay (Transactional Outbox); the email service then receives copies of its own to deduplicate, and its call to the mail provider is again an effect outside its transaction.
A broker-side filter, Message Deduplication, drops copies it has seen within a time window, but it cannot see what the handler committed, so it reduces the copies while the processed-id row makes the one that arrives harmless.
o-88213 event should end, in your drawing, in a rejected insert and an acknowledgement. The client side of the brief is covered in Idempotency Key.