One tap, two requests
Most duplicate writes are not caused by code that misbehaved. They come from code doing exactly the right thing: a client that heard nothing back, and asked again.
In a distribution centre, a picker takes six USB-C chargers from bin A-14-03 for order 88412 and taps Confirm on her handheld. The inventory service subtracts six, 18 down to 12, and replies, but the reply is lost while the handheld roams between Wi-Fi access points. Ten seconds later its HTTP client sends the same request again, and the service subtracts six more. The evening count finds 12 chargers on the shelf and 6 in the system, and since 6 is below the bin's reorder point of 10, an order for 200 more has already gone to the supplier.
A request lost on the way in and a reply lost on the way out look the same from the handheld: silence. Copies are the price of not losing work, and the job is to make them cost nothing.
Every layer can send it again
The handheld is one source of copies among several:
- The client. Anything that times out and tries again. Retrying at one layer, with backoff, keeps copies few (Retry), not one.
- A proxy or load balancer configured to retry can resend a request the first instance already handled.
- The queue. A broker that keeps the
StockPickedevent until the replenishment worker acknowledges it hands it out again when no acknowledgement arrives in time. - A crashed worker. One that applied the event and died before acknowledging looks, to the broker, like one that never started.
Subtract, set, or insert
The service ran UPDATE stock SET on_hand = on_hand - 6 WHERE bin = 'A-14-03'. Every run subtracts again. An operation is idempotent when running it twice, or ten times, leaves the system in the same state as running it once, and a subtraction is not.
Write the absolute value, SET on_hand = 12, and a copy is harmless, but only while nothing happens in between: if another picker takes four chargers (12 to 8) before the delayed copy lands, the copy puts 12 back. A set is idempotent against a repeat of itself, not against a repeat that arrives after someone else's write. Two shapes survive that:
- A conditional write. The handheld sends the bin version it saw when it scanned, 41, and the service writes
SET on_hand = on_hand - 6, version = 42 WHERE bin = 'A-14-03' AND version = 41. A late copy still carries 41, finds 42, and changes nothing, but the service cannot tell it from another pick that got there first, so zero rows changed calls for a re-read, not a success reply. - An insert with a unique key. Record the pick itself as a row whose key identifies this pick and nothing else, and adjust the stock in the same transaction. A second copy hits the unique constraint, the transaction rolls back, and the stock is untouched.
Order and line number would not make a key here, because large lines are split across totes, so the handheld creates a pick id, pk-90412, on Confirm, and every attempt carries it.
The pick row and the stock change commit together; the second copy finds pk-90412 recorded and gets the first one's result.
The id and the stock change, committed together
pk-90412, or a crash landing between the subtraction and the record. An atomic claim such as SET NX in a separate store does stop two copies running at once, but it does not commit with the subtraction, so it needs an in-progress/done state, and a crash between the subtraction and marking the key done leaves the claim in progress with the outcome unknown: recovering it means first finding out whether the stock already changed. Idempotency Key follows a claim kept in the same database and marked done in the transaction that commits the effect.A lookup inside the transaction is not enough at PostgreSQL's default isolation level unless a unique constraint backs it (Transactions & isolation levels). Recognising the copy is only half the job, because "duplicate key" is nothing the handheld can show the picker: a repeat is recognized and answered with the stored original result (accepted, 12 left) instead of being executed again. Doing that for API requests, including a copy that arrives while the first is still running, a key reused with a different body, and how long to keep keys, is the Idempotency Key pattern.
A shortcut that looks similar fails both ways: deduplicating on the request body instead of an explicit key means a retry whose body differs slightly isn't recognized (one stamped with a new sent_at, say), while two genuinely separate picks of six chargers from the same bin for the same order have identical bodies and are wrongly merged. Disabling Confirm after one tap does not help: the copy came from the HTTP client.
The same copy, through a queue
A broker that redelivers until acknowledged (Guaranteed Delivery) delivers at least once, so the worker that adds each StockPicked to the bin's weekly demand needs its own record of applied events, written with the update. That is the Idempotent Consumer, which also owns the worker that commits and crashes before acknowledging. A broker can drop some copies earlier (Message Deduplication), which thins them without replacing the worker's check.
Exactly once, as far as one commit reaches
With both defences, each pick and each event changes the warehouse's data once, provided the records outlive the copies. That combination, at-least-once delivery + idempotent processing, is the practical substitute for exactly-once delivery. Delivery alone cannot be exactly-once when a message or its acknowledgement can be lost: the sender can only choose between sending again and risking a duplicate, or not and risking a loss. When a product advertises exactly-once, it usually means a guarantee that holds inside its own boundary: Kafka's covers a job that reads from and writes to Kafka, provided the job commits its outputs and its read positions in one transaction and the readers downstream skip uncommitted data.
The processed-event record and the demand update commit together. The reorder call to the supplier is outside that commit and needs a guard of its own.
The substitute ends where the transaction ends. When an event shows the bin below its reorder point, the worker calls the supplier's API to order 200 chargers, and no rollback on its side takes that call back. Call first and record afterwards, and a crash in between brings the event back and orders 400. Record first and call afterwards, and a crash in between leaves the event marked done and nothing ordered. No ordering of a local record and a remote call rules out both, so effects outside your transaction need other tools:
- A key the other side honours. The worker writes reorder
RO-5531in its own transaction and sends that id on every attempt; if the supplier's API deduplicates on it, retrying until one attempt succeeds is safe. Relaying the reorder from a row, as the Transactional Outbox does, still sends at least once, so it relies on that key too. - Ask before acting again. A retry first asks the supplier whether
RO-5531exists, if it can look orders up by your reference; a first call still in flight can be missed, so this narrows the window without closing it. - Accept the duplicate. A second low-stock email to the buyer costs little; a second 200 chargers costs money. Decide per effect.
- Reconcile. A daily comparison of the supplier's orders with yours catches what slipped through.
Card payments are the sharpest such call; Payments & billing integration follows one through a provider.
One question per arrow
The Idempotent Processing Challenge is this warehouse in pattern form. Its design is finished when every arrow a copy can travel along answers one question: if this fires twice, what record makes the second a no-op, and which commit wrote it? The client's arrow, with a Retry on it so the duplicate is drawn rather than assumed, answers with an Idempotency Key; the queue's arrow with an Idempotent Consumer, backed by Message Deduplication on the broker. An arrow that leaves your system, like the reorder, has no node to answer for it, only one of the four tools above.