"Someone pressed the doorbell once. Why did my phone ring twice?" A smart-home company's support team had 212 tickets like that, all from owners of its video doorbell. Neither the doorbell nor the app was at fault: the notification service had received two doorbell.pressed events for one press, and sent a push for each.
A press goes from the doorbell to the home hub, from the hub to the company's cloud, onto a queue, and from there to the notification service, which pushes it to every phone on the household's account. The hub waits 3 seconds for the cloud to acknowledge each event and sends it again if no acknowledgement comes. On a slow home connection the first send gets through, its acknowledgement does not make it back in time, and the retry gets through too. Two events, one press. A sender that retries until it hears back is how an event survives a lost packet; Guaranteed Delivery explains why that trade produces copies.
The id the doorbell writes
The fix sits between the hub and the notification service: a filter that remembers the id of every message it lets through, for a window of time, and drops any later message carrying an id it still holds. That filter is Message Deduplication, and everything rests on the id: it must be assigned by the producer, once per fact, and carried unchanged by every copy of the message.
The doorbell produces the press, so the doorbell names it: db-7731:1840, its serial number and a press counter it keeps in flash memory. The hub resends the event exactly as stored, so the retry carries the original's id and the filter recognises it.
Ids that look usable fail in both directions. If the hub stamped a fresh id on each send, or the cloud assigned one on arrival, every copy would get its own and the filter would let each one through. And a counter that restarts at 1 after the doorbell reboots repeats old ids, so a real press a minute after a reboot can be dropped as a copy of one from before it.
The hub's retry carries the same id as the original. The filter records the id once and passes only the copy that recorded it.
On the broker, or a step of your own
The filter can live in two places. Some brokers offer it on the way in: an Amazon SQS FIFO queue drops a message whose deduplication id it accepted in the previous 5 minutes, and Azure Service Bus, with duplicate detection on, does the same by MessageId within a window you configure, 10 minutes by default. The other place is a step of your own in front of the handler, with its own store of seen ids: a key per id in Redis, or a row under a unique constraint, each with an expiry.
The two do not catch the same copies. A broker checks messages as they are sent, so it catches the producer sending one twice, like the hub's retry. A copy the broker makes itself, delivering a message again because the consumer's acknowledgement never arrived, is the same accepted message, and the broker's duplicate detection does not stop it. A step placed in front of the consumer sees both kinds.
Fourteen minutes late
The window has to be longer than the latest a copy can arrive after the first, and for the doorbell that is not 3 seconds.
On a Sunday evening a household's router drops out for 14 minutes. The hub had sent press 1840 just before, got no acknowledgement, and kept it in its buffer along with every event since. When the line comes back the hub sends the buffer, press 1840 included. A 5-minute window forgot 1840 nine minutes earlier, so the copy goes through, and two phones buzz for a visitor who left a quarter of an hour ago.
So the window comes from how late any copy can reach the filter, not from a default: the producer's retries plus the time a copy can spend in the queue or waiting to be redelivered. The hub keeps an unacknowledged event for up to an hour, so the company chose 75 minutes for its own filter, the hub's hour plus a margin for those delays, a window an SQS FIFO queue cannot provide: its 5 minutes is fixed. If the evening peak of about 300 presses a second lasted all 75 minutes, that would mean around 1.35 million short keys held at once. A copy later than the window still gets through, and the handler behind the filter has to cope with it.
Two copies in the same millisecond
In a filter of your own, the obvious code is a lookup and then a write: is 1840 in the store? No, so record it and pass the press on. The filter runs on six instances, and the hub's original and its retry can reach two of them a few milliseconds apart. Both look up 1840, both find nothing, both record it, both pass it on, and the phones buzz twice.
The check and the record have to be one operation that only one caller can win. Redis SET with the NX option writes the key only if it is absent and reports whether it did; an insert against a unique constraint fails for everyone after the first. The instance whose write failed drops its copy.
Keys that look like ids
Hashing the payload instead of reading an id is tempting, and SQS FIFO offers it as content-based deduplication over the message body. It fails both ways. If the event body says only which doorbell was pressed, a visitor who presses, waits and presses again 20 seconds later produces two identical bodies, and the second ring is dropped. If the hub adds its own sent_at time to each send, the retry's body differs from the original's, and no copy is caught.
A Correlation Identifier is not a dedup key either: related but different messages, such as every reply to one request, share it. If the doorbell tagged everything about one visit with a visit id, the press, the motion alert and the "clip ready" event would all carry it, and a filter keyed on it would pass the first and drop the other two.
What the filter never learns
The filter knows which ids it has seen, not what happened after it passed a message on. Suppose the step records 1840 and hands the press to a notifier instance that crashes before sending any push. The broker delivers 1840 again, the filter finds the id, and drops the only copy that could still have rung the phones. Record the id only after the notifier reports success instead, and a copy that arrives while the first is still being sent finds nothing and goes through.
A short claim, extended once the push succeeds, narrows both gaps without closing either, because the outcome the filter would need to know lives in the handler's own store.
A seen id alone cannot tell a push that finished from one that was interrupted, so the filter cannot be the safeguard, and when in doubt it should let a copy through: a press it drops after a crash is lost, and nothing behind it can bring it back. The filter reduces the copies, and the handler, recording the id it applied together with its effect, makes the one that gets through harmless; that half is Idempotent Consumer. A push to the phones, though, is a call to another system that no local transaction covers, and why that keeps a single effect hard to promise end to end is explained in Idempotency & exactly-once illusions. A key a client attaches to its own API requests, so a repeated request gets the first result back, is the Idempotency Key.
The filter in the Idempotent Processing Challenge
The Idempotent Processing Challenge asks for Message Deduplication as the broker-side companion defense, next to an at-least-once Queue and an Idempotent Consumer. Draw it between the two, in front of the consumer, and label its edge with what it keys on and for how long, such as "press id, 75 min": the producer's id, and a window taken from how late a copy can arrive, not from a default.