A €4 coffee declined, a €1,900 laptop approved
The ticket reached the payments team on a Monday morning. Between 09:00 and 09:20 the fraud check had declined a run of small, ordinary card purchases and approved several large ones later reported as fraud. The fraud-scoring service's logs looked fine: every score was sensible for the payment it had scored. The scores had been handed to the wrong payments.
The authorisation service sends each card payment to fraud scoring as a message on a request channel, then waits up to 800 milliseconds for a score on its reply channel. Fraud scoring runs twelve instances, and some payments take longer to score than others, so replies come back in a different order from the requests. Authorisation matched them by arrival: the first reply went to the oldest payment still waiting. At night, with one payment in flight at a time, that was right. At 09:00, with hundreds in flight and one scoring instance stuck on a slow query, a quick reply for one payment was taken as the answer for another.
Putting the replies back in request order is what a Resequencer does, and that is a separate problem: matching does not need order at all. It needs each reply to say which request it answers.
A name for each question
A Correlation Identifier is a unique id the requester puts on each request and the replier copies, unchanged, onto its reply. Authorisation now creates an id for every payment, such as c-7f3c, sends it with the request and, when a reply arrives, reads the id off the reply instead of guessing which payment it answers. Where the reply should be sent is a separate field, the Return Address: one says which request a reply answers, the other where to deliver it.
On the requester's side the id is a key into a table of everything still waiting, the pending-request table. Each entry is stored under a correlation id and holds what authorisation needs to resume: the payment that is waiting and when its request was sent. One entry's life:
- Authorisation creates the id
c-7f3cfor payment 4471 and adds an entry under it to the pending table. - Authorisation sends the score request, carrying
c-7f3c, to the request channel. - A fraud-scoring instance scores the payment and sends its reply, with
c-7f3ccopied onto it, to authorisation's reply channel. - Authorisation reads
c-7f3cfrom the reply, finds the entry, removes it and resumes payment 4471 with that score.
The scorer copies the id onto its reply. Authorisation looks the id up in its pending table and resumes the payment stored there, whatever order replies arrive in.
The order of steps 1 and 2 matters. If the request went out before the entry was written, a fast reply could arrive while there was still nothing to match it to. And if several authorisation instances read one shared reply channel, a reply can reach an instance whose table never held its entry; giving each instance its own reply channel, named in the Return Address, avoids that.
Unique enough, for long enough
The id has one requirement: no two requests that are still waiting may share it. If the id is reused, too short or a constant, two pending requests collide and a reply gets matched to the wrong caller, which is the Monday-morning bug again by a longer route.
Waiting includes replies still on their way. Suppose authorisation numbered its requests with a counter that starts at 1 with the process. After a deploy, the new process sends request 1 while a reply to the old process's request 1 is still on the reply channel, and the old score resumes the new payment. A random id, such as a version 4 UUID per request, is, for all practical purposes, unique across instances and restarts, with nobody coordinating who hands out the next number.
Too late, or twice
An entry that never gets a reply cannot wait forever. The scoring instance may have crashed holding the request, or the reply may have been lost, and an entry never removed leaves a payment that never resumes and grows the table with every lost reply. So each entry has a deadline: after 800 milliseconds authorisation stops waiting, decides the payment with its fallback rules and removes the entry. How to pick the number is worked through in Timeout.
At 950 milliseconds the score for payment 4471 arrives anyway. Its entry is gone, the lookup finds nothing, and that is the right outcome: the payment has been decided, and the late score must not be attached to anything else. Authorisation drops the reply and counts it: a rising number of late replies says the timeout is too tight or scoring is slowing down. What it must not do is fail the message over an unknown id, so that the broker hands the same reply back again and again. RabbitMQ's request-reply tutorial makes the same choice and discards replies whose correlation id it does not recognise.
A reply can also arrive twice, for instance when a scoring instance crashes after replying and before acknowledging the request, and the broker hands the request to another instance. The first reply already removed the entry, so the second finds nothing and is dropped like a late one; removing the entry on the first match, as one atomic find-and-remove so that a concurrent reply or the timeout cannot claim it too, is what makes the copy harmless here, and what scoring twice means for the scorer is taken up in Idempotent Consumer.
Three ids that look alike
A payment's messages carry more than one id, and swapping them undoes all of the above.
- The message id names one message. The request and the reply each have their own, and a copy the broker redelivers normally keeps it, which is what lets duplicates be recognised.
- The correlation id names one pending request. The request and every reply to it carry the same one, so on its own it is not a deduplication key: legitimate replies share it too, and it cannot tell a redelivered copy from another reply; Message Deduplication usually keys on each message's own id. It is the requester's key, used to find its own entry. Usually the requester chooses it and the replier copies it; by a common JMS convention the replier instead copies the request's message id into the reply's
JMSCorrelationID, and the requester keys its table on the message id it sent. - The trace id names a whole flow. A trace id generated once at the edge follows the payment from the card terminal through authorisation, scoring and the ledger, and following that flow is Distributed tracing. Because it is the same on every message in the flow, it cannot tell apart two requests made in the same flow, such as two score requests sent for one payment. The correlation id itself must be unique per pending request, and it can travel alongside the trace id.
One request, several replies
The Scatter-Gather Challenge raises the count. One quote request goes out to several providers, so its entry expects several replies with the same correlation id and stays open until they are all in or the deadline passes, and since it no longer closes on the first reply, each reply also has to name its provider for a repeat to be recognised; the gathering itself is what Scatter-Gather and Aggregator describe. The Challenge's palette has no Correlation Identifier node, so carry the id in the labels: the outgoing requests and the returning replies share one quote id, such as "quote q-118", and the gather step's timeout gets the label the Challenge asks for. A reply to q-118 that comes in after that timeout meets the fate of authorisation's late scores: no entry is open for it, so it is dropped and counted, and the answer to the next quote never sees it.