£612.40 against £612.39
The first line in the comparison log, on the first morning, read Q-88213 legacy 612.40 new 612.39 MISMATCH. A motor insurer was replacing the engine that prices its policies, fifteen years of rules about drivers, cars, postcodes and claims history, with a rewrite. Brokers' websites ask it for about 40,000 quotes a day. Nobody could list every rule the old engine applied, so nobody could write the tests that would prove the new one applied the same rules. What the team could do was ask both.
In a Parallel Run, the legacy and the replacement both execute on the same real inputs, their results are compared, and the new one is trusted only once it has matched for long enough, across enough kinds of input. Only the legacy result is used: the new path runs in shadow mode, its side effects (writes, payments, e-mails) are suppressed or sandboxed, and its output is only compared. The broker was quoted £612.40. The missing penny was the team's problem, not the customer's.
Only the legacy engine's price goes back to the broker. The new engine gets a copy of each request off the customer's path, and the comparer records both answers.
A copy that must not touch anything
The new engine was written to do everything a quote does, and pricing a quote does more than compute a number. It writes the quote to the policy database, emails the customer a summary, and calls a credit bureau that charges for each lookup. In a shadow each of those is a defect: a second quote row with a different price, a second email, a second bill from the bureau. So each is cut off before the shadow runs:
- Its writes go to shadow tables of its own, never to the tables the legacy engine and the policy system use.
- Its outgoing email goes to a sink that records what would have been sent, which is itself worth comparing.
- It does not call the bureau. The legacy engine's bureau responses are recorded, and the shadow is handed the recorded answer with the request.
The last one is about more than cost. A credit score can change between two lookups, and the claims-history service can be updated between two reads, and then a mismatch says nothing about the code. The comparison means something only when both engines priced the same facts, so whatever the legacy side fetched from outside services is recorded and given to the shadow; reference tables each engine holds itself are not, and have to be kept in step another way.
The shadow must not hurt the live path either. The broker's request never waits on the copy: it goes out after the legacy engine has answered, carrying the bureau answer the legacy side recorded, and is dropped rather than queued when the shadow cannot take it. The comparison is recorded off the broker's path, and the shadow shares no servers, connection pools or database with the legacy engine. What the live path still pays is the cost of sending the copy, whether the new engine is slow, crashing or switched off. Copying live requests to a path whose responses are thrown away is often called shadow traffic, or mirroring, and running new code on real traffic while users still see the old behaviour is also called a dark launch.
Six percent
By the end of the first day, 6% of quotes did not match. A raw mismatch rate says how much disagrees, not why. Sorted by cause, the mismatches fell into four groups, and each called for a different decision:
- Rounding. The legacy engine rounded each factor to the penny as it went; the rewrite rounded once, at the end. Hence Q-88213. The team changed the rewrite to round the old way, because until someone decides otherwise, the legacy output is the specification.
- Data, not code. The legacy engine read a postcode risk table refreshed every month; the new one had a copy from March. The fix was a data feed, and no code changed.
- Bugs in the rewrite. The no-claims discount stopped growing at nine years instead of fifteen. These are what the run exists to find.
- Bugs in the legacy engine. Drivers whose birthday fell on the quote date were priced at their old age. The rewrite was right, and the business decided its answer would stand after the cut-over.
The last group becomes a known difference: a recorded, approved reason why the two may disagree, which the comparer applies before it counts a mismatch. Each one has to be narrow, matching that cause and nothing else: here drivers whose birthday is the quote date, and only by the amount the extra year of age explains. A broad rule such as "ignore differences under £1" would also hide the next real bug that happens to be small. The comparer also ignores what is meant to differ between any two runs: generated ids, timestamps, the order of fields in a response.
When the match rate is not enough
Two weeks in, the overall match rate after known differences was 99.97%, with every remaining mismatch explained, and cutting over was tempting. The total hid what had never been tried. The run had seen almost no quotes for drivers over 85, and none for cars imported from outside the country, because brokers send few of either. It had not yet crossed the 1st of the month, when a renewals batch reprices every policy due that month through a code path daytime quotes never take, and unless that batch goes through the splitter too, the run never sees it at all. Cut over as soon as outputs match, and a rare input path that only surfaces occasionally never gets exercised before the new engine is the one answering.
So the decision rests on coverage, not on the total: the match rate per segment (age band, vehicle origin, type of cover, new quote or renewal), a minimum number of requests seen in each, and at least one pass through each cycle the business has, such as a month-end or a change of tariff. The run also shows the new engine's latency and error rate under real traffic rather than a test's load, though without the bureau call the shadow skips.
After the cut-over the insurer kept comparing for a few more weeks the other way round, with the legacy engine as the shadow, so a surprise still had a known answer to compare against and an engine able to take the traffic back, as long as it could still read what the new engine had written.
What a parallel run cannot tell you
It compares, so it fits where the old and the new are supposed to give the same answer; a deliberate change of behaviour, a new pricing model say, would be one large known difference, and the run would check nothing about it. It never shows a customer the new output, so it says nothing about how customers react, which takes real users on the new version, as in a Canary Deployment. And it costs a second engine's compute, the plumbing that keeps the shadow's effects contained and people reading diffs, so it is usually kept for calculations where a wrong answer is expensive, or for each route a Strangler Fig migration is about to move.
The Progressive Delivery Challenge asks you to model Parallel Run comparing old and new outputs on the same requests, and to label which release risk each technique reduces. On your Grid, make the legacy Service the only one whose answer travels back to the Client, and give the new Service an edge that reads as a copy, not a route. For the risk, write the one it covers and the other three techniques do not: a rewrite that computes a different answer on real requests, caught before anyone acts on it.