Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Resequencer

A stateful step that holds out-of-order messages by a sequence number assigned at the source and releases them in order: the next-expected counter, what the numbers must be, how long to wait for a missing one, what to do when the wait ends, and the message that arrives after its gap was skipped.

Two lines in the graphics log

  • 21:14:02.412 match m-3305 seq 88 VAR_DECISION goal overturned
  • 21:14:02.917 match m-3305 seq 87 GOAL home 1-0

Those two lines put a wrong score on air for forty seconds. The decision to overturn the goal reached the broadcaster's on-screen graphics first, found no goal to take back and was ignored. Half a second later the goal arrived, the score went to 1–0 for a goal that had not counted, and an operator had to fix it by hand.

The broadcaster buys its match events from a live-sports data provider: goals, cards, substitutions, VAR decisions, each numbered by the provider within its match. Before sending them, the provider runs events through a pool of workers that attach player details, and a goal, which picks up the scorer's season figures, took longer there than the short decision that followed it. Workers sharing one queue keep their own messages in order and lose it across each other, as the section "Each worker in order, the queue out of order" of Competing Consumers shows. The provider could keep each match in one ordered lane, which is what Sequential Convoy does, but the broadcaster does not run the provider's pipeline. What it controls is the receiving end.

Hold 88 until 87 is in

There, a stateful step can hold each event until everything numbered before it has arrived. For each match it keeps one number, the next expected, and a buffer of events that came early. With the next expected at 87:

  1. Event 88 arrives. It is not 87, so the resequencer puts it in match m-3305's buffer and releases nothing.
  2. Event 87 arrives. It is the next expected, so the resequencer releases it to the graphics and moves the next expected to 88.
  3. The resequencer finds 88 in the buffer, releases it and moves the next expected to 89.

The graphics now get the goal and then the overturn, in the right order; the overturn waited about half a second in the buffer. A step that buffers messages by sequence number and releases them in order is a Resequencer. The same rules deal with copies: an event numbered below the next expected has already been released, unless its gap was skipped (see below), so a copy of it is dropped, and the buffer holds one entry per number, so a copy of an event still waiting there is ignored too.

One match, back in order

Events of match m-3305 arrive out of order. The resequencer releases them by sequence number and sends a gap it gave up on, and any event that arrives after that, to the match desk.

What the numbers have to be

That loop works only when the numbers have three properties, and all three are the source's job. They are assigned at the source: the provider numbers each event when it records it, before any parallel step. A number stamped by the broadcaster on arrival would faithfully record the wrong order.

They are counted per stream key: each match has its own sequence. With one counter shared by every match, m-3305 would jump from 87 to 93 whenever five events of other matches took the numbers in between, and the resequencer could not tell those from missing events.

And they are contiguous and start at 1 in each match: every number is used, so a hole means something, and the next expected is never guessed from whichever event arrives first. When 96 arrives and 95 has not, an event is missing. Timestamps cannot say that. Two events can share a millisecond, the clocks of the provider's servers drift apart, and four seconds between two timestamps does not tell you whether anything happened in between.

One more condition sits on the receiving side: every event of a match has to reach the same resequencer instance, by routing on the match id, and that instance handles one match's events one at a time. Split one match across two instances and each one sees holes that are only the other's events.

When 95 never comes

In the 63rd minute of another match, event 96 arrives and 95 does not. The resequencer holds 96, then 97 and 98, and the screen shows nothing new while a yellow card and a substitution wait behind a number that may never come. With no limit, one missing message stalls everything after it, for good.

So a gap gets a deadline, started when the first event past it arrives. The broadcaster measured how long a missing event takes to arrive after the first event past it and found almost all within 600 milliseconds, so it waits 2 seconds. What happens when the deadline passes is something to decide before it happens, and there are two answers:

  • Skip and flag: move the next expected past 95, release 96 onwards up to the next gap, and mark the first released event with the gap, so the graphics and the match desk know something is missing. The broadcaster does this, and the desk checks the match on the provider's own feed page and corrects the screen if 95 mattered on air.
  • Hold and alert: keep waiting and page someone. That suits a stream where a skipped message makes everything after it wrong, such as balance changes applied in order. For live graphics, a frozen screen is the worse failure.

A deadline bounds the wait, not the memory. On a Saturday with dozens of matches live, each match's buffer also has a size limit, 200 events, and a match that reaches it treats its gap as timed out at once instead of growing.

95 turns up after all

Four seconds later, 95 arrives: a yellow card. The next expected is already 99. Releasing it as new would show a card from seconds ago as if it had just been given, after the substitution that followed it; rewinding to 95 would replay 96 to 98. The resequencer does neither. It keeps the numbers it skipped until the match ends, and 95 is one, so it goes to the match desk with its number, where someone decides whether the screen needs a correction. A number below the next expected that was never skipped is a copy, and is dropped.

The next expected numbers, the buffers and the skipped numbers are state. A restart that loses them waits for numbers already released, or drops a late 95 as a copy, so they go in a durable store, together with when each pending gap's deadline started. Saved right after each release, they still leave one window: after a crash between a release and the save, that event goes out again if the restart recovers it from the buffer or it is delivered again.

Two neighbours look alike from a distance. An Aggregator also buffers and times out, but it combines a group into one message, while a resequencer passes every message on, one at a time. Matching replies to requests needs no order at all, only a Correlation Identifier; and when order is defined by event time across many sources, with lateness handled by windows and watermarks instead of a contiguous counter, the subject is Stream processing: windows, watermarks & state.

Where to practise it

No Challenge asks for a Resequencer yet. The nearest is Splitter & Aggregator, whose palette includes one: there the parts are combined into one result, so the order they arrive in does not matter, which is the line between the two patterns. The resequencing itself is best sketched on a blank Grid: match m-3305, a 2-second gap deadline and a match desk for late events. Distributed Message Queue asks for ordering within a partition, order kept at the source so nothing downstream has to rebuild it.

Practice this