Control Bus

Eleven processes, three ways in

A hotel channel manager keeps room rates and availability in step between 140 hotels and three booking partners, Tripwell, Staylane and Roomfinch (the names are made up). Every change a hotel makes lands on one publish-subscribe channel, rate-updates, and each partner has its own adapter, a process that reads every update from its subscription and turns it into calls to that partner's API; Channel Adapter is the pattern behind it. With replicas, importers and a cleanup job, that is eleven processes on three hosts.

One evening Staylane's API starts answering 503 to everything, and within the hour 1,800 rejected updates are parked on a Dead Letter Channel. The engineer on call wants to stop the Staylane adapter, leave the other two alone, and later have Staylane catch up. What she has is three adapters built in three different years: one has an HTTP admin port, one reads a flag file only when it starts, and the third can be told nothing short of a redeploy, twenty minutes away.

Use the channels they already read

A Control Bus builds on the one thing all of those processes already do reliably: read messages from a channel. Next to the channels that carry rate updates, the integration gets one or more channels that carry messages about the integration itself, and every component also listens for the control messages addressed to it. Pausing Staylane becomes publishing one message, a pause addressed to adapter.staylane, from the operators' console. The adapter stops taking updates from its subscription until a resume arrives. No admin port, no per-component API, no host to log in to. A control message for one component or for all of them is the same point-to-point or publish-subscribe choice as for any Message Channel, with one catch: if the Staylane adapter runs as several replicas, each needs its own copy of the pause, because a queue they share hands it to only one.

While the adapter is paused, Staylane's updates wait in its subscription instead of failing against a dead API, for as long as the broker's retention allows. A pause moves the backlog to a safer place; it does not make it smaller.

Commands, questions and news

Pause and resume are commands: they ask a component to change what it is doing. Reroute and replay are others: send Staylane's traffic to a backup endpoint the partner provides, or send the 1,800 dead letters again once it recovers. Queries expect an answer: "how many updates have you sent, failed and still queued?" goes out once, each adapter replies on a reply channel the query names, and the console shows the three side by side. Events are published without being asked: adapter started, adapter paused, Staylane calls failing above 50%. They tell the console that the pause took effect, one event from each replica, instead of leaving it to assume so.

Why the pause cannot ride with the rates

It is tempting to skip the new channel and put the pause on rate-updates itself, as a special message the adapter recognises. At the moment the pause is needed, though, Staylane's subscription holds 200,000 updates, and the adapter, retrying each one against a failing API, gets through about one every two seconds. A pause queued behind them arrives days later, long after the damage it was meant to limit.

So control traffic gets a path that data cannot clog: its own channel, read by its own consumer thread inside the component, so that an adapter deep in a retry loop still hears the command. Message priority, where the broker supports it, lets the pause overtake the updates still waiting, but it does not reorder updates already delivered to the adapter, and how long those delay the pause depends on how the adapter processes them, which is as far as a Priority Queue can help. A separate channel with its own reader receives the pause however long the data backlog is, but the pause takes effect only when the adapter checks it before its next call to Staylane: a call already in flight finishes, and what happens to updates it has already fetched depends on how the adapter is written.

A replay that runs twice

Control messages travel over the same messaging system as everything else, so they inherit its delivery rules, and under at-least-once delivery a command can arrive twice. For pause that costs nothing, since pausing a paused adapter changes nothing. Replay is not: run twice, it sends all 1,800 updates a second time. Each command therefore carries an id, and the component records the ids it has carried out and ignores a repeat; keeping that record straight is the whole subject of Idempotent Consumer. Commands also carry a version of their format, so an adapter still on last month's release rejects a command it does not understand, and says so in an event, instead of guessing. The replay's own hazards, what to fix first and in what order to resend, stay with the Dead Letter Channel.

Who may pause a partner

One message on the bus can pause every partner at once, so treat the Control Bus as the administration interface it is. Only the operators' console and a few trusted automations get credentials to publish commands (components may publish only their own replies and events), each command carries a sender identity the component can verify, and the component checks that this sender may issue this command, so that someone allowed to read statistics cannot order a replay (modelling those rights is the subject of Authorization: roles, relationships & policies). Every command and its outcome also go into an audit trail, kept as Audit logs & tamper evidence describes, because "who paused Staylane on Tuesday?" will be asked.

The pause that a restart forgets

In this design the pause lives in the adapter's memory. If the Staylane adapter restarts at two in the morning, it reads its configuration, which still says Staylane is on, and starts failing again. The bus changes a running component; a setting that must survive a restart goes where the adapter looks when it starts, an External Configuration Store.

The bus cannot reach a component that has stopped reading, either. An adapter whose process has hung, or whose control listener waits on the same lock as its work, ignores its control channel exactly as it ignores its data. Whether a process is alive is checked from outside it, by Health Endpoint Monitoring; the bus is for components that are alive and need telling.

Apache Camel's controlbus component and Spring Integration's control bus do this inside one application; across separately deployed components like these eleven, the bus is a convention you build: channel names, a command format and a listener in each component.

Replay, sent as a command

Dead Letter & Invalid Messages stops one step short of this article: its last requirement is an operator path to inspect and replay dead-lettered messages, and it offers no Control Bus node to draw that path with. The requirement is still the place to try the idea, using an Actor, a Message Channel and a Handler. Two questions from this article decide whether the drawing holds: does the replay command travel on a channel of its own, or wait behind the messages that failed, and does the edge that carries it say the command has an id that the replayer checks, so that a second copy changes nothing?