Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Edge Workload Configuration

Publishing a configuration does not mean a fleet runs it. When devices are many, often offline and not all alike, the control plane has to track what each one should run and what it reports, target cohorts by model, firmware and region, let devices catch up after days away, and widen a rollout only while the reports stay good.

Published is not applied

A regional transit authority runs 4,200 fare validators, the card readers by the doors of its buses and trams, across two cities. In March the fare zones change: a ring of stops moves from zone 2 to zone 3. The configuration team publishes the new zone map on a Thursday. On Monday the head of operations asks how many validators are charging the new fares, and nobody can say. The configuration service shows the map as published. That is a fact about the service, not about the buses.

Servers in a datacenter that stay connected to one central store and run the same build pick up a published value within a refresh interval; that case is External Configuration Store. The validators break each of those assumptions: a bus spends the night in a depot on Wi-Fi, the day on a cellular link that drops in tunnels and in the hills, and now and then a week in the workshop, switched off. And they are not alike: two hardware models from two vendors, four firmware versions in service, two cities with different fare rules.

Edge Workload Configuration is the pattern for that subject: configuration for a large fleet of devices or edge sites that are often offline, on slow links, and not all the same. A central control plane decides what each device should run, the devices get there on their own schedule, and the design is about knowing, per device, how far each one has got.

One control plane, two cohorts

Operators set rules per cohort. On every check-in a validator sends the version it has applied and gets back the one it should run.

Two records for every validator

The first change is in what the control plane stores. A central store answers one question: what is the value? A fleet's control plane has to answer two: what should validator 3117 run, and what does validator 3117 say it runs? So it keeps two records per device. The desired configuration is what the operators intend for it. The reported configuration is what the device last said it had applied, and when it said it.

Each published configuration gets a version number that only goes up, and a report names the version applied rather than repeating the values. Comparing the two records per device sorts the fleet into three groups: applied, pending (the device has not checked in since the change, or has not finished applying it), and failed (it reported that it rejected the version, with a reason). The Monday question becomes a query over reports: 3,610 validators report the new map, 540 still report the old one, 50 have not been heard from since Wednesday. Without the failed group, a validator that refuses the new map looks exactly like one sitting in a tunnel.

Who starts the conversation matters too. A push from the control plane reaches only the devices online at that moment; a validator that misses it stays on stale settings, with no plan to catch up. So the device starts it: when it connects, and every few minutes while connected, it sends its reported version and receives its desired one if they differ. A push still helps as a hint to check in now, but convergence comes from the check-in, which every working device makes sooner or later.

Nine days in the workshop

Bus 214 comes back from nine days in the workshop, its validator off the whole time. Meanwhile the authority published version 42, the new zones, and version 43, which raised the number of trips a card can take while the validator is offline. The validator boots on version 41.

It does not need to replay 42 and then 43. The desired configuration is a complete state, not a list of edits, so a device that missed several versions fetches the latest one and is done. Had the authority sent changes instead, such as move stop 118 to zone 3, a device that missed one change would apply the next on top of the wrong base, and nothing in the edit would tell it so.

Until that first check-in, the validator charges fares on version 41, its last-known-good configuration: the last version it applied successfully, kept on its own storage, which a version it rejects or a half-finished download does not replace (External Configuration Store covers the same fallback for servers).

Running on stale values is a decision to make per setting, though. Charging old fare zones for a day is tolerable only if the authority records every tap and can reconcile the fares afterwards. For each setting, decide how old is too old, and what the device does past that age: fall back to a conservative behaviour defined in advance, rather than keep trusting a value nobody has confirmed for weeks.

The same file, read by two firmwares

Version 44 adds a new setting, a shorter card-read timeout to speed up boarding. Firmware 3.x understands it. Firmware 2.8, on 600 validators from the older vendor, rejects the whole file because it contains a key it does not know, and says so in its report. In the fleet, mixed firmware is the normal state, not a deploy in progress.

So a fleet does not have one desired configuration; it has rules that compute one per device, from properties the device reports about itself: model, firmware version, city, depot. Devices that share such properties form a cohort. The authority's rules now read: the new zone map for every validator in the northern city, version 44 for firmware 3.0 and later, a variant of it without the new key for firmware 2.8. Most settings come from a fleet default with a few cohort overrides on top; an override for a single device stays rare, because each one is a device that no longer matches its cohort. Layering and validation work as for any configuration (Configuration management).

Because cohorts come from reported properties, a device can change cohort without anyone touching its configuration. When a validator's firmware is updated from 2.8 to 3.1, its desired configuration changes with it, at its next check-in.

Widening while the reports stay good

Version 44 goes out in stages. First 40 validators in one depot, chosen to include both vendors and every firmware version, because a first stage of one model cannot reveal a problem in the other. Then 5% of the fleet, then 25%, then everyone. The control plane widens a stage only on evidence from that stage: applied reports from enough of its devices, and health reports that look like the baseline, such as the rate of failed card reads and the number of reboots.

Watch outSilence is not success. A stage where half the validators have not checked in has not passed; it has not been measured. Widen once enough devices have applied the version and stayed healthy under representative use for an observation window; time elapsed alone approves nothing.

When the reports go bad, the rollout halts on its own: no further stage starts, and an alert goes to a person. Rolling back means publishing version 43's settings again as version 45, since versions only go up, and making that the affected cohort's desired configuration; on a fleet that rollback is slow too, because it reaches each validator only at its next check-in. That is the reason to halt early and widen slowly; the general canary mechanics are in Progressive delivery: blue-green, canary & rollback.

Roll a build out in OTA Firmware Updates

The OTA Firmware Updates Challenge runs this loop with firmware images as the payload. Keep the desired and reported version per device on the config service, and let fleet metrics feed the alert that halts a stage. Then test your design against bus 214: a device offline for the whole rollout should still end up on the right version.

Practice this