Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Progressive delivery: blue-green, canary & rollback

Separate deploying a version from releasing it: expose the new version in steps that each have a way back, choose between blue-green, canary and feature flags, let the metrics trigger the rollback, and keep the database working for both versions while they run side by side.

Running is not released

Shipping new code is two events that most pipelines fire in the same command. To deploy is to get the new binary running somewhere in production. To release is to send users to it. Fused together, they make every release an all-or-nothing bet: whatever is wrong with the new code, every user is exposed to it as soon as the deploy finishes.

A fitness app's sync service lived like that for two years. It takes workouts from 2 million phones and stores them in a shared PostgreSQL database, and its release replaced all 40 instances with the new build in about four minutes. A bad release reached every phone within those four minutes, and the way back was another four-minute deploy of the old build, once somebody had noticed.

Progressive delivery pulls the two events apart and closes the gap in steps: the new version runs, a small share of users reach it, the next share follows only when the measurements say the last one went well, and every step has a way back. Getting the binary built and onto production stays the pipeline's job, as CI/CD pipelines: integrate, build once, promote explains; this article is about what happens between deployed and released.

A useful way to hold it in your head: a rollout is an experiment with a kill switch. The hypothesis is "v2 does not burn error budget faster than v1", tested on the smallest sample that could disprove it, and the rollout tool's job is to stop the experiment before the sample becomes everyone. That experiment asks whether v2 is broken, not whether it is better. Measuring better takes a random split held steady, a metric chosen in advance and a planned sample, the ground of A/B testing & experimentation.

Three ways to open the gap

The sync team has three changes coming, and they do not all want the same technique.

  • Blue-green. A move to a new language runtime touches every instance, and the team wants it running and tested in full before any phone reaches it. A second full environment takes all the traffic in one switch, and switching back is the rollback; it costs double capacity while both stand and tells you nothing from real users until everyone is on it. The switch and how long the way back stays open belong to Blue-Green Deployment.
  • Canary. A rewrite of how workouts recorded on two devices are merged is code the team does not fully trust. v2 gets 1% of traffic, then 5%, 25% and 100%, with a pause at each step while its metrics are compared with v1's, so a problem the metrics can see at 1% stops the rollout there instead of at 100%, though what v2 has written stays in the shared database (see below). The price is two versions running side by side for hours. Canary Deployment takes the steps, the comparison and the waits apart.
  • Feature flags. Training plans, a new screen, ship in a release with the feature switched off and are turned on per user or cohort with no deploy. A flag changes behaviour inside code that is already deployed, a canary moves traffic onto a new deployment, and teams use both: a canary limits the risk of the deploy, a flag the risk of the feature. See Feature flag services: targeting, rollouts & kill switches for the service and Feature Toggle for the switch in the code.

Two relatives answer different questions: Parallel Run checks whether v2 computes the same results without users depending on them, and Strangler Fig replaces a large old system route by route.

Canary at the balancer, judged by version, rolled back by the controller

The rollout controller owns the weights; humans set the policy, not the dial. Every metric carries the version, so v2 is compared with v1, not with the total.

A rollback nobody has to approve

Every step needs a verdict on whether v2 is worse, which takes metrics labelled by version and compared with v1's (how, and why the overall error rate misleads, is worked through in Canary Deployment). The verdict then has to act. In the diagram the rollout controller owns the weights: it reads the error-budget burn of v2's share, as SLOs & error budgets defines it, or a canary score from the comparison, and when the number crosses the line it sets v2's weight back to zero without a human. People decide the policy, meaning the steps, the thresholds and the waits; the controller turns the dial. A rollback that waits for someone to wake up, read the alert and agree is a rollback that can happen 20 minutes late, and for those minutes the canary's users keep getting failed syncs. The signals themselves come from the instrumentation in Observability: metrics, logs & alerting.

Both versions are live at once

The merge canary spends hours at its larger steps, and for those hours v1 and v2 read and write the same database at the same time. Blue-green does not escape this: switching back puts v1 in front of every row v2 wrote while it was live. So every schema change in a progressive rollout has to work with both versions, in the expand/contract order from Schema migrations without downtime: expand first (add the nullable column, the new table), deploy code that can read either shape, backfill, switch, and contract (drop the old column) only once no running version, and no version you might roll back to, needs it.

The sync team learned the other half of the rule from a rollback. A release added rowing, stored as a new value, rowing, in the activity_type column. It had reached 25%, with each account pinned to one version, when a latency regression in the merge code tripped the automated rollback, and v1 came back in front of 40,000 rowing workouts it had no case for. Its sync endpoint failed for every account that had logged one, so the rollback that fixed the latency started an incident of its own.

Watch outRollback restores the code, not the data. If v2 writes rows v1 cannot read, such as a new enum value or a field in an encoding v1 does not know, the rollback itself becomes an incident. Ship the tolerant read in one release and the new writes in the next, so the version you might roll back to already understands them.

Two more consequences. Do not run the destructive step of a migration in the same pipeline step as the deploy: if the old column is dropped as the rollout starts, the v1 instances still serving lose a column they read, and the way back can take a restore from backup. And a rollout with an irreversible migration that v1 cannot run against is not progressive, however slowly the dial turns.

Drawing the toolbox

The Progressive Delivery Challenge puts Blue-Green, Canary, Feature Toggles and Parallel Run on one Grid, and two of its requirements carry this article: show rollback as flipping traffic back, not redeploying, and label which release risk each technique reduces. For each rollback edge you draw, the label worth writing beside it is what the version it returns to will find in the database.

Practice this