Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Blue-Green Deployment

Run the next release in a second, complete production environment, test it there while nobody uses it, then move all traffic with one switch and switch back to roll back: where the switch lives, what happens to requests and sessions already on the old side, the database both sides share, and how long the way back stays open.

Why undoing a release is slow

On many teams, undoing a release means deploying the previous one, and that takes about as long as the release did. A city's journey planner found this out with release 42, which a rolling deploy spread across the instances behind its load balancer a few at a time. Before the rollout had finished, riders began reporting that release 42 planned trips through a station closed for works, and putting release 41 back meant a second rolling deploy, with apps getting answers from either release until it was done.

The cure is to keep the previous release running. A Blue-Green Deployment has two complete production environments. Blue serves every request with release 41; green gets release 42 while no rider is using it. When green is ready, one change sends all traffic to it, and rolling back is the same change in reverse, because release 41 is still running on blue, untouched.

Blue serves, green waits

Blue serves every request; green runs release 42 against the same database and receives only the smoke tests. Going live, or going back, is one change at the load balancer.

Before any rider reaches green

Green being idle is what makes it testable. Before the switch the team reaches it through a route the public never takes, a private hostname or a header the load balancer sends to green, and runs the smoke tests: a short set of real requests through the paths that matter most. Plan a trip across two lines, open a stop's live departures, save a trip. They run against green's real production dependencies, the timetable database and the vehicle-position feed, so they use test accounts, because whatever they write lands in the database riders use.

Watch outSwitch before green has passed its smoke tests and you find out it is broken only once every rider is on it. The way back is fast, but every request in between got release 42's answer.

Passing the tests is not the same as being ready for the load: a freshly started green has empty caches and connection pools, and handed the whole morning peak at once it can answer slowly for a minute. Replaying a sample of recent read requests against it first means the first rider it serves is not its first request.

Which switch you flip

The switch belongs where a change takes effect quickly and in one place. At a load balancer or a reverse proxy it is a configuration change: point the listener at green's pool of instances instead of blue's, or move a route, and new requests go to green within seconds. Load balancing & horizontal scaling explains how the balancer spreads requests across the instances inside each pool.

DNS looks like a simpler switch, one record changed from blue's address to green's, and it is a slow one. Resolvers and clients cache the answer for the record's TTL, and some resolvers and client runtimes keep it longer than that, so traffic leaves blue gradually over minutes or hours instead of at once, and a rollback through DNS drains back just as slowly. A team that switches by DNS keeps blue fully serving until the last stragglers have gone, and does not count on the way back being quick.

Riders caught mid-request

The switch changes where new requests go. Requests already inside blue finish there, as long as blue keeps running and the switch drains connections rather than cutting them, one more reason not to stop it right after the switch. Two kinds of state do not move by themselves:

  • Long-lived connections. The displays at the stops hold a WebSocket open to the planner for live arrivals, and an open connection stays on the instance it was opened to. After the switch they are still on blue, on release 41, until something closes the connection; blue closes them gradually, and each display reconnects through the balancer, to green.
  • Sessions held in memory. A rider signed in to save trips has a session, and if it lives in blue's memory, the switch signs them out. Sessions that must survive a switch live outside both environments, in a store both can read, in a format both releases understand.

Work that never passes through the balancer, such as the job that imports the overnight timetable, has to move with every switch, back as well as forward, and stay off on whichever side is idle, or it runs twice, once on each release.

One database under both colours

Blue and green are separate sets of instances, but they usually share one timetable database and one store of saved trips, because copying live data to a second database at the moment of a switch is a hard problem of its own. Sharing it is what makes the fast rollback conditional. At the load balancer, rollback is an instant flip back to blue, as long as both versions can work against the same data: keep database changes backward-compatible (expand/contract) so blue still runs after green has written to it. The flip does not undo what green wrote: those rows stay, and Progressive delivery: blue-green, canary & rollback sets the rule for both versions sharing it.

Release 42 replaced a saved trip's step_free flag with a richer accessibility field. Had it stopped writing step_free, a switch back would have left release 41 reading trips saved by green with no step_free value, and planning routes with stairs for riders who had asked for none. So green keeps writing the old field for as long as blue may come back; Schema migrations without downtime gives the full sequence of steps.

How long the way back stays open

The rollback lasts as long as blue does. It ends when the team deploys the next release onto blue, now the idle side, or runs a contract step that blue's code cannot survive, or tears blue down to stop paying for it. Choosing that moment is choosing how long a problem may take to show itself: a bug that appears only at the Monday peak needs a blue that still exists on Monday.

Keeping blue costs a second production environment able to carry the full load, for as long as the window stays open. And the switch moves every rider at once, so a fault only real load reveals reaches all of them; limiting who sees a release first is what Canary Deployment does, or a Feature Toggle that ships the new code switched off and turns it on for a few users first.

Blue and green in Progressive Delivery

The Progressive Delivery Challenge asks you to model Blue-Green with two full environments and an instant traffic switch, and to show rollback as flipping traffic back, not redeploying. Put the Blue-Green Deployment node in front of two Service nodes, mark which one is live, give the idle one an edge that only the smoke tests use, and make the rollback edge point at an environment that is still running, not at a build. The release risk to write beside it is the one blue-green reduces, a slow way back: with blue still running, once the team decides to roll back, the flip takes seconds instead of the length of a redeploy. A bad release reaching fewer riders is the canary's label, not this one.

Practice this