Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Canary Deployment

Send a small share of traffic to a new version, compare it with a baseline of the old one, ramp it up in steps while the comparison holds and abort automatically when it fails: where the split happens, keeping each user on one version, which metrics to compare, and how long each step has to last.

How long at 1% is long enough?

The forecast API behind a weather app answers about 2,000 requests a second from 20 instances. Version 2 replaces the code that interpolates a forecast between the weather model's data points, and the on-call engineer asks the practical question: if v2 gets 1% of the traffic, how long before its numbers mean anything?

That 1% is what makes this a Canary Deployment: the new version receives a small percentage of traffic while the rest stays on the old one. You watch the canary's error rate and latency; if it stays healthy you ramp it up to 100%, and if it does not you roll back, having exposed only the users in that slice to v2, though what v2 wrote to shared data stays.

At 1%, the canary sees 20 requests a second, 12,000 in ten minutes. If v1 fails 0.2% of requests, the canary should show about 24 errors in that window at v1's rate, and about 48 if v2 doubled it, a gap that stands well clear of chance. A v2 that is 10% worse would show about 26, which no ten-minute window at 1% can tell apart from noise. So the length of a step is not a habit: it is the sample that step needs to see the regression you care about, and a smaller slice needs a longer wait.

Where the split happens

Something has to choose a version for each request. The usual places are a load balancer with weighted backend pools; a service mesh, whose routing rules split by weight and also by header, so requests from employees' phones reach v2 before any customer's; and DNS weights, which move whole regions or client populations and give only an approximate share that changes slowly, because resolvers and clients cache the answers. The balancer side is the territory of Load balancing & horizontal scaling.

The measuring side has to split by version too. Every metric and log line carries the version that produced it, so "is v2 worse?" is a query rather than a guess.

The same user, two forecasts

On the first day at 5%, a support ticket came in: the app showed 14°C for Saturday afternoon, then 16°C after a pull to refresh, then 14°C again. The split was made per request, so consecutive refreshes from one phone landed on different versions, and v2's interpolation gave slightly different numbers. Nothing had failed; the user had watched the release flicker.

So the split is made sticky: each device stays on one version while the weight holds. The splitter hashes a stable identifier, here the device id, into one of 100 buckets and sends the buckets below the current weight to v2. At 5%, buckets 0 to 4 go to v2; at 25%, buckets 0 to 24, so devices already on the canary stay there as it grows and only new ones join. For browsers, a cookie set on the first response can carry that stable identifier instead. Stickiness changes what the sample is: about 5% of devices rather than 5% of requests, and a device that sends many requests weighs more in the canary's error rate.

Pinning users does nothing for their data: what v2 writes, v1 may have to read, which is why Progressive delivery: blue-green, canary & rollback sets the rule for both versions sharing one database.

Compared with what?

The analysis needs a comparison, and the obvious one is wrong. Watching the overall error rate while 5% goes to v2 hides a regression behind the 95% that are fine: if v1 fails 0.2% of requests and v2 fails 2%, ten times as often, the total moves from 0.2% to about 0.29%, which can sit inside an ordinary day's noise.

Comparing v2 with v1 over the same window fixes that and leaves a subtler bias. The stable instances have been running for days with warm caches, while the canary started an hour ago, so a cold start can look like a v2 regression. Careful setups also deploy a baseline: fresh instances of the old version, as many as the canary has, started at the same time and given the same share of traffic. The comparison is then canary against baseline, and the code is the main thing that differs between them.

Canary and baseline, same share, same age

The splitter keeps each device on one version. The analysis compares the canary with a fresh baseline of v1, not with the long-running stable fleet, and its verdict moves the weights.

Which metrics? Error rate and latency percentiles, p99 above all, are the starting set, compared per version and per endpoint, since a regression on a rarely used endpoint disappears into the average. Add one for what the service is for: here, the share of forecasts with missing hours, which an interpolation bug can raise while every request still returns 200. A health check is necessary but not sufficient: Health Endpoint Monitoring tells you an instance is up, and a canary with a real regression usually still passes its health checks. Collecting and labelling them is instrumentation work, as Observability: metrics, logs & alerting shows.

The step has to outlast the cycle

The forecast data reloads every six hours. v2 passed 1% and 5% in the morning, went to 100% after the 25% step had looked clean for twenty minutes, and at the evening reload it held a lock long enough to time out a third of all requests for 90 seconds. A step shorter than the longest cycle in the system can miss what happens at its end: cache entries expiring, hourly jobs, a connection pool that runs dry only at peak, memory that leaks a little every hour. The pause at each step, the soak, has to be long enough, and see enough users, to catch a real regression before the ramp goes on, and a step before the large ones has to cover the cycles you know about.

Abort without asking

The policy the team settled on: 1% to v2 and 1% to a fresh baseline for 30 minutes, then 5%, 25% held across at least one data reload, 50% and 100%, each step only after the analysis passes; 100% is the promotion itself, so the last comparison of canary against baseline is the one at 50%. A failed analysis at any step sets the canary's weight to 0 and stops the ramp, without waiting for a person: promote or abort on the analysis, not on someone watching a dashboard. Two rules keep that automation honest. Missing data is a failure, not a pass: a canary that received no traffic because of a routing typo has no errors to report, and promoting it would ship code no user has run. And a timer is not a gate: the ramp moves on the analysis, so a step that has not yet seen enough requests waits. The thresholds themselves usually come from burn rate against an error budget, from SLOs & error budgets.

The Code Deployment Platform Challenge asks for this loop in two of its requirements: "Shift traffic gradually to the new version with canary steps gated on live metrics" and "Roll back automatically when a rollout degrades key metrics". On your Grid, the edge that raises the canary's weight should start at the metrics; if it starts at a clock, you have rebuilt the twenty clean minutes that sent the forecast v2 to 100% before its evening reload.

Practice this