Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Feature Toggle

Switch behaviour on or off at runtime without redeploying, with the decision made in one place in the code and each release toggle removed once its rollout is done.

Six weeks of unfinished code on main

A podcast platform is replacing the transcoder that turns each uploaded episode into the formats its players stream. The new one is faster and cheaper to run, and it will take six weeks to finish. The team merges to main every day. Six weeks on a branch ends in a merge nobody can review properly; merging daily puts half-built code in every deploy. Both stop being a problem if the new code is in production but unreachable until someone decides otherwise.

A Feature Toggle (also called a feature flag) wraps new behaviour in a runtime switch. You deploy the code dark, then turn it on for some or all users, without a redeploy. That is what lets trunk-based development, gradual rollouts, kill switches and A/B tests work inside one deployed version.

Where the decision is made

A toggle has three parts, and keeping them apart is most of the pattern. The names come from Pete Hodgson's article "Feature Toggles (aka Feature Flags)", published on Martin Fowler's site:

  • The toggle point is the place in the code that branches: if (toggles.newTranscoder(episode)).
  • The toggle router decides the answer for this request, from the configuration and the context: who uploaded the episode, on which plan, in which environment.
  • The toggle configuration holds the state: a constant, which changes only with a deploy; a file, usually re-read on restart; or rules in a flag service, which change while the process runs.

The toggle point asks a question about the feature, newTranscoder(episode), not about the mechanism, config.get("flags.new-transcoder.enabled"). When the rule changes from "on for staff" to "on for 10% of uploads", only the router changes. Often the router is a thin wrapper over a flag service's SDK; how that service stores rules, ships them to every process and evaluates them without a network call per check is Feature flag services: targeting, rollouts & kill switches.

One decision per episode

The job worker asks the router once and picks a transcoder; the code below it never sees the toggle.

Decide once, at the edge

The first draft scattered the check through the upload handler, the job worker, the waveform builder and the code that names the stored files: eleven toggle points for one decision. Every one is a place where the two paths can disagree. When someone moved the rollout while an episode was in flight, the worker had read the toggle as on and the storage code, minutes later, read it as off, so the new transcoder's files were saved under names the players never ask for.

So decide once, as early in the work as possible, and pass the decision down as an object. The job worker asks the router once per episode, picks a Transcoder, new or legacy, and passes it down; nothing below it, the code that names the stored files included, knows a toggle exists. That is the Strategy pattern with a toggle choosing the strategy. It also leaves a single toggle point to delete when the toggle goes.

Four kinds, four lifetimes

Toggles differ in how long they live and how often their answer changes, and that decides how each is built:

  • Release toggles hide unfinished or unproven work, like the new transcoder. They live days or weeks, and their rules change only when someone moves the rollout.
  • Ops toggles let operators degrade the system under stress, such as switching off waveform previews when the transcoding queue backs up; a kill switch is one, and it can stay for years.
  • Experiment toggles split listeners between variants, say two loudness targets, and must give each listener the same variant on every request. Measuring which variant wins is A/B testing & experimentation.
  • Permission toggles turn a feature on for some users for good, such as video episodes on paid plans. They last as long as the product rule does, and are decided per request.

Two paths, two test runs

Every toggle doubles the paths through the code it guards, and the tests have to follow. Run the suite with the new transcoder off, as production is today, and again with it on, as production will be after the rollout. Ten independent toggles make 1,024 combinations, so nobody tests them all: test the combinations production runs now and the next one it will run, and keep toggles few so that set stays small. Let a test set a toggle through the router, so reaching a branch never means editing a configuration file.

The toggle nobody removed

Three weeks after the new transcoder reached 100%, a search for its toggle still finds it in the router, the job worker, two tests and the flag service. The legacy transcoder is still deployed and still compiled, and nothing exercises it. That is toggle debt: flags accumulate, and every combination of them becomes an untested code path behind a switch someone may flip without knowing what is behind it.

The best-known bill came due in 2012 at Knight Capital, a US trading firm. New code reused an old flag, and one server that had not received the new code still held a long-dormant feature behind it. Turning the flag on reactivated that dormant code, which flooded the market with orders, and in under an hour the firm lost more than $460 million.

So a release toggle is created with its removal attached: an owner, an expiry date the router warns about once it passes, and a ticket to delete it. Removing it means deleting the toggle point, the losing implementation and its tests, then the flag's definition, and never giving the old name to a new feature.

Watch outRelease toggles are debt — track them and remove each one once its rollout is complete. Long-lived toggles (kill switches, permission flags) are fine, but keep them few, named for their purpose and covered by tests.

Next to canary and blue-green

The Progressive Delivery Challenge puts Feature Toggle beside Blue-Green and Canary, which move traffic between deployed versions (Blue-Green Deployment, Canary Deployment), and Parallel Run; a toggle switches behaviour inside one version, per audience. Label the audience each toggle serves, and focus on one thing: treating each release toggle as temporary and planning its removal. A good run has no release flags left on after their rollout.

Practice this