Off in seconds, on 400 servers
At 17:40 on a Friday, a note-taking app raises its new table editor from 5% of its free-plan users to 50%. By 17:46 support has eleven tickets: pasted tables lose their last column on save. The code has been on all 400 API servers for three weeks, and nobody needs to deploy anything. Someone turns the flag new-table-editor off, and within seconds every server is sending every user to the old editor.
That sentence hides a system: something stores the rules, gets a change onto 400 servers in seconds, decides per user who is in the 5%, and keeps answering when it is itself down. The if in the code that reads the flag is the design pattern Feature Toggle.
The rules live in one place
A feature flag is a named switch whose value is decided at runtime, for each user or request, by rules kept outside the code. Shipping the editor with its flag off is a deploy; turning it on for someone is a release, weeks later if you like. The rules live in a control plane, a service with a console and an API for creating flags and editing their rules; LaunchDarkly, Unleash and Flagsmith are examples; LaunchDarkly is a hosted service, while Unleash and Flagsmith can also run in-house.
The obvious design is one call to the flag service per check. At 30,000 document loads a second, that adds 30,000 network calls a second, a few milliseconds to every load, and a new way to fail: when the flag service is slow or down, so is the app.
Ship the rules, not the answers
So the flag service sends rules instead. Each API server runs its SDK, a library that downloads every flag's rules at startup, keeps them in memory and evaluates them locally: a check is a function call, well under a millisecond, with no network. Changes reach the SDK in one of two ways. With streaming, the SDK holds a long-lived connection, often server-sent events, and each saved change usually lands within a second or two. With polling, it asks every 15 or 30 seconds, which is simpler through proxies, and a kill switch can take a full interval to arrive, longer if a poll fails. Browser and phone SDKs usually receive their user's evaluated values instead, since the rules can name customers.
Rules travel once, and again on each change; every check is a function call inside the API server.
Who is in the 5%
A flag's rules are read top to bottom against the user the application passes in, an id plus attributes such as plan, and the first one that matches decides. For the table editor in production:
- If the user is in the segment
staff, serve on. - If the user is on the free plan, serve on to 5% of them.
- Otherwise, serve off.
A segment is a named set of users reused by many flags. Each environment, such as staging and production, has its own rules for the same flag key and its own SDK key, so turning the editor on in staging cannot turn it on in production.
Rule 2 is where a naive version goes wrong. A random draw per request, Math.random() < 0.05, gives a user the new editor on one load and the old one on the next. Storing each assignment brings back a lookup per check. The SDK computes it:
- The SDK joins the flag key and the user id:
new-table-editor:user-48213. - The SDK hashes that string, with a hash such as SHA-1 or MurmurHash, and maps the result to one of 10,000 buckets, numbered 0 to 9,999.
- The SDK serves on if the bucket is below the rollout: under 500 for 5%.
Every server computes the same bucket for user 48213 without asking anyone: the rollout is deterministic and sticky. Raising 5% to 50% keeps everyone already in, since a bucket under 500 is also under 5,000, and adds the users in between. Without the flag key in the hash, the same 5% of users would see every new feature first.
Hash on the id the experience must be consistent for: the account for an editor, perhaps the organisation for a feature colleagues share. A logged-out visitor needs a stable cookie or device id, and signing in can move them to the other side. A rollout says who gets the editor, not whether it is better; that is A/B testing & experimentation, which assigns users with the same hashing.
When the flag service is down
When the flag vendor has an outage, the running servers barely notice: they keep evaluating the last rules they received while the SDK reconnects. What stops is change: a kill switch flipped during the outage reaches no server until the stream comes back.
A server that starts during the outage has no rules at all. The SDK waits a few seconds for its first rules, so an outage cannot hold up startup indefinitely, and every check names a default, served when the flag cannot be evaluated, as in variation("new-table-editor", user, false). Choose it for the moment nothing is known: off, for a new editor. Some SDKs can also boot from rules saved in Redis or a file.
A switch you have already tried
Back to the broken tables. Turning the flag off, which overrides every rule, is its kill switch, and it works only if three things hold, all checkable before launch: changes arrive fast, the old path is still deployed and tested, and the old editor can read what the new one saved. The last is the easiest to miss: if the new editor wrote tables in a new format, sending its users back to the old one is a second incident, because the flag changes which code runs, not what was saved.
So flip it before the launch, in staging and then in production for staff only; a kill switch first tried during the incident makes the incident its test. Flags pair with canary deploys: the deploy ships the code to a slice of servers, the flag decides who sees it and stays as the off switch that needs no deploy. Canaries and their rollbacks are Progressive delivery: blue-green, canary & rollback.
Monday's review then asks who raised the rollout to 50%. The control plane records every change, who, when, which flag and environment, the rules before and after, so the 17:42 error spike lines up with the 17:40 change; a record that holds up even against the service's own admins is Audit logs & tamper evidence.
Two things do not stay behind a flag. A release flag at 100% is code to delete, the lifecycle Feature Toggle owns, and a value that releases nothing, such as a timeout or a page size, belongs in configuration: Configuration management.
Build the service yourself
The Feature Flag Service Challenge asks for this system, from targeting rules to evaluations that survive a brief control-plane outage. Put the evaluation inside the application service, label the edge that streams rule changes, and write next to each flag the default it falls back on. What that Grid cannot tell you is whether the 50% were better off with the new editor; the A/B Experimentation Platform Challenge adds the exposure logging and metrics that answer it, on the same assignment.