Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Configuration management

Tell configuration apart from code, secrets and feature flags, layer it so you know which value wins, check it against a schema when it loads, and treat every config change as a deploy that is versioned, staged and easy to roll back.

Six lines in one file

The loans API of an e-book lending platform, used by a few hundred public libraries, reads its settings from a file that is copied into the container image with the code. Six of its lines:

  • db.host: loans-db.internal
  • db.pool_size: 40
  • db.password: 7f…
  • search.timeout_ms: 800
  • retry.backoff: exponential
  • new_reader_app: false

Changing any of them means a new build and a full deploy, the same pipeline run whether someone rewrote a query or nudged a timeout. That cost is what usually starts the conversation. The harder part is noticing that the six lines are not one kind of thing.

Code, config, secret, flag

Configuration is whatever is likely to differ between deployments of the same build: staging and production, one region and another, one instance and the next. The Twelve-Factor App uses that as its definition, and it gives a test you can apply line by line. Applied to retry.backoff, the test gave this team its answer: nothing about staging or production calls for a different strategy, so they ship it and test it with the code, even though it sits in a settings file. db.host, db.pool_size and search.timeout_ms differ between staging and production. They are config.

db.password differs too, but leaking it is a breach, while leaking a timeout is nothing. A secret is config whose disclosure does damage, and it gets its own storage, access rules and rotation, described in Secrets management & encryption; the config keeps only a reference to where it lives. new_reader_app is a release toggle, often targeted per user or cohort, switched on for some readers, then for all, then deleted; that lifecycle belongs to Feature flag services: targeting, rollouts & kill switches. The three differ in who changes them, how often, and how much a wrong value breaks.

Once config is out of the image, the build can be made once and promoted unchanged, the rule in CI/CD pipelines: integrate, build once, promote, and the values come from files, environment variables or a shared config service at deploy or run time. How one store gets a change to many running instances, and keeps them working when it is down, is the pattern External Configuration Store.

Don't confuse this with infrastructure configuration management (Ansible): that sets up hosts and software and keeps them in their desired state; the subject here is the values that steer the application, loaded at startup or updated while it runs. Declaring the infrastructure itself in versioned files is Infrastructure as code.

Which value wins

Taking config out of the image raises the next question: from where, and what happens when two places disagree? Most setups answer with layers, each one overriding the layers before it. For the loans API:

  1. The code holds a default: search.timeout_ms is 800 unless something says otherwise.
  2. A file per environment, kept in version control, overrides it: production sets db.pool_size to 40, staging to 5.
  3. Each instance gets values at start-up, usually environment variables, that override the file: its region, the database host for that region.
  4. Values from the config service, changed while the service runs, override the rest.

The order is a decision, and frameworks make it differently; Spring Boot, for example, lets an environment variable override the same property in an application file. What matters is that it is written down and the same for every service.

Layers hide things. In March an engineer chasing slow searches in the EU region set SEARCH_TIMEOUT_MS to 3000 on that region's deployment, and forgot it. In June the team lowered the timeout in the production file and spent an afternoon wondering why EU searches still waited three seconds. The fix is cheap: each service reports its effective config, every key with its value and the layer it came from, secrets masked, in its start-up log and on an internal endpoint. Then the question "which value is this instance using, and why?" takes a minute.

A typo that starts cleanly

Layers also let a mistake pass quietly. Someone adds serch.timeout_ms: 600 to the production file. Nothing reads that key, so nothing fails: the service starts, keeps the timeout it already had, and the change simply is not there. Someone else writes search.timeout_ms: 2s, and a JavaScript loader that calls parseInt on it gets 2, a two-millisecond timeout that fails almost every search.

The defence is a schema: one declaration in the code of every key the service accepts, with its type, its unit and its allowed range, checked when config is loaded. Unknown keys are rejected, so the typo fails; 2s fails the integer type; a pool size of 4000 fails a range of 1 to 100. A service that finds invalid config at start-up should refuse to start, because that is the cheapest moment to fail: in a rolling deploy that waits for new instances to become ready, the first new ones stop with an error naming the key, and the old instances keep serving. A value changed at runtime gets the same schema twice, once in the pipeline before it is published and once in the service, which keeps its current config and reports the error instead of applying a value it cannot use.

Watch outA loader that reads each key lazily, the first time code asks for it, defeats the check: a bad value surfaces hours later on the one code path that uses it. Load and validate the whole set at start-up.

Valid, reviewed, and still an outage

In November the database team asked the loans API to hold more connections for the evening peak. The change was one line, db.pool_size from 40 to 400. The schema's maximum was raised to 500 to let it through, a colleague approved it, and the config service delivered it to all 60 instances within a minute. As evening traffic rose, each instance opened more connections, until 60 instances could ask for up to 24,000 against a database that accepts 5,000. New connections were refused, and loans failed everywhere at once.

Nothing about the value was malformed. It was wrong only in combination with the database's limit and the number of instances, which a schema of types and ranges does not check. What catches it is what catches a bad code release: exposing the change a little at a time. A config change is a deploy, and it needs what a deploy has:

  • Versioned. Config lives in version control, or in a store that keeps every version, so each change has an author, a reviewer, a diff and a number.
  • Staged. The change reaches a few instances first, and moves on only if they stay healthy and the projected connection total fits what the database accepts. 400 on three instances is 1,200 connections; with the other 57 still at 40 the total is 3,480, under 5,000, and the three look healthy. Projected to all 60, the same value asks for 24,000, so the rollout stops there.
  • Rolled back in one step. Going back means publishing the previous version, not retyping an old value from memory in the middle of an incident.
A config change takes the deploy path

The change is reviewed and schema-checked like code, published as a numbered version, and reaches three instances first; the other 57 get it only if those stay healthy and the projected connection total fits the database.

Config is often the fastest way to change production, faster than code, and that is exactly the risk: a change that skips the release process reaches everything together. In July 2024 a content configuration update for CrowdStrike's Falcon sensor, sent to Windows hosts without a staged rollout, crashed about 8.5 million machines, and staged deployment of such updates was among the fixes the company announced. How to run the stages, and what to watch while they run, is Progressive delivery: blue-green, canary & rollback.

Drawing the line in the Feature Flag Service

The Feature Flag Service Challenge asks for a flag and configuration service with environments and an audit of who changed what, which makes it the place to practise the boundary this article draws. Put a non-flag value next to the flag rules, an evaluation timeout for instance, and show how it is handled: one value per environment instead of per user, a schema check before it is published, a numbered version, and a rollout that reaches a few instances before the rest. If a value on your Grid targets particular users, it is not config: it is a flag, or data the product keeps per user.

Practice this