Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

External Configuration Store

Move configuration out of each deployment into one shared, versioned store that every instance reads: how instances learn of a change, what they run on when the store is down, how they apply a whole version at once, and who may change which keys.

How long until all 240 see it?

The question comes from the operations lead of a company that runs online exams for universities. Its submission service runs 240 instances in three regions, and on the morning of a large exam a lecturer asks for five more minutes of grace on late uploads. The value, submission.grace_seconds, is an environment variable set when each instance starts. Changing it means restarting 240 instances while 30,000 students are uploading answers.

An External Configuration Store takes values like that out of each deployment and keeps them in one shared, versioned place that every instance, and often every application, reads at runtime. The value is changed once, the change applies without redeploying, and every instance converges on the same versioned config. The store can be a key-value store such as etcd or Consul, or a dedicated config service (AWS AppConfig, Azure App Configuration, Spring Cloud Config), which stores configuration centrally and delivers updates to running services at runtime, usually by the services polling for new versions (or via a push/refresh event, as with Spring Cloud Bus), with validation and versioned rollback. What belongs in it, how its values are layered and validated, and why a change to it deserves a deploy's care is Configuration management; this article is about the store and its readers.

Read once, ask again, or be told

Moving the value into a store is half the job. Each instance still has to find out that it changed, and there are three ways to do that, which decide how long the 240 take:

  • At start-up only. The instance reads the store when it boots and never again. The value is no longer in the image, but a running instance never sees a change, so the grace period still needs 240 restarts. It is a common mistake with this pattern, and it is easy to miss because the first test, on a freshly started instance, passes.
  • Polling. Each instance asks every N seconds, usually sending the version it already has so the store can answer "unchanged" cheaply. A change takes up to one interval to arrive, longer if a poll fails, and 240 instances polling every 15 seconds put 16 requests a second on the store whether anything changed or not.
  • Watch or push. The store tells the instance. An etcd watch is a long-lived stream of change events, a Consul blocking query holds a request open until the key changes, and Spring Cloud Bus broadcasts a refresh event over a message broker, after which each Spring Cloud Config client fetches again. A change usually arrives within seconds. An instance that was disconnected when it went out misses it only if the notification is neither kept nor resumable, as a broadcast refresh event often is, and a slow periodic re-read guards against that case.

Whichever way it arrives, the instance keeps the config in memory and every request reads that copy. The store stays off the request path: an upload does not wait for it, and its latency does not add to anyone's. Watches themselves are explained in Coordination services: small state, leases & watches, and a flag service is this store with per-user targeting on top (Feature flag services: targeting, rollouts & kill switches).

The store goes down at 09:10

One store for 240 instances is also one dependency for 240 instances. Ten minutes into the exam, the store's region has an outage. The running instances and the ones about to start are in different positions.

A running instance has its config in memory. It keeps serving with it, keeps retrying the store in the background, and simply cannot learn about changes until the store is back. That works only if the instance treats its copy as the truth until a newer valid version arrives, rather than dropping its config, or falling back to defaults, when a poll fails.

An instance that autoscaling starts now has nothing in memory. If it refuses to start without the store, the exam loses capacity just when uploads peak. So each instance writes every version it accepts to local disk, its last-known-good copy, and starts from that if the store does not answer. A brand-new machine has no such copy, so it needs another source, such as a snapshot file the store also publishes to object storage, which has its own availability. The choice to avoid is the silent one: an instance that starts on the defaults in its code and serves with a grace period nobody chose, without saying so.

Every instance keeps a copy

One region follows changes through a watch, another polls. Each instance serves from memory and saves every version it accepts, so a restart during a store outage still has config.

Two keys, one version

In the spring the company moves exam files to a new storage bucket in another region, and two keys change together: uploads.bucket and uploads.region. Stored and read as separate keys, they can arrive separately, one watch event before the other or one on each side of a poll. For that moment an instance holds the new bucket with the old region, and every upload it handles fails.

So the unit an instance applies is a snapshot: every key it reads, at one version, published as a single write. The instance fetches the whole snapshot, validates it, and swaps it in with one reference change in memory, so a request that picks up the reference once sees the old set or the new one, never a mix. Each snapshot carries a version number, and each instance reports the version it is running, so "v18 on 237 of 240" is a number on a dashboard instead of a guess.

Versions do not make 240 instances switch at the same instant. For a few seconds, or a full poll interval, some upload with v17 and some with v18, so a change must be safe while both are live: here, the old bucket stays writable until every instance reports v18 and the uploads that started on v17 have finished. Staging the change itself is part of Configuration management.

Who may change which keys

A store that every application reads is also one place where a single write reaches everything, and one place to read whatever it holds. So access is granted per key prefix, separately for reading and writing. The submission service reads submission.* and uploads.* and nothing of billing's. The lecturers' tool may write only grace-period keys, and a change to uploads.* goes through review. Every write is recorded with who made it, when, and the version it produced, so after an incident "what changed at 09:05?" has an answer.

Credentials stay out of the general keys: they live in a secrets manager with its own rotation, and the store holds at most a reference to them, as Secrets management & encryption explains. And when the readers are thousands of devices that may be offline for days, the store has to track what each device should run and what it reports running, which is Edge Workload Configuration.

The Feature Flag Service Challenge asks for changes pushed to clients in near real time and for evaluations that continue while the control plane is briefly unavailable, the two hardest parts of this pattern. On your Grid, label two things: what a running instance uses while the store is unreachable, and what a newly started one uses.

Practice this