Forty-four keys
The most telling thing about a coordination service is how little is in it. Here is everything a video-transcoding service with 40 workers keeps in its etcd cluster:
/transcode/leaderholdstc-04, attached to lease7a3f, which has a 10-second TTL./transcode/members/tc-01through/transcode/members/tc-40each hold one worker's address, attached to that worker's own lease./transcode/config/max-parallelholds6, and/transcode/config/profileholdsav1-main-v3./transcode/locks/reindexholdstc-17, attached to lease9e10.
The uploads, the jobs and their progress live in a PostgreSQL database. etcd holds a few kilobytes that change a few times an hour and that every worker depends on: who hands out new uploads, who is alive, how many jobs each may run, who is re-indexing the storage bucket tonight. That is what a coordination service such as etcd, ZooKeeper or Consul is for: a small replicated store that gives the whole cluster one agreed history for each key, and keeps doing so while some of its own machines are down. How its members reach that agreement is a story of its own, told in Consensus; this one is about what the keys above are made of, and why the list is this short.
Keys that delete themselves
The leader key and tc-04's member key share lease 7a3f. A lease is granted by the service with a TTL (time to live), and the holder keeps it alive by sending keep-alives. Keys can be attached to a lease, and when no keep-alive arrives for longer than the TTL, the lease expires and every key attached to it is deleted at once. If tc-04 crashes, nobody cleans up after it: roughly 10 seconds after its last keep-alive on a healthy cluster, its leadership and its membership disappear in one step. ZooKeeper's equivalent is the ephemeral node, deleted when its client's session ends.
So the members prefix is a membership list that maintains itself: a worker joins by writing its key on its own lease, and reading the prefix lists the workers that renewed recently. Recently is the honest word: a worker frozen by a long garbage-collection pause keeps its key until the TTL runs out. Choosing the TTL is worked through in Leader Election; what a late holder can still break, in Distributed locks & leader election. A key written without a lease outlives its writer, so a lock held that way stays taken forever once its holder crashes. Finding an instance by name is a different job (Service discovery & the mesh), and clusters too large for one central list track membership with Gossip Protocol instead.
Write only if
A lease explains how the leader key goes away, not how exactly one worker got it: forty workers that all see the key missing and write their own name would overwrite each other. The service offers a conditional write instead. In etcd it is a transaction that compares and writes as one step: if the leader key's create revision is 0, meaning the key does not exist, put my name there on my lease; otherwise do nothing. One worker's transaction succeeds and the other 39 get a clean no.
The same move protects the config. Two operators open max-parallel at the same time and both read 6; one writes 8, the other 4, and a plain write keeps whichever lands last. A write conditioned on the key's mod revision, the point in the store's history where the key last changed, refuses the second one, because the key changed after that operator read it. This is compare-and-set: update only if the key has not changed since you read it, with no lock held in between.
Those revisions matter beyond config. etcd numbers every change to the store with one counter that only grows, and ZooKeeper gives every change a zxid, a transaction id that also only grows. So the revision at which tc-17 created the reindex lock key is higher than any earlier holder's and lower than any later holder's: a fencing token, handed out for free. How the protected resource uses it to reject a holder whose lease expired is shown in Distributed locks & leader election.
Told, not polling
Every worker needs to know when max-parallel changes or the leader moves, and asking every second would be 40 reads a second for a value that changes twice a week. With a watch, the worker subscribes to a key or a prefix and the service pushes changes. An etcd watch can start from a given revision and then delivers every change from that revision on, in order, so a worker that reconnects after a network blip resumes where it left off, unless that history has been compacted away, in which case it reads the current value and starts a new watch at the revision right after that read. Not every watch is that complete (a classic ZooKeeper watch fires once and must be set again), so the safe habit is to treat a watch as "something changed, go and read", not as a complete change log. And keep watchers few per key: 300 tasks watching one lock all wake on each release to race for it, which is why ZooKeeper's lock recipe has each waiter watch only the one queued just ahead of it.
The key that stayed out
Someone proposes adding job progress: each of the 240 running jobs (40 workers, 6 each) would write its percentage under /transcode/progress/ once a second, and the dashboard would watch the prefix. The values are tiny and etcd would accept them. They still belong in the database, because of what they would do to the keys already there.
- Every write costs agreement. In etcd or ZooKeeper, each write is ordered by one leader and flushed to disk on a majority of members before it is acknowledged. That is the right price for deciding who leads, and a waste on a progress bar at 47%.
- History and load land on everyone. 240 writes a second is about 20 million new revisions a day, each kept until the store is compacted, in a service built for values measured in kilobytes (by default etcd refuses a request larger than 1.5 MiB). If those writes saturate the members' disks, lease renewals slow down with everything else, and a healthy leader can lose its key to a timeout it would easily have met on a quiet cluster.
What belongs in a coordination service is small, changes rarely, and is something many processes must agree on: who leads, who holds a lock, who is a member, what the shared configuration says.
The same reasoning keeps the service off the hot path, the path a user's request travels. A worker reads its config at startup and then follows a watch, and knows who leads by watching the leader key, not by asking before each upload. A slow or briefly unreachable coordination service then delays changes, not uploads; an outage longer than the leader's lease also pauses the handing out of new uploads.
tc-04 keeps its lease alive and tc-01 follows the config through a watch. Neither waits on etcd to transcode a file; progress goes to the database.
Running it
On one machine, the service is a single point of failure for every leader, lock and config key in the fleet, so it runs as three or five members spread over availability zones; why those sizes and not four is answered in Consensus. Losing one member is routine: the cluster keeps serving, and the job is to replace it before a second one goes. Losing the majority is not: writes stop, so no lock can be taken, no lease renewed, no leader replaced, and clients see errors and timeouts. The service has picked consistency over availability (Consistency models: strong vs eventual (CAP)). The workers must be ready for that: a leader whose renewals fail stops acting at its own deadline, and a transcode already running carries on.
Building the lock service yourself
Everything above is what the Distributed Lock Service challenge asks you to provide, ZooKeeper-style. Draw a lock service of your own between the clients and etcd, since in this Challenge the store alone does not count as the service, and send watch notifications back to the clients. Two details decide whether the design is safe: a leased lock, so a crashed holder releases automatically instead of blocking forever, and a fencing token (the lock's revision number) passed with every write, so the resource, which keeps the highest token it has seen, rejects a stale holder whose lease expired while it was paused. For the leader key on its own, the smaller Leader Election challenge is enough.