Two writers, one lock
Two workers both want to run nightly job 42, which rewrites one customer's invoice. Only one should, so each tries to take a lock first. With a 30-second lease, this can happen:
- Worker A acquires the lock. It is valid for 30 seconds.
- Worker A stalls for 40 seconds: a long garbage-collection pause, a frozen VM, a network partition.
- The lease expires at 30 seconds, and the lock service considers the lock free.
- Worker B acquires the lock and writes the invoice.
- Worker A wakes up, still believes it holds the lock, and writes the invoice too.
Two workers wrote under the same lock, and neither saw anything go wrong. Nothing in that sequence is a bug in the lock service; it did exactly what it promised. To see why, look at what a distributed lock actually is.
A lock is a lease
A distributed lock that must survive a crashed holder is in practice a lease: SET lock:job:42 <owner> NX PX 30000 in Redis, or a key with a TTL in etcd (how such a store agrees among its own nodes on who holds the key is the subject of Consensus). It says "I hold this until time T". The TTL (time to live, the 30 seconds after which the lease lapses unless renewed) is not an optimisation, it is the whole point — a holder that crashes must release automatically, or the job stays locked forever — and it is also the source of many bugs in the family, because the holder's idea of how much lease is left can be wrong: a pause, a slow network or clock drift lets the lease expire while the holder still believes it is valid. That is step 5 above.
So the mental model is this: a lock protects the holder's intent, not the resource. The lock service knows who holds the lease; the database, the file store or the payment provider you are about to write to knows nothing about it. Anything that arrives at the resource after the lease expired — a paused process, a GC stall, a slow network — is indistinguishable from a legitimate holder unless the resource itself can tell.
Let the resource say no
The fix is a fencing token: the lock service returns a strictly increasing number with every acquisition (etcd's revision, ZooKeeper's zxid, or a counter incremented in the same atomic step as the acquisition, in a store that cannot lose or repeat it on failover). Every write to the protected resource carries the token, and the resource rejects any token lower than the highest it has seen. Replay the trace: A acquired token 33, B acquired 34. B's writes carry 34 and the resource records it; A's late write carries 33 and is refused. This needs the resource's cooperation: a payment provider or any other API that cannot check the token gets no protection from it.
The lock only orders who should write; the token is what stops the one who shouldn't.
Now the resource refuses every write that arrives after a newer token, and the lock is merely what makes contention rare. In a database, the check is a condition on the write itself, as in the diagram, and the same statement records the token (SET last_token = 34, … WHERE 34 >= last_token); Transactions & isolation levels explains why a condition inside the write is safe where a separate read-then-write is not. One gap remains: a stale write that lands before the new holder's first write still passes, because the resource has not seen 34 yet. So the new holder should register its token at the resource, with a write that only ever raises last_token, before it reads.
SET NX PX is fine for efficiency locks — avoiding duplicate work where a rare double-run costs a wasted CPU minute. It is not enough for correctness locks, where a double-run corrupts data: there you need fencing, and a lock store that survives failover without handing the same lease to two owners (a consensus-backed one).Leader election is a long lock
Leader election through a lock service is the same lease with a longer purpose: many instances run, one holds leader and does the singleton work (schedule jobs, drive a compaction, own a partition), the rest watch the key and stand by. Two rules keep it from repeating the trace above:
- The lease must be renewed well before its TTL: renew every 10 s on a 30 s lease, so one slow renewal does not cost the leadership.
- The leader must stop acting once its own deadline passes — the moment it sent the last successful renewal, plus the TTL, minus a margin for clock drift — rather than wait to see a renewal fail, which a paused or hung leader never sees.
Followers take over via the same acquire, and the fencing token becomes the term number that every downstream action carries and every resource checks. That token is what stops a leader that paused past its deadline, because the second rule only helps a leader that is running well enough to check its clock.
Often the best lock is none
Locks exist because two workers can reach for the same resource. A single-writer design removes that possibility instead of refereeing it: each resource gets exactly one owner by structure. Job 42 rewrites customer 17's invoice, so partition the job queue by customer, and only the consumer that owns customer 17's partition runs it; there is no lock to take. The same move routes all writes for account a to one shard, or turns the state change into a conditional update the database serialises. What remains is the handover: when an owner fails, its partitions move, and the new owner may repeat work the old one half-finished, so idempotent processing has to cover the retries. Ownership is itself held on a lease, so an old owner that was only paused can still write after the move; its writes need the same fencing, with the assignment's number as the token. Choosing the key is covered in Sharding & partitioning, the conditional update in Transactions & isolation levels, and harmless repeats in Idempotency & exactly-once illusions.
How lock-based designs fail
- No fencing token. The lock "works" in every test and fails the first time a process pauses longer than the TTL while another worker takes the lock — and at scale, such pauses are a matter of when, not if.
- Renewing from a background thread while the work continues. If the renewal thread is alive but the worker is stuck, you have a live lease on a dead worker; tie renewal to progress.
- A TTL shorter than the work, with no heartbeat. The lease expires mid-job by design and a second worker starts the same job. Either heartbeat, or make the TTL comfortably longer than the p99 job time, and keep fencing for the jobs that still outlast it.
- An idle fleet of followers. If leadership only matters for one background task, the other N−1 instances are paying only for fast takeover when the leader dies; if that is not worth it, partitioned ownership, as above, puts every instance to work.
Draw the token, not just the lock
Open the Distributed Lock Service challenge and focus on one thing: the fencing token — draw it as a label on the acquire edge and again on the write edge into the protected store, with the store's rejection rule spelled out. A good run makes clear the store, not the lock, is the last line of defence. Leader Election applies the same lease with renewal; Distributed Job Scheduler is where a single leader dispatches and each job runs under its own fenced claim; Scheduled jobs & cron covers the scheduler itself.