Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Service Registry

How a registry's list of a service's live instances stays close to true as they register, renew, expire and leave, and what callers do while it is wrong or down.

An entry someone will act on

An entry in a registry is a claim that a caller will act on: this address answers, and it can take work now. A wrong entry sends work to nowhere, because the caller trusts it. A team shooter's dedicated servers show why that is hard to keep true. Each hosts up to four ten-player matches and is stopped when demand falls; around 100 run on a weekday morning and 900 on a Saturday evening, and while the fleet grows or shrinks one starts or stops every few seconds. The matchmaker, which places waiting players into matches, has to send each group of ten to a server that is running, has a free match slot and runs the build their game client expects.

No config file can list those servers; the fleet has to write the list itself, as fast as it changes. That list is a Service Registry: a directory with one entry per live instance, holding its address and whatever callers need to choose, here the region, the build and the free slots. Whether the matchmaker, a load balancer or a proxy does the lookup is a separate choice, weighed in Service discovery & the mesh.

How an entry gets written

The match servers use self-registration. Once a server has loaded its maps and opened its port, it sends the registry its id, address, region, build and slot count. Registering last matters: a server registered as soon as its process started would be handed players while still loading.

In third-party registration, something else writes the entry: the platform that started the instance, or an agent beside it. Kubernetes does it for its Pods: a Pod is listed as ready in its Service's EndpointSlices once its readiness check passes, and stops being listed as ready when the check fails or the Pod begins shutting down. The instance needs no registry client, but the platform knows only what its checks tell it, so those checks decide what "registered" means, and designing them is the business of Health Endpoint Monitoring. Self-registration keeps that knowledge in the server's own code, and every service then carries a registry client.

Silence as the signal

A crashing server sends no goodbye, so an entry lives only while its owner renews it. Every match server sends a heartbeat every 10 seconds, and each entry carries a TTL (time to live) of 30 seconds: if no heartbeat arrives for 30 seconds, the registry expires the entry. Expiring registrations that stop heartbeating means dead instances drop out within one heartbeat TTL. The crash does not wait for that, though. Server 41 dies a moment after a heartbeat, and the next half minute goes like this:

  1. Server 41's process crashes; its entry still says it has two free slots.
  2. The matchmaker reads the registry, sees server 41, and assigns it a group of ten players.
  3. The players' clients connect to server 41 and get no answer.
  4. Once 30 seconds pass with no heartbeat from server 41, the registry expires its entry.
  5. The matchmaker's next read no longer includes server 41.
Register, renew, look up

Server 41 renews every 10 seconds and expires 30 seconds after its last renewal; server 17, shutting down, removes its own entry before exiting. The matchmaker reads the list, then talks to a server directly.

The stale window runs from the crash to the expiry, up to that half minute, plus however long the caller keeps its own copy of the list before reading again; no registry that learns of a crash from silence avoids it. A matchmaker refreshing every 5 seconds can lag a crash by up to 35 seconds, if the registry expires entries on time. Callers still need a timeout and a retry against another instance for that window. Here, the matchmaker gives a server a few seconds to confirm a new match. A refusal (no free slot) sends the group to another server; silence leaves the assignment uncertain, since the server may have taken it, so placing the group again has to allow for that. How long to wait, and how to retry without piling on, are worked out in Timeout and Retry.

Why not a 3-second TTL, then? It means renewing about every second, so 900 servers send 900 writes a second, ten times the 90 it takes at a 10-second interval, almost all of them saying "still here". Worse, a garbage-collection pause or a burst of packet loss longer than the TTL evicts a server that is mid-match and fine. And a heartbeat proves only that the process can send heartbeats, not that its game loop runs, which is why it is often paired with a health check.

Leaving on purpose

Hundreds of servers a night shut down on purpose, and none of them should cost players the half minute a crash does. A match server leaves in order:

  1. The server enters drain mode: it refuses any new assignment and sets its free slots to zero in the registry, so the matchmaker stops choosing it.
  2. The server lets its running matches finish.
  3. The server sends a deregister call, and the registry deletes its entry at once.
  4. The server keeps refusing for one more refresh interval of the matchmaker, so no copy still offers it, then exits.

Refusing comes first because setting the slots to zero changes the entry, not the copies callers already read.

When three hundred go quiet together

One evening the link between the registry and one data centre drops for a minute. Three hundred match servers there go unheard, and after 30 seconds the registry would expire them all, though every one is still running and reachable by players. To the registry, a mass death and a cut link look the same: silence.

Some registries refuse to believe a silence that large. Eureka's self-preservation mode stops expiring entries when renewals over the last minute fall below 85% of what it expects (the default threshold), and resumes once they recover. It prefers stale entries to an empty list: callers with timeouts can step around a dead entry, but nobody can call a live server missing from the list. In exchange, a real mass failure keeps its dead entries longer, so the retry path works hardest exactly when the most servers are gone.

When the registry itself is down

Every new match is placed from the matchmaker's copy of the list, refreshed from the registry. When the registry stops answering, the matchmaker goes on placing matches from the last list it read, which grows staler every minute: new servers are invisible, so capacity goes unused, and dead ones stay on it, so timeouts and retries carry more of the load. Servers keep retrying their registrations and heartbeats, so the list catches up once the registry is back.

To make outages rare, the registry runs as several replicas, often on a coordination service such as etcd or ZooKeeper, where an entry can be a key tied to a lease (etcd) or an ephemeral node tied to the client's session (ZooKeeper), and disappears when the renewals stop, the heartbeat and TTL above under another name, explained in Coordination services: small state, leases & watches. Those stores order every write through Consensus, so every replica applies the same registrations in the same order, and when they lose a majority, registrations and renewals stop until a majority is back.

How long can a dead entry fool the Client?

That is the question your design has to answer in the Service Discovery Challenge, whose brief asks for registration on startup, eviction of dead instances, a client reading the registry and deregistration on shutdown. Six of its eight nodes are enough: three Service nodes, the Service Registry, a Client and a Health Endpoint Monitoring node. Put the answer on the edges: the heartbeat interval and TTL on a registration, a deregister edge from an instance shutting down, and on the Client the longest it can be sent to a gone instance and what it does meanwhile. A planned shutdown should come out close to zero. A crash cannot, which is why the Client's timeout and its retry against another instance stay in the design.

Practice this