"Why does every answer take a beat?"
A learner in Osaka wrote to the support team of a language-learning app: when she taps an answer, it takes a moment to turn green, and her friends' apps feel instant. The app ran in one cloud region, in Virginia. Every answer check crossed the Pacific and back, roughly 150 ms of network round trip before the server did any work, and a lesson is a few dozen of those. The same single region had a second cost: when it had an outage, learners on every continent lost the app at once.
A CDN had already helped with the audio clips and pictures, which are the same for every learner and can be cached near her. It could not help with the answer check, which reads and writes her own progress.
The same app, in every region
Geode, short for geographical node, deploys identical copies of the backend in several regions, here Virginia, Frankfurt and Tokyo, and lets each copy serve any user's request. Nobody has a home region. The learner from Osaka is served by Tokyo, and when she spends a week in Madrid her phone is served by Frankfurt, with nothing moved or reassigned for her. That separates it from a CDN, which keeps cacheable copies near users and sends everything else to the origin, and from Deployment Stamps, where each copy serves only its own tenants.
Each request is routed to the nearest healthy geode. GeoDNS answers a hostname lookup with the address of a region near the resolver that asked; anycast announces one address from every region and lets internet routing carry each packet to the closest one by network path, which is usually, not always, the closest on the map. Health checks take a failing region out of the DNS answers, or the region withdraws its anycast route, and its users go to the next nearest. How GeoDNS chooses its answers, and why a cached DNS answer can keep sending users to a dead region for a while, is DNS, proxies & the network edge.
Every region writes
Copying the code is the easy part. In Tokyo, the answer check records a finished exercise, adds experience points and extends her streak. A week later, in Madrid, Frankfurt has to know all of that. Every geode must read and write every user's data, so the data layer spans the regions, and there are three broad ways to build it, each paid for somewhere:
- One write region, replicas everywhere. Reads are local, but every write from Tokyo crosses to Virginia and back, and losing Virginia stops writes everywhere until another region takes over. For writes, the regions are not yet equal.
- Every region accepts writes and replicates them to the others asynchronously. Writes are local and fast, but for the length of the replication lag the regions disagree, and two regions can accept conflicting writes to the same record.
- A database that commits each write to a majority of replicas across regions, as Google Spanner and CockroachDB do. There are no conflicting copies to reconcile, and a write whose majority spans regions waits for at least one cross-region round trip.
The app takes the second option for learner progress, which is written constantly and must feel instant. How lag arises and how to watch it is Replication & read replicas; what a reader in Frankfurt may see while it lasts is Consistency models: strong vs eventual (CAP).
GeoDNS sends the learner to Tokyo. Each pair of regions replicates writes in both directions, so any geode can serve her later.
Two teachers, two regions, one deadline
One learner rarely writes from two regions at the same moment. Shared records are another matter. Class 88 has two teachers, one in Berlin and one in New York, and both edit its homework deadline:
- The teacher in Berlin sets the deadline to Friday, and Frankfurt commits it and confirms.
- A fraction of a second later, the teacher in New York sets it to Thursday, and Virginia commits it and confirms.
- Frankfurt sends its write to Virginia, and Virginia sends its write to Frankfurt.
- Each region receives a write that contradicts one it has already accepted.
If each region simply applies whatever arrives last, Frankfurt ends on Thursday and Virginia on Friday, and from then on each region serves a different view of the same class. Deploying identical geodes before deciding this is the classic mistake. Every region has to apply the same rule and reach the same answer: the later timestamp wins, with a fixed tie-break such as the region name, and the other edit is silently dropped, the two values are merged, or the conflict is kept and shown to a teacher. Choosing among those is Conflict resolution for concurrent writes. Data can also be shaped so conflicts rarely matter: if each finished exercise is recorded as an event with its own unique id and its date, the events form a set that only grows, two regions that each add to it can take the union and lose nothing, and points and the streak can be computed from those events rather than stored as counters.
When Tokyo goes dark
Tokyo's region fails, routing stops sending users there, and learners in Japan reach Frankfurt or Virginia instead: slower, still working. Failover covers the switch itself, including the cached DNS answers that keep some users on Tokyo for a while.
With asynchronous replication, the writes Tokyo confirmed but had not yet shipped exist only in Tokyo: lost if its storage is lost, late and possibly conflicting if it comes back. The app accepts that cost for progress, where what is lost is whatever Tokyo had confirmed but not yet replicated, an amount that depends on how far replication was lagging, and sends a teacher's submitted grades through the third option instead, where a write confirmed by a majority of replicas in three different regions survives the loss of any one of them and every such write pays a cross-region round trip.
What stays in one region
Not every record may travel. A school district in the EU can require its pupils' data to stay in the EU, so that data cannot replicate to Virginia, and for those pupils the promise that any region serves anyone narrows to the regions allowed to hold their data. Home regions and routing by jurisdiction belong to Regional routing & geo-partitioning.
Practice: lose a region on purpose
The Multi-Region Active-Active Challenge is this article as a brief: route each user to the healthiest nearby region, accept writes in every region, pick a conflict rule, and expose replication lag as a metric. Residency was pointed to above, and rehearsing the evacuation belongs to Chaos engineering & game days. A run that tests the pattern:
- Draw three regions, each with a load balancer, an application service and a database, with DNS in front.
- Show replication between every pair of regions, in both directions, and label it with its expected lag.
- Write next to the databases which conflict rule they use for a record edited in two regions at once.
- Mark one region as lost, and write on the diagram which writes may be gone and which cannot be.