Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Replication & read replicas

Copy data to extra nodes for read scaling and failover, knowing async replicas lag and can lose writes on failover.

5,500 reads a second, one server

An online shop runs on one PostgreSQL server. At the evening peak it handles about 6,000 queries a second, and 5,500 of them are reads: show this product, list this category, is this size in stock. The server sits at 90% CPU, and nearly all of that work is answering questions about data that changes a few times a minute.

Reads are the easy kind of work to share out, because any copy of the data can answer them. Replication keeps copies of the data on additional nodes. The common setup is one primary that takes every write and several read replicas that serve reads. The primary records each change and ships it to the replicas, which apply it to their own copy. Point the product pages at two replicas and the primary is left with the writes and the reads that cannot go elsewhere. The replicas give you something else too: a standby that already holds the data, ready to be promoted if the primary fails.

Writes to primary, reads from replicas

The copy is a little behind

In the usual configuration the primary does not wait for the replicas. It commits the write and answers the application without waiting for any replica to confirm it has the change. That is asynchronous replication, and it means a replica can trail the primary by some amount, called replication lag. Usually the lag is brief. It grows under load: a burst of writes, a long-running query on the replica, a slow link, a single statement that rewrites millions of rows.

Lag has a symptom users notice. A customer changes their delivery address, the page reloads, the read goes to a replica that has not applied the change yet, and the old address comes back. Reads from a replica are eventually consistent: they can return an older value for a while. Read your own writes from the primary when freshness matters. Which reads can tolerate staleness, and the cheaper guarantees that fix this particular complaint, are the subject of Consistency models: strong vs eventual (CAP).

Two habits keep lag from turning into an incident. Monitor it, since most databases report how far each replica is behind, in bytes or in seconds, and a replica that is minutes behind is serving minutes-old answers. And keep reads that must be transactionally consistent, such as billing or the last unit of inventory, on the primary, and make the check and the update one atomic step there, such as a conditional update or a transaction that locks the row: reading the primary alone does not stop two buyers from taking the same unit, which Transactions & isolation levels covers.

The night the primary dies

At 21:14 the primary's disk fails. One replica is promoted, the application is pointed at it, and the shop is back in a couple of minutes. The next morning, support finds three orders that customers have confirmation emails for and the new primary has never heard of. They were committed on the old primary in the last fraction of a second before it died, and the replica had not received them yet. With asynchronous replication, promoting a replica after the primary fails loses every write it had not yet received.

Synchronous replication closes that gap: the primary waits until a replica confirms it has the change before acknowledging the commit, so an acknowledged write exists on two nodes. It costs a network round trip on every commit. And if the synchronous replica goes down or slows down, commits stall, unless the database falls back to asynchronous mode after a timeout, which reopens the gap. A common compromise is one synchronous replica and the rest asynchronous; failover then has to promote that synchronous replica, because only it is sure to hold every acknowledged write. One detail matters for reads: in many databases, synchronous means the replica has stored the change, not necessarily applied it, so a read from that replica can still be slightly behind.

Failover has a second trap. A health check that cannot reach the primary does not prove the primary is dead; it may only be cut off from the checker. Promote a replica while the old primary is still accepting writes from some application servers, and you have two primaries taking writes that will not reconcile on their own, a state called split-brain. The old primary has to be fenced, made unable to accept writes, before the new one takes over, and the whole switch needs rehearsing before the night it is needed. Distributed locks & leader election covers how leases and fencing stop a former leader from writing.

What replicas do not give you

Every replica applies every write, so adding replicas adds read capacity and leaves write capacity where it was. When the primary runs out of room for writes, the answer is splitting the data across nodes, which Sharding & partitioning covers. And a replica copies mistakes as faithfully as good data: a DELETE without its WHERE clause reaches every replica within the lag. Getting that data back is the job of backups, in Backups, high availability & disaster recovery.

No primary at all: quorums

The shop's cart service has different priorities. Losing the ability to add to a cart because one node is down costs sales, so it runs on a store with no primary, where any replica accepts writes. Each item is copied to N replicas, a write succeeds once W of them acknowledge it, and a read waits for answers from R of them and keeps the newest version it sees, comparing the version each stored value carries. With N = 3, W = 2 and R = 2:

  1. The app writes the new cart to all three replicas.
  2. Two replicas acknowledge, and the write succeeds; the third is slow and still holds the old cart.
  3. A read asks all three replicas and uses the first two answers.
  4. At least one of those two answers holds the new cart: two replicas acknowledged the write and two answered the read, and with only three replicas, at least one is in both groups. The read keeps the newer version.

That overlap is the point of the quorum rule. With a strict quorum, R + W > N means every read reaches at least one of the same N replicas that acknowledged the latest write, so the reader can pick the newest version. The slow replica is brought up to date later, by the read that noticed it was behind or by a background sync. Smaller quorums, where R + W ≤ N (W = 1 and R = 1 with N = 3, say), trade freshness for latency: answers come faster and may be older.

Watch outR + W > N is weaker than it looks. Sloppy quorums, where a write is accepted by stand-in nodes and handed back later (hinted handoff), break the overlap, and concurrent writes to the same item leave no single newest version, so the store needs a rule for resolving the conflict, which Conflict resolution for concurrent writes covers. Even a strict quorum is not strong consistency on its own: a write still in progress can be returned to one reader and missed by the next.

Draw the shop, then take the leader away

Open the Leader-Follower Replication challenge and draw the shop's setup: one leader for writes, followers serving reads, and the promotion path when the leader fails. Label the replication edge synchronous or asynchronous, and next to it write what a failover loses in that mode. Then try Quorum Reads & Writes, where there is no leader: pick N, W and R so that R + W > N, and show the read taking the newest version from the replicas it reached.

Practice this