Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Consistency models: strong vs eventual (CAP)

When copies of data disagree, a consistency model says what a read may return. Choose per operation: strong, eventual, or the cheaper guarantees in between, during a partition and on a normal day.

“I clicked Save and got my old name back”

The ticket reads: “I changed my display name, clicked Save, and the page came back with my old name. I saved again and then it worked. Is something broken?”

The write went to the primary database and succeeded. The page reloaded, and the read that fetched the profile went to a replica 300 milliseconds behind, which still had the old name and returned it. By the second Save, the replica had caught up. Why a replica trails its primary, and by how much, is the subject of Replication & read replicas; this article is about what a reader is allowed to see while it does.

Write to primary, read a (possibly stale) replica

Normal operation: a replica read can be stale even with no partition. This is a freshness choice, not the partition-time choice CAP describes.

So is the ticket a bug? It depends on what the system promised. A consistency model is that promise: what a read may return when data is copied across several nodes. At one end sits strong consistency: every read sees the latest write, as if there were a single copy. At the other sits eventual consistency: a read may return an older value, but if writes stop, every copy converges to the same value. This system is eventually consistent, and by that standard it behaved exactly as promised. The user still had a bad experience.

A fix for one user, not for every read

Making every read strongly consistent would close the ticket, and it would cost something on every request. Reading everything from the primary puts all the load back on the one node the replicas were protecting. Making the primary wait until replicas have applied each write before acknowledging it adds a round trip per write, a long one when replicas are far from the primary.

But the user never asked to see everyone's latest data, only their own change. That narrower promise is read-your-writes: a user always sees their own writes, while others may see them a little later. It is much cheaper to provide. The app can send that user's reads to the primary for a few seconds after they write, which works only while replication lag stays under that window, or remember the position of their last write in the replication stream and read only from a replica that has applied it. Everyone else keeps reading replicas.

Add a second replica and a stranger ticket arrives: “I refreshed and saw my new name. I refreshed again and the old one was back.” The first read landed on a replica that had caught up, the second on one that had not. The promise that rules this out is monotonic reads: once a user has seen a value, they never see an older one. A common way to provide it is to send each user's reads to the same replica. That holds while the user stays there; if a failover moves them to a replica further behind, the app also has to remember the position they have already seen, as with read-your-writes.

Read-your-writes and monotonic reads are promises about what one client observes, so they sit between the two ends of the dial. They fix the tickets users actually file without paying for strong consistency everywhere.

When the replica cannot reach the primary

So far the replica was only late. Now the link between the two data centres drops. Both nodes keep running, but neither can reach the other. A network partition like this turns a question of freshness into a forced choice.

A profile read arrives at the cut-off replica. It can answer with what it has, which may be stale, and stay available. Or it can refuse, or wait until the link returns, and stay consistent. It has no third option, because it cannot know what it has missed.

The CAP theorem states that choice: when the network partitions, a distributed store must choose between consistency (every read sees the latest write) and availability (every request that reaches a working node still gets a real answer, not an error). A strongly consistent store keeps reads current and gives up availability during the partition, at least on the side that cannot reach the rest. A store that chooses availability keeps answering, with data that may be stale. Whatever your store does when nodes cannot reach each other is your CAP choice, whether anyone made it on purpose or not, so it is worth deciding before an incident decides it for you. Services that need exactly one owner of something, a lock holder or a leader, face this choice in its sharpest form; Distributed locks & leader election covers them.

The trade-off on a normal day

Partitions are rare. The cost that made the blunt fix unattractive, the extra trip to the primary or the wait for replicas, is paid on every request, with the network perfectly healthy. The two are easy to mix up, and PACELC keeps them apart in one sentence: if there is a Partition, choose between Availability and Consistency; Else, choose between Latency and Consistency.

That second half is why the choice is made per operation, not per database. Some stores expose it per request: Amazon DynamoDB reads are eventually consistent by default, and a request to a table can ask for a strongly consistent read at a higher cost in read capacity (global secondary indexes offer only eventually consistent reads). Inside one application, the choices fall out of what each read is for:

  • A balance or a payment reads from the primary. Strong consistency is worth its latency, and refusing during a partition is safer than showing a wrong number.
  • The profile page after an edit needs read-your-writes, and monotonic reads if there are several replicas.
  • A like counter or a feed can be eventually consistent. Insisting on strong consistency there makes every read pay latency, and turns a partition into an outage for a feature that could have kept working on slightly stale data.

One word causes confusion here. The “consistency” in CAP is about recency across copies; the C in ACID is about a database's rules holding inside a transaction, which Transactions & isolation levels covers. And quorum reads and writes, another way stores tune freshness, are explained with their caveats in Replication & read replicas.

One seat, one buyer

Open the Event Ticketing challenge, where the per-operation choice is the design. Selling a seat needs strong consistency on that one seat: the sale is a single conditional write, sell only if still free, so two buyers can never both get it, and during a partition, a write that cannot reach the seat's authoritative copy refuses rather than guesses. Browsing the seat map can be served from stale copies to survive the on-sale spike, and the buyers who reach checkout are let in at a pace it can finish (Virtual waiting rooms: admitting a crowd fairly). For each operation, write down the guarantee it needs, how stale it may be, and what it does during a partition: refuse, or answer with possibly stale data.

For the other side of CAP, try Offline-First Mobile Sync: a phone with no network is a partition that lasts for hours, the app keeps accepting writes, and the design has to make every device converge when the link returns.

Practice this