Two plays, one view
A counter is the smallest piece of state most systems have, one integer per item, and it is where correctness and speed collide first. The naive version is three lines of code: read the count, add one, write it back. Two people press play on the same video in the same millisecond, and their requests reach two app servers running those three lines.
- Server A reads the video's count: 48,213.
- Server B reads it too, before A has written anything: 48,213.
- Server A writes 48,214.
- Server B writes 48,214.
Two views, one recorded, no error anywhere. This is a lost update, and on a video going viral it happens constantly, because some request is almost always between its read and its write. Lost updates in general are handled in Transactions & isolation levels. A counter has an easy way out: the application never needs to compute the new value.
Correct, and now everyone queues
Send the arithmetic to the store instead of the result: UPDATE videos SET views = views + 1 WHERE id = 42 in SQL, or INCR views:42 in Redis. That is an atomic increment: the store reads and writes the number as one step, so no other increment lands in between. In PostgreSQL or MySQL at their default isolation levels, the row lock makes the second update wait and then add to the first one's result; Redis executes commands one at a time. Both views are counted.
The waiting is still there. Every increment to video 42 goes through the same row, one after another. If each update holds the row's lock for about a millisecond, until its commit reaches the disk, that row takes at most about 1,000 increments a second, however many app servers send them and however large the database is. At 20,000 views a second, increments arrive twenty times faster than the row can take them; they pile up behind the lock and start to time out. That is a hot spot: a throughput problem on one key, not a correctness one. Redis is far faster per increment, but one key still lives on one node.
One number, stored as thirty-two
If one row takes about 1,000 increments a second, give the video more rows. A sharded counter keeps N sub-counters for one item, here views:42:0 to views:42:31. Each increment adds one to a shard picked at random, and the count is the sum of all of them. With 32 shards, 20,000 increments a second become about 625 per shard, under the ceiling with room for a spike, and in a cluster the shards can sit on different nodes.
Diagram components: Application Service, Redis, Redis, Redis, Application Service.
Each increment touches one shard; each read touches all 32.
The bill arrives on the read side, and in a view counter reads usually outnumber increments. If the page asks for video 42's count 10,000 times a second, that is 320,000 shard reads a second for one number. So the summed total is cached for about a second: 32 shard reads a second per cache, for a count up to a second old, which a view counter can afford. Only hot items are worth splitting, since 32 shards for each of a billion videos would be 32 billion counters. Note what is being split: one hot key, not the dataset. Spreading a whole dataset across machines by a shard key is a different design (see Sharding & partitioning), and a key that is hot for reads rather than writes needs the defences in Hot keys & cache stampedes.
Stop sending every +1
Sharding spreads the increments; write combining sends fewer of them. Each app server keeps an in-memory map from item to pending increments and flushes each entry once a second as one write: UPDATE videos SET views = views + 347 WHERE id = 42, or INCRBY views:42 347. With 50 app servers sharing 20,000 views a second, each sees about 400 a second for video 42, and the row receives 50 writes a second instead of 20,000. The same batching instinct runs through the firehose designs in High-write ingestion.
Two things can go wrong. A crashed server takes its unflushed deltas with it: with a one-second flush, up to about 400 views of video 42, plus its share of every other video. A clean shutdown can flush first; a killed process cannot, so the flush interval sets how many views you are prepared to lose. And a flush that times out leaves the server unable to tell whether the 347 landed: send them again and they may count twice, drop them and they are lost. Atomic is not the same as idempotent, and that was already true of the single INCR.
Write the view down, then count it
To stop losing views and keep batching, make each view durable first and do the arithmetic afterwards. The API appends every view to a durable log such as Kafka and acknowledges it only after the append succeeds. A stream processor consumes the log, counts per video in one-minute windows, and writes each closed window's total to a durable store. If the processor crashes, the events are still in the log and it resumes from the last position it recorded. That resume is where views get counted twice, because a log consumed this way delivers events at least once:
- The processor closes the 12:03 window for video 42 at 412 views and adds 412 to the video's total.
- The processor crashes before recording that it has read past those events.
- On restart it reads the same events again and adds 412 a second time.
The fix is in the shape of the write. total = total + 412 changes the result each time it runs; views(42, 12:03) = 412 leaves the same row however often it runs. Store one row per video per window as an upsert, and derive the total from the windows. That holds if the restarted job rebuilds each window from the same events, which is why stream processors typically checkpoint their window state together with their log position, a mechanism Stream processing: windows, watermarks & state describes. A retried append from the API is the other source of copies; an id minted once per view lets the processor drop them (the Idempotent Consumer pattern).
Two numbers, on purpose
The log path is exact but a minute or more behind, and the page wants a number now. So each view takes two paths: an atomic increment in Redis, sharded where needed, and an append to the log. The page reads Redis; reports, and anything that pays money, read the totals rebuilt from the log. This is approximate now, exact later.
Diagram components: React, API Gateway, Redis, Kafka, Stream Processor, Cassandra, Admin Console.
Every view takes both paths. The page reads the fast counter; reports read totals rebuilt from the log.
The fast number is allowed to drift. Redis replicates asynchronously by default, so a failover can drop increments the replica had not received (Replication & read replicas has the details); a retried increment can land twice; a crashed server's combined deltas never arrive. None of that touches the log, so the exact count can be recomputed from it for as long as the events are kept, and the fast counter is never the source of truth.
Users accept the drift because they cannot see it: "1.2M views" covers a range 100,000 views wide, whichever way the page rounds. A payment to a creator cannot hide an error that way, so it is computed from the log after duplicates and invalid views are removed, to the standard of Usage metering & billing pipelines. Ad budgets add a ceiling to the count, and Ad serving & real-time auctions shows how they are split into local leases.
When the question is who
A poll that allows one vote per user needs something a counter cannot hold: who has voted. Store each vote as a row with a unique key on (poll, user), count it only if the insert succeeds, and the tally becomes an ordinary counter beside it, one a recount of the rows can repair if a crash falls between the two. The uniqueness check spreads by itself, one row per user; the tally is the row every vote hits, so in a viral poll that is the part to shard or combine. And when the question is only how many different people, HyperLogLog estimates it in a few kilobytes without storing anyone (Data structures behind big systems), though it can never enforce one vote each.
Distributed View Counter asks for the system this article built: very high increment throughput, approximate counts now and accurate ones eventually, no lost updates. A good design answers three questions without being asked: which store the page reads, what a crash of the fast counter loses, and why the aggregator can restart without counting anything twice. Online Polling System is the poll, where the interesting choice is which of the two stores a viral spike actually strains.