Two goals in the same minute
At 20:47 on a Saturday, two goals land in a sports app. Ferro Atlético, a top-flight club, scores in a derby, and 30 million of the app's users follow Ferro. Pradoverde, a fourth-division side, scores against its local rival, and 1,800 users follow Pradoverde. Each goal has to show up in every follower's My feed tab, the list of recent events from the clubs, players and competitions that fan follows, within a few seconds of the goal, at their next refresh.
Getting one event to many receivers is fan-out, and there are two places to pay for it. You can pay at write time: the moment the goal happens, put it into every follower's list. Or you can pay at read time: store the goal once, and build each follower's list when they open the tab. The two goals are the same kind of event with audiences four orders of magnitude apart, and that gap weighs most in choosing between the two.
Push: the list is ready before anyone looks
Push, also called fan-out on write, keeps a precomputed inbox per user: a short list of event IDs, newest first, in a fast in-memory store. When Pradoverde scores, the app looks up the club's 1,800 followers and adds the goal's ID to each of their inboxes. Opening My feed is then one lookup of one list, plus fetching the events behind the first IDs on screen. Reads cost the same however many sources a fan follows.
Then Ferro scores. Same code path, 30 million inserts. Say the fan-out workers sustain 500,000 inserts a second between them: this one goal keeps them busy for a minute, the last Ferro fans see it a minute late, and Ferro's yellow card 40 seconds later, and Pradoverde's goal, wait behind it if they share the queue and the workers take its jobs in order. Push makes a write cost O(audience), and when a few sources have tens of millions of followers, a handful of their events can use up the whole write budget.
Pull: one write, and every open does the work
Pull, or fan-out on read, stores each event once, in a list per source: Ferro's recent events, Pradoverde's recent events. Opening My feed means fetching the recent events of every source the fan follows and merging them by time. A goal costs one write whatever its audience, and nothing is computed for fans who never look.
The work moves to the reader, and to the moment the reader is waiting. A fan here follows about 60 sources, a couple of clubs, their players and two competitions, so every open is 60 reads and a merge, and even with the reads issued in parallel, the slowest of the 60 holds up the response. At half-time in the derby, say 300,000 Ferro fans open the app in the same second: 18 million source reads in that second, and every one of those opens reads Ferro's list, one key read 300,000 times. That concentration has failure modes of its own (Hot keys & cache stampedes). Pull makes a read cost O(sources followed), paid on every open, by everyone.
Splitting by audience
Side by side, the two costs have different shapes. Push spends, per event, one insert per follower, in the background: no request waits on it, though a follower sees the event only once their insert lands. Pull spends, per open, one read per followed source, in the request a person is waiting on. Push is usually the better deal for a source whose audience the workers can absorb quickly, and the worse one for the few whose single event swamps them.
So mix them by audience size, a hybrid. Below a threshold, say 100,000 followers, set from measured worker throughput and the delay you promised, not taste, a source's events are pushed into inboxes; above it, they are only stored, and pulled when someone reads. In this app that sets apart a few hundred big sources, the top clubs, star players and national teams, from everyone else. Opening My feed becomes: read the fan's inbox, read the recent events of the big sources this fan follows, usually a handful, and merge. The big lists are few and read by millions, so they stay in cache, and the tens of millions of inserts per event never happen.
The fans who never look
Forty million people have the app; in a typical week perhaps 10 million open it. Pushing every event to every follower can spend most of the write budget on inboxes nobody reads before the entries scroll out of them. So push only to receivers active recently, say in the last 30 days, and treat the others like a cache miss: when a dormant fan comes back, resume pushing first, then fill their inbox once by pulling, inserting each pulled event into it by ID and event time instead of writing a rebuilt list over it, and trimming the oldest past the bound, so the fill never overwrites an event pushed in between; the bound drops only the oldest entries, as it always does. The price is a slow first open after an absence. Inboxes are bounded, a few hundred IDs each, so a fan back after a month sees the recent slice, not everything since they left.
A goal taken back
At 20:49 the referee reviews the Pradoverde goal on video and disallows it. Under pull this is one write: the stored event is marked disallowed, and reads show it that way once any cached copy of it is refreshed. Under push, the cost depends on what the inboxes hold. If each inbox held a copy of the event, the correction is another 1,800 writes, and for a club with millions of pushed followers it would be millions, a second fan-out that may still be racing the first. If inboxes hold only the ID, the event is fetched by ID whenever a feed is shown, so one write to the stored event corrects every inbox. Deleting works the same way: inboxes can keep a dead ID until a read finds nothing behind it and drops it.
Unfollowing has the same shape. Under pull it takes effect on the next open, because the follow list is read every time. Under push, the inbox still holds IDs from the club the fan just left: either remove them, one write per entry, or filter them out at read time against the current follow list. Following a new club is the mirror image: pull shows its recent events at once, push shows only what happens next unless you backfill the inbox.
Two shortcuts that cost later
- Fanning out inside the request. If the service that records the goal also fills every inbox before it answers, whoever enters match events waits on 1,800 inserts for Pradoverde and on minutes for Ferro. Record the event, enqueue one fan-out job, return, with the job handed to workers as in Message queues & async workers.
- Retries without a delivery key. Workers crash and retry, so a job can run twice. A plain append then shows the goal twice; key the insert by event ID, for example in a set scored by the event's own time, so a repeat changes nothing. Why retries are the normal case is the argument of Idempotency & exactly-once illusions.
Drawing the threshold
The News Feed Challenge requires both models and a plan for accounts with millions of followers, so it is where this split gets drawn under a brief: the push path through a queue and workers into per-user feeds, the pull path at read time for the few accounts above the line, and the threshold written on the edge where they divide. The design around that split, with a social feed's numbers, its ranking stage and its paging, is the next read: Worked design: a news feed.