Green for four days
The ticket reads: Odile has shown as online since the 14th at 18:40. She is on holiday. The app is a workplace chat for a company of 7,500 people, and its first presence design was two lines of code. When a client's WebSocket opens, set users.online = true; when the client sends goodbye before closing, set it to false. At 18:40 on the 14th, Odile's laptop ran out of battery on the train. Nothing sent a goodbye, so nothing ever wrote false.
Presence is this kind of state: who is online, who is typing, where a colleague's cursor sits. It is only true right now, and the moments it stops being true are the ones nobody announces. A crash, a closed lid, a phone losing signal in a lift: none of them says goodbye. Presence has to correct itself.
Silence means offline
So turn the rule around and require proof that someone is still there. Each client sends a heartbeat, a tiny message on its open WebSocket, every 10 seconds, and the gateway holding that socket refreshes a key in an in-memory store with a TTL (time to live): SET presence:odile online EX 30. When the heartbeats stop, the key disappears 30 seconds after the last one, and Odile is offline with no cleanup job. If the gateway itself crashes, its refreshes stop too, and every user it held expires on the same schedule.
The TTL is three heartbeat intervals on purpose. With a TTL equal to the interval, one heartbeat delayed by a slow network expires the key and the dot blinks grey for nothing. With a TTL of several minutes, a dead laptop looks alive for several minutes. You are choosing how many lost heartbeats to forgive, against how long a crashed client keeps its green dot.
Offline can never depend on a goodbye: heartbeat expiry is the backstop that catches crashes; an explicit logout or socket close may only mark offline sooner.
This heartbeat is not the ping that stops a proxy or load balancer from closing an idle socket, which Realtime transport: long-polling vs SSE vs WebSocket covers: that ping keeps the connection open, while the presence heartbeat tells the system a person is still behind it, and one message can do both jobs.
Who notices the key is gone
Expiry fixes the dot for anyone who asks the store, but it tells nobody, and Odile's colleagues expect to be told when she turns grey. Something has to turn the silence into an event:
- Store notifications. Redis can be configured to publish an event when it deletes an expired key. It deletes a key when a client touches it or when its background sampling finds it, so the event can arrive some time after the TTL ran out, and, like any Redis pub/sub message, it is lost if the listener is disconnected at that moment.
- A sweep. Also record each user's last heartbeat time in a sorted set, and have a job fetch, every few seconds, the users whose last beat is older than 30 seconds, then, in one atomic step per user, remove only those whose beat is still that old and declare them offline. It notices a little later but needs no listener connected at the right moment, so many designs keep it as the backstop behind the notifications.
Online now, last seen at 18:40
Last seen on the 14th at 18:40 is the opposite kind of state from the green dot. The dot is ephemeral: true for seconds, rewritten constantly, worthless once it lapses, and kept in TTL'd memory. Last seen is durable: written once, by whatever declares the user offline, with the last heartbeat time kept outside the expiring key, such as in the sweep's sorted set, and expected to survive a restart, so it goes to the database.
Mixing the two fails in both directions. Keep last seen in the TTL'd key, and the expiry that marks Odile offline deletes the very timestamp you wanted to show. Put the dot in the database, and every heartbeat becomes a durable write: 7,500 connected users beating every 10 seconds is 750 writes a second of state that is stale ten seconds later.
Tell the people who are looking
Sending every change to every connected client costs one delivery per online user per change, and since every user also produces changes, the total grows with the square of the user count. So the event goes only to interested subscribers: the members of the channels she is in or, tighter, the clients currently showing her name, which subscribe to her presence while she is on their screen and unsubscribe when she leaves it. The event rides the WebSockets those clients already hold, and finding the gateway that holds each watcher's socket belongs to WebSockets at scale: gateways, routing & draining.
Each device's heartbeat refreshes its entry in the TTL'd store. A change in Odile's status goes only to the gateway holding a watcher's socket, and last seen is written once, when she is declared offline.
The phone in the tunnel
Back from holiday, Odile works from her phone on a train that keeps losing signal. Her socket keeps dropping and coming back a few seconds later. Because a closed socket marks her offline sooner, each drop publishes offline and each reconnection online, and her colleagues watch her dot strobe. The fix is a grace period on the way out:
- Odile's socket closes at 09:00:00.
- The gateway starts a 15-second timer and publishes nothing.
- At 09:00:05 her phone reconnects, to another gateway, and refreshes her key.
- At 09:00:15 the timer fires, finds a heartbeat newer than the close, and drops the offline event.
Coming online is published at once; only going offline waits, so a user who really left stays green 15 seconds longer. And while her laptop is open, the phone losing signal should not turn her grey at all: a person is online while any of their devices is. So presence is kept per device: a hash per user, one field per device holding its last beat, with a TTL on the whole key as a backstop. She counts as online while any field is younger than 30 seconds. A field is removed, in one atomic step that first checks no newer beat has landed, when its beat turns 30 seconds old (the sweep's job) or when its closed device's grace period ends; the offline event and the last seen write come only when no field with a beat younger than 30 seconds remains.
Updates you can afford to lose
Odile is typing… is presence with a lifetime of seconds. Rather than an event per keystroke, the client sends typing at most once every 3 seconds while keys are being pressed, and each receiver shows the indicator until 5 seconds pass without another. No stopped typing message is needed, and a lost event costs at most a brief flicker, since the next one covers for it.
These events are lossy by design: not stored, retried or acknowledged by the application, because a late one is worse than none. Live cursors in a shared document follow the same rule at a higher rate, perhaps ten positions a second per person: send only the latest position, replace any that has not left yet, and when a connection is backed up, drop old positions instead of queueing them behind real messages. A cursor that arrives two seconds late shows where a colleague was.
The rule ends where state stops being disposable: a chat message must not be lost, and a player's position in a game decides who hit whom, so it needs an authoritative server; Multiplayer game state: authoritative servers & tick rates covers that.
Practise on a chat, then elsewhere
In the Chat System Challenge, give presence its own line beside the message path: heartbeats into the gateway, a TTL'd store and events to room members only, with the interval and the TTL on the edges. Your Grid should answer two questions: what tells Odile's colleagues she went offline when her laptop died, and what keeps her dot steady on the train. The same expiry carries a location in Nearby Friends (the radius query is Geospatial indexing & proximity search), live cursors on a lossy path beside the edits in Collaborative Document Editor (how the edits themselves merge is Collaborative text editing: OT vs CRDTs), and the list of who is still in the call in Video Conferencing, whose audio and video belong to Realtime media: WebRTC, SFUs & NAT traversal.