One message, 199 deliveries
A message sent to a 200-member room is written once and delivered 199 times. That ratio, more than the message rate, is what shapes a chat system. The product here: one-to-one and group chat; messages arrive in order, survive restarts, and reach offline users later; typing and online indicators. A first estimate, done the way Approaching a system design problem sets out: say 10 million daily users averaging 40 messages: 400 million messages a day ÷ 86,400 s ≈ 4,600, so ~5,000 messages a second on average. Chat traffic peaks well above that, so plan for a few times the average (~15,000/s at 3×). Every message is stored once and pushed to each other member, so deliveries ≈ writes × recipients: about equal for one-to-one chat, 199 pushes for one message in that room.
Yet the defining constraint isn't throughput, it's connections: if a tenth of those users are online at once, that's a million mostly idle WebSockets, more with several devices each, that must be held open cheaply and found quickly when a message needs delivering. The rest of the design comes down to three questions: where the sockets live, who numbers a message, and what happens to a message before anyone sees it.
Sockets at the edge, logic behind them
A short HTTP request/response cannot push: the server waits for the client's next request. This design holds a two-way channel open: clients keep a WebSocket to a gateway tier; Realtime transport: long-polling vs SSE vs WebSocket weighs that choice against the alternatives. Gateways are stateful in exactly one way, they know which users are connected to them, so keep them otherwise dumb: authenticate, hold sockets, forward. Keeping chat logic out of the gateways lets the chat service deploy without touching their sockets; replacing a gateway process still drops the sockets it holds. A session registry (user → gateway, one entry per session, since a user can have several) is where the chat service records each connection and later looks up which gateway holds a recipient's socket; it then publishes to that gateway's topic on a pub/sub bus, so no gateway needs to know about the others. Sizing a gateway and draining one without a reconnect storm belong to WebSockets at scale: gateways, routing & draining; here the gateways just hold sockets.
Who numbers a message
Clients render a conversation by a per-conversation sequence number that the chat service assigns, not by timestamps. With many service instances, that number needs one authority per conversation. Without one, this happens in a group where Lena and Tomás both write at once:
- Lena's message reaches chat service instance 1; Tomás's, a moment later, reaches instance 2.
- Both instances read the conversation's last sequence number: 41.
- Instance 1 stores Lena's message as 42.
- Instance 2 stores Tomás's message as 42. In a store keyed by (conversation, seq), where a write to an existing key replaces it, Lena's message is gone.
So seq comes from one authority per conversation: a single owner per conversation partition (Sharding & partitioning), an atomic counter, or a conditional write that fails if the number is taken. When the write that stores the message also assigns the number, for example inserting the message at (conversation, seq) only if that slot is empty and retrying on a conflict, no number exists without a stored message. With a separate counter, a writer that crashes after taking 42 leaves a gap, which the service closes by storing a skipped marker at 42 and refusing a late write to that number. After a reconnect, a client asks "give me everything after seq N", which, with a push notification to wake the device, is how offline delivery works. Ordering is per conversation, never global: a total order across all chats is an expensive non-requirement.
Persist, then push
A message is stored before it's delivered: the store is the source of truth, delivery is an optimization. Push to sockets before the store acknowledges, and a crash can lose messages that recipients already saw. The service acknowledges the sender once the write is durable, then publishes message available in conversation X to the bus, which both the delivery consumer (members online) and the push consumer (members offline) read; a crash between the write and the publish leaves a stored message nobody was told about, the gap a Transactional Outbox closes. The offline push hangs off the bus, not off the send call: a push provider (Notifications & transactional messaging) is a third party with its own latency and outages, and the sender should never wait on it.
When a piece fails
- Gateway dies → its clients reconnect to another, re-register, and catch up by sequence number. The registry entry is TTL'd (see Presence & ephemeral state), so stale routes heal themselves.
- Delivery bus hiccup → messages are already persisted, but sockets stay open, so connected clients won't reconnect. They need gap detection: a client that sees seq N+2 right after N asks for everything after N (a periodic sync covers a quiet conversation). Push-miss degrades to pull, never to loss.
- Duplicates are possible (retry after an ack is lost). The sender generates a message ID, and the service dedupes on it with a uniqueness constraint on that ID within the conversation, so a retry, even one racing the original, does not become a second stored message; recipients also dedupe by message ID to absorb redelivery. At-least-once + idempotent writes and clients beats chasing exactly-once through the stack; Idempotency & exactly-once illusions explains why.
- Stated trade-off: a write-optimized message store, partitioned by conversation, accepts slower ad-hoc queries; history and catch-up reads are "one conversation, a range of seq", which it serves well.
If the server must not read messages
Some products promise end-to-end encryption: only the participants' devices can read a message. The skeleton survives, because the sequence, the registry, the bus and dedup work on metadata that, in most designs, the server still sees (the conversation, the message ID, the seq). What changes is what the server holds: it stores and routes ciphertext. Anything that needs the text on the server goes away: server-side search of message text, previews the server generates, a push notification whose text the server writes. And delivery is tracked per device rather than per user, because in the common designs each of a member's devices holds its own keys, so a phone and a laptop catch up separately. How the keys get there is End-to-end encryption: keys on devices, a server that can't read.
Two designs that look fine and are not
- Timestamps as ordering. Client clocks skew and servers race; order must be assigned by the server, per conversation (a sequence number or a partition log offset), to survive reconnects.
- Designing group fan-out as N point-to-point sends from the sender's phone. The service fans out server-side; a 200-member room is one send from the sender, not 200 uploads; the service then publishes once per gateway holding members. When an audience is too large to push to at all is a question for Fan-out: push vs pull.
Try it on the Chat System Challenge
In the Chat System challenge, number the edges (1. persist, 2. fan out, 3. push) so the persist-then-push guarantee is visible on the diagram, and hang the push service off the bus, not the chat service. A good run keeps the gateways dumb enough that killing one visibly loses nothing.