System design

How a chat message reaches someone who is offline

Alice sends a message while Bob's phone is off. Whether it reaches him, once and in order on every device, depends on knowing where he is: on another server, nowhere, or on two devices at once. Five versions of a chat backend, from one server to a group of 500.

Jump to the final architecture

Alice types “Are we still on for Saturday?” and presses send. Bob's phone is in a drawer with a dead battery, and his laptop is closed on the desk. An hour later he plugs the phone in, and the message should be there: once, in the right place in the conversation, and on the laptop too when he opens it.

So where is Bob? He might be connected to the same server as Alice, to a different one, to none, or to two at once from two devices. Each of those places needs its own way for the message to reach him, and most of a chat backend is the sum of those ways.

The rest of this post follows that one message. Each section below is a case the version before cannot handle, mostly a place Bob can be, and the smallest change that handles it, with what it costs. If you know chat backends, the headings tell the story and the last drawing is the result; if not, start where everything runs on one server. Learn has the finished design, a worked realtime chat; this post is about how a design like that comes about.

Two million people, half a million sockets

The product is one-to-one and group chat in a mobile app. Every number here is made up for the exercise and rounded to keep the arithmetic easy; none is measured from a real service:

  • 2 million daily users. At the evening peak about 300,000 are connected at once, many from two devices, so call it half a million open connections.
  • About 60 messages per user per day: 120 million a day, about 1,400 a second on average, and a few times that at the evening peak, say 4,000 a second.
  • One chat server holds about 50,000 mostly idle WebSocket connections. That is a sizing assumption for this exercise; the real figure depends on memory per connection, TLS and what the server does with each one.
  • Groups have at most 500 members. That is a product limit, stated as one.
  • A stored message takes about 200 bytes with its IDs and timestamps: some 24 GB a day.

Look at the first and third lines together. Half a million connections at 50,000 per server is ten servers just to keep everyone connected, before a single message is sent. A few thousand messages a second is ordinary work for one database. With these numbers, open connections run out long before messages per second do.

Before drawing anything, three time limits, all assumptions. Alice should see “sent” well under a second after pressing send, and only once the message is stored. A connected Bob should see it within a second. An offline Bob should have a notification sent to him within seconds, and find the whole conversation there when he opens the app. The third limit has two halves for a reason: the notification only tells Bob to look, and the store is what actually hands him the message.

Everyone on one server

Start with the case where everyone is in the same place. Alice's phone and Bob's phone each keep a WebSocket open to one chat server. A WebSocket is a connection that stays open in both directions, so the server can send Bob a message the moment it has one instead of waiting for his phone to ask. An ordinary request and response cannot do that: the server answers only when the client asks.

When Alice's message arrives, the server does three things, in this order: it writes the message to the message store, it acknowledges Alice (her phone shows “sent”), and it writes the message to Bob's socket. The order is the first real decision in the design. If the server delivered first and stored second, a crash between the two would leave a message Bob has already read that the system no longer has: missing from Alice's history, and from any device Bob uses next. Store first, then deliver, and every message anyone has seen is a message the system still has.

Stage 1 of 5: One server: store, then deliver
One server: store, then deliverAlice's phone and Bob's phone each hold a WebSocket to a single chat server. The chat server writes each message to the message store, a relational database, and then sends it down Bob's WebSocket.WebSocketAlice's phoneSPA / MobileCLIENTS & UIChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESBob's phoneSPA / MobileCLIENTS & UI
One server: store, then deliverAlice's phone and Bob's phone each hold a WebSocket to a single chat server. The chat server writes each message to the message store, a relational database, and then sends it down Bob's WebSocket.WebSocketAlice's phoneSPA / MobileCLIENTS & UIChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESBob's phoneSPA / MobileCLIENTS & UI

The chat server stores the message before it writes it to Bob's socket.

Learn moreRealtime transport: long-polling vs SSE vs WebSocketWebSockets at scale: gateways, routing & draining

The message store can be an ordinary relational database: a table of messages with the conversation, the sender, the text and a timestamp. And Bob's dead phone? In this version, Alice's message waits in the store, and when Bob next opens the app, it loads the recent messages of each conversation. So Bob does get it eventually, if he happens to open the app. Nothing tells him to.

For a launch with a few thousand people online, this is enough. One server holds every socket, finds Bob's in its own memory, and writes to it. The costs are already visible, though. The server is a single point of failure: when it restarts, every user is disconnected at once. And it has a ceiling that has nothing to do with messages: at around 50,000 connections it is full, and our evening peak needs ten times that.

Bob is on another server

So there are several chat servers now, about a dozen for the evening peak, which leaves room to lose one, with a load balancer in front choosing a server for each new connection. A WebSocket stays on the server where it was opened until it closes, so once the load balancer has chosen, that is where the user lives.

Replay Alice's message, with Bob's phone switched on this time. Alice's connection landed on one server; Bob's landed on another. Alice's server stores the message, acknowledges her, looks for Bob's socket in its own memory and finds nothing. Bob is connected, just not here.

The servers need a way to pass messages to each other, and the simplest one is publish/subscribe (pub/sub). Every user gets a channel on a pub/sub broker. When Bob's socket opens on a server, that server subscribes to Bob's channel, and it unsubscribes when his last socket there closes. Alice's server does not need to know where Bob is: it publishes the message to Bob's channel, the broker hands it to whichever server subscribed, and that server writes it to Bob's socket.

Stage 2 of 5: Bob is on another server
Bob is on another serverA load balancer now spreads connections across several chat servers. Alice's chat server stores the message, then publishes it to Bob's channel on a pub/sub service; another chat server, the one holding Bob's WebSocket, is subscribed to that channel and delivers the message to Bob's phone.to Bob's channelWebSocketAlice's phoneSPA / MobileCLIENTS & UILoad BalancerTraffic RouterPLATFORM & RUNTIMEChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESUser channelsPublish / subscribeMESSAGING & WORKFLOWSAnother chat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISBob's phoneSPA / MobileCLIENTS & UI
Bob is on another serverA load balancer now spreads connections across several chat servers. Alice's chat server stores the message, then publishes it to Bob's channel on a pub/sub service; another chat server, the one holding Bob's WebSocket, is subscribed to that channel and delivers the message to Bob's phone.to Bob's channelWebSocketAlice's phoneSPA / MobileCLIENTS & UILoad BalancerTraffic RouterPLATFORM & RUNTIMEChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESUser channelsPublish / subscribeMESSAGING & WORKFLOWSAnother chat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISBob's phoneSPA / MobileCLIENTS & UI

Alice's server publishes to Bob's channel; the server holding Bob's socket is subscribed to it.

Learn moreLoad balancing & horizontal scalingPub/sub: topics, subscriptions & replay

A plain broker such as Redis Pub/Sub keeps nothing: it delivers to whoever is subscribed at that instant, and a server whose link drops for two seconds misses those two seconds for good. That sounds alarming and is acceptable here, for the reason the one-server version already gave: the message was stored before it was published. The broker is a fast path to a connected Bob, not the place his messages live. What does need care is how Bob gets back what a fast path dropped.

What happens when a chat server dies with 50,000 sockets? Every one of those clients notices within seconds and reconnects through the load balancer to the other servers, which subscribe to their channels again. Messages published to those channels in the gap reached nobody, but they are in the store, and each phone loads what it missed when it reconnects. One detail decides whether that works: a reconnecting phone should subscribe first and ask the store second. In the opposite order, a message stored between the two steps is neither in the answer nor delivered live.

The reconnection itself is the second cost of several servers. Fifty thousand phones that all reconnect in the same second, each opening an encrypted (TLS) connection, checking a token and asking the store for what it missed, can overload the servers that are still standing; if those slow down, handshakes time out, phones retry, and the load feeds itself. Clients wait a random, growing delay before reconnecting (backoff with jitter), and deploys close sockets in batches rather than all at once (draining). That failure has a post of its own on this blog, linked at the end. It also explains the dozen servers rather than ten: the survivors need room for one server's connections.

Bob is offline

Now the case the post opened with. Bob's phone is off. Alice's server stores the message, acknowledges her and publishes to Bob's channel. Nobody is subscribed, so the broker delivers it to nobody, and that is the end of it. Nothing in this design records that Bob is offline, so nothing tells him there is a message waiting. The message is safe in the store and invisible until he opens the app on his own.

To notify Bob, the server has to know that he is connected nowhere. That is the job of a connection registry: a fast key-value store that maps each user to the chat servers holding a socket for them. A server writes Bob's entry when his first socket opens there and removes it when the last one closes. A server that crashes removes nothing, and a phone whose battery dies closes nothing, so every entry carries a time to live (TTL) that the server renews only while the phone keeps answering its heartbeats; when either goes silent, its entries expire within seconds. The same registry can answer “is Bob online?” for the green dot other users see, though a trustworthy dot needs heartbeats and a grace period, which Learn's article on presence covers.

With the registry, Alice's server can tell the two cases apart. If Bob has an entry, the publish should reach him, and no push is needed. If he has none, the server puts a push job on a delivery queue, and a notification service takes jobs off the queue and sends a push notification through the phone platform's push service: Apple Push Notification service (APNs) for iPhones, Firebase Cloud Messaging (FCM) for Android. The dashed edge to Bob's phone in the drawing stands for that hop through Apple or Google.

Stage 3 of 5: Bob is offline
Bob is offlineA connection registry, a key-value store, now records which chat server holds each user's sockets; the server holding Bob's socket writes his entry. Alice's chat server looks Bob up there, and when he has no entry it puts a push job on a delivery queue. A notification service takes the job and sends a push notification to Bob's phone through the phone platform's push service.to Bob's channelWebSocketpush jobAlice's phoneSPA / MobileCLIENTS & UILoad BalancerTraffic RouterPLATFORM & RUNTIMEChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESDelivery queueQueueMESSAGING & WORKFLOWSUser channelsPublish / subscribeMESSAGING & WORKFLOWSConnection registryGeneric KV lookupsDATA STORESAnother chat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISNotification serviceNotificationservicePush / EmailSERVICES & APISBob's phoneSPA / MobileCLIENTS & UI
Bob is offlineA connection registry, a key-value store, now records which chat server holds each user's sockets; the server holding Bob's socket writes his entry. Alice's chat server looks Bob up there, and when he has no entry it puts a push job on a delivery queue. A notification service takes the job and sends a push notification to Bob's phone through the phone platform's push service.to Bob's channelWebSocketpush jobAlice's phoneSPA / MobileCLIENTS & UILoad BalancerTraffic RouterPLATFORM & RUNTIMEChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESDelivery queueQueueMESSAGING & WORKFLOWSUser channelsPublish / subscribeMESSAGING & WORKFLOWSConnection registryGeneric KV lookupsDATA STORESAnother chat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISNotification serviceNotificationservicePush / EmailSERVICES & APISBob's phoneSPA / MobileCLIENTS & UI

No entry in the registry means no socket anywhere: the server queues a push job instead.

Learn morePresence & ephemeral stateNotifications & transactional messagingMessage queues & async workers

Why a queue between the chat server and the notification service? Because the push service belongs to someone else, with its own latency and its own bad days, and Alice should never wait on it. Her message is stored and acknowledged whether or not Apple answers quickly.

The push tells Bob to look, but it does not deliver the conversation. APNs and FCM do not promise that every notification arrives: a phone that was off for hours may get some late, several collapsed into one, or none at all. So delivery stays where it already was. When Bob opens the app, his phone asks the store for every message after the last one it has. In this version, the chat server stamps each message with its own clock when it stores it, and “the last one” means the timestamp of the newest message on the phone. Keep an eye on that cursor; the next section takes it apart.

The new boxes fail in ways worth naming:

  • The push service is slow or down. Jobs pile up in the queue and notifications arrive late. Chat itself is unaffected, and Bob still gets everything when he opens the app.
  • The registry is stale. For a few seconds after a chat server crashes, Bob's entry still points at it. A message sent in that window is published to Bob's channel, which no live server is subscribed to any more, and no push goes out, because the registry said he was connected. When his phone reconnects and catches up, the message is there. A stale registry costs a notification, not a message.
  • The registry is down. The server cannot tell connected from offline, and it has to guess one way. This design treats everyone as offline: it still publishes to the recipient's channel, and it also queues a push. The cost is redundant notifications for people who already saw the message; the other guess would cost missed notifications for people who did not, which is worse. Live delivery never asks the registry, so it carries on.

Bob on two devices, and a message sent twice

Bob also has a laptop. Some evenings both devices are connected at once, and the routing already handles that: in the drawing the laptop is on the same server as the phone, which subscribed to Bob's channel when his first socket arrived; on any other server, that server would subscribe to his channel too. Tonight is the other case: the laptop was open earlier in the evening, while the phone was already dead in the drawer. What breaks now is subtler: three ways the same message can reach Bob wrongly, and each one is fixed by a rule rather than by a new box.

Twice. Alice's message is stored, but the acknowledgement never reaches her phone; she went into a tunnel. Her phone does what it should and sends the message again, and Bob sees it twice. The fix starts on Alice's phone: it gives every message an ID when she presses send, and a resend carries the same ID. The store refuses a second message with an ID it already holds in that conversation and answers the resend with the original's acknowledgement, and every client ignores a message whose ID it already shows. Resending until an acknowledgement arrives makes delivery at least once; deduplication by ID turns that into shown once.

Out of order, or with holes. The timestamp cursor from the offline case loses messages, and not rarely. Two chat servers' clocks differ by a few milliseconds, so a message stored later can carry an earlier timestamp. Two messages can share the same millisecond. And a message whose write takes a moment longer to commit can appear in the store after Bob's phone has already read past its timestamp; asking “everything after the newest timestamp I have” then skips it forever.

The fix is a sequence number per conversation, assigned by the store in the same write that stores the message: 41, 42, 43, with no two messages sharing one. In a relational store, that can be a counter on the conversation's row, incremented in the same transaction that inserts the message, so concurrent writes to one conversation take their numbers one after another. That gives each conversation exactly one authority for its order. We assume even a busy group sends far fewer messages a second than one row can take updates, so waiting for that row costs nothing noticeable at these numbers.

Each device then keeps its own “last sequence number I have” for each conversation. Bob's phone died knowing up to 40; the laptop, open earlier in the evening, knows up to 41, the last message before Alice's question. When the phone reconnects, it asks for everything after 40 in that conversation; when the laptop is opened, it asks for everything after 41. The two catch up independently, since each may have missed different messages.

A gap while connected. The broker can drop a message while Bob's sockets stay open, so his phone never reconnects and never catches up. Sequence numbers make that visible: a device that receives 41 and then 43 knows 42 exists, asks the store for it, and keeps its cursor at 41 until 42 arrives; a cursor that jumped to 43 would skip 42 for good if the device died first. A gap at the very end of a quiet conversation has no next message to reveal it, so clients also ask the store now and then for the latest sequence number of every conversation.

Stage 4 of 5: Once, in order, on every device
Once, in order, on every deviceBob's laptop now holds a WebSocket to the same chat server as his phone, which delivers each message to both. The rest of the system is unchanged; this stage adds rules, not components: client message IDs, per-conversation sequence numbers assigned by the message store, and a catch-up cursor kept by each device.to Bob's channelWebSocketpush jobAlice's phoneSPA / MobileCLIENTS & UILoad BalancerTraffic RouterPLATFORM & RUNTIMEChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESDelivery queueQueueMESSAGING & WORKFLOWSUser channelsPublish / subscribeMESSAGING & WORKFLOWSConnection registryGeneric KV lookupsDATA STORESAnother chat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISBob's laptopSPA / MobileCLIENTS & UINotification serviceNotificationservicePush / EmailSERVICES & APISBob's phoneSPA / MobileCLIENTS & UI
Once, in order, on every deviceBob's laptop now holds a WebSocket to the same chat server as his phone, which delivers each message to both. The rest of the system is unchanged; this stage adds rules, not components: client message IDs, per-conversation sequence numbers assigned by the message store, and a catch-up cursor kept by each device.to Bob's channelWebSocketpush jobAlice's phoneSPA / MobileCLIENTS & UILoad BalancerTraffic RouterPLATFORM & RUNTIMEChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESDelivery queueQueueMESSAGING & WORKFLOWSUser channelsPublish / subscribeMESSAGING & WORKFLOWSConnection registryGeneric KV lookupsDATA STORESAnother chat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISBob's laptopSPA / MobileCLIENTS & UINotification serviceNotificationservicePush / EmailSERVICES & APISBob's phoneSPA / MobileCLIENTS & UI

No new service: message IDs, per-conversation sequence numbers and a cursor per device.

Learn moreMessage DeduplicationIdempotency & exactly-once illusions

The sequence number also says how the store will grow. At 24 GB a day, about 9 TB a year, it will outgrow one database sooner or later, and when it does, the conversation is the natural partition key: every catch-up above reads one conversation, and the sequence authority for a conversation lives in the same partition as its messages. That is a future step, not this one, so the drawing keeps one box.

One question this leaves to the product: when Bob is connected on the laptop and his phone is off, should the phone also get a notification? In this design it does not, because the registry is asked whether Bob has a socket anywhere, not on every device. Messaging apps answer that question differently, and either answer works with the rest of the design.

Alice writes to 500 people

Alice asks the same question in her neighbourhood group, 500 members: “Are we still on for Saturday?” Run that through the design so far and the send path becomes a loop: for each of the 499 other members, look them up in the registry, publish to their channel, and queue a push for each one who is offline. The chat server's work per message now grows with the size of the group, on the same servers that hold everyone's sockets. Worse, the server acknowledges Alice once the message is stored, before that loop finishes; a server that crashes in the middle of it has already told Alice “sent”, and the pushes it had not queued yet are simply gone. That gap was already there in the one-to-one design, between storing a message and queuing its push; a group stretches the window across the work for 499 members.

The fix is to take the fan-out off the send path. The chat server stores the message and, in the same database transaction, records one job: deliver message 42 of this group. A small relay in the chat servers moves recorded jobs onto the delivery queue. Writing the job in the same transaction as the message is what makes the acknowledgement safe: if the message is stored, its job exists, even if the server dies the next instant. This is the transactional outbox pattern, and it costs one more table in a database the server already writes to. It closes the one-to-one gap too, because every message's job is recorded this way now.

It also answers a question the delivery queue raised when it arrived: what if the queue is down? The relay cannot move jobs, so they wait in the outbox table. Alice keeps sending while the outbox table grows, and notifications and group messages arrive when the queue is back. One-to-one messages, which also keep a direct publish, still arrive live.

Fan-out workers take jobs off the queue and do the loop away from the servers holding the sockets. The job carries only the message and the group. A worker reads the member list from the store, publishes the message to each member's channel, asks the registry in one batched lookup which members have no socket, and hands those to the notification service. A worker acknowledges a job only once its loop is done and the notification service has stored the pushes it will send, and the relay marks a job as moved only once the queue has stored it, so a crash on either side repeats work instead of losing it: a member may be sent the message twice, which its ID absorbs, or get a second notification. Those reads and the registry lookup are not drawn, to keep the drawing free of crossing lines. Asking the registry when the job runs, rather than when Alice sends, also means a worker that is behind decides on who is connected now, not on who was connected when the message was sent. The member list works the other way round: the worker reads the group as it stood at message 42 (each membership records the sequence numbers where it starts and ends), so someone who joins a second after Alice sends is not handed a message from before they joined. Push jobs for one-to-one messages travel the same way now, as jobs with a single recipient, so the chat servers stop reading the registry: they only write entries for the sockets they hold, and the workers do the reading.

Stage 5 of 5: Fan-out off the send path
Fan-out off the send pathThe final architecture. For a group message the chat server stores the message and one delivery job, which reaches the delivery queue. Fan-out workers take each job, publish the message to every member's channel on the pub/sub service, and pass the members with no open connection to the notification service, which sends the push notifications.to Bob's channelWebSocketone job per messageoffline membersAlice's phoneSPA / MobileCLIENTS & UILoad BalancerTraffic RouterPLATFORM & RUNTIMEChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESDelivery queueQueueMESSAGING & WORKFLOWSUser channelsPublish / subscribeMESSAGING & WORKFLOWSConnection registryGeneric KV lookupsDATA STORESFan-out workersConsumerMESSAGING & WORKFLOWSAnother chat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISBob's laptopSPA / MobileCLIENTS & UINotification serviceNotificationservicePush / EmailSERVICES & APISBob's phoneSPA / MobileCLIENTS & UI
Fan-out off the send pathThe final architecture. For a group message the chat server stores the message and one delivery job, which reaches the delivery queue. Fan-out workers take each job, publish the message to every member's channel on the pub/sub service, and pass the members with no open connection to the notification service, which sends the push notifications.to Bob's channelWebSocketone job per messageoffline membersAlice's phoneSPA / MobileCLIENTS & UILoad BalancerTraffic RouterPLATFORM & RUNTIMEChat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISMessage storeGeneric SQL storeDATA STORESDelivery queueQueueMESSAGING & WORKFLOWSUser channelsPublish / subscribeMESSAGING & WORKFLOWSConnection registryGeneric KV lookupsDATA STORESFan-out workersConsumerMESSAGING & WORKFLOWSAnother chat serverRealtime server (WebSocket / SSE)Realtime server(WebSocket / SSE)SERVICES & APISBob's laptopSPA / MobileCLIENTS & UINotification serviceNotificationservicePush / EmailSERVICES & APISBob's phoneSPA / MobileCLIENTS & UI

One durable job per message; workers do the per-member work.

Learn moreFan-out: push vs pullTransactional OutboxWorked design: a realtime chat

One-to-one messages keep their direct publish to the recipient's channel, so the common case gets no slower. Group messages take a few more milliseconds to reach connected members, because they pass through the queue and a worker first. In exchange, the queue becomes the place a backlog shows: if the workers fall behind, group messages and notifications arrive late, while one-to-one messages still arrive live.

The workers publish once per member. A larger system would group members by the server the registry says holds them and publish once per server, far fewer messages for the broker; the price is that the registry moves onto the live path, and a registry outage then stops live delivery instead of only causing extra notifications. At 500 members, this design keeps the registry out of the live path: a registry outage still costs only extra notifications.

There is a size beyond which pushing a copy to every member stops making sense at all: a channel followed by hundreds of thousands of people. There, readers fetch the channel's messages when they open it, instead of each one being sent a copy. That is a different product, with its own trade-offs between push and pull, and this design does not need it at 500 members.

Where Alice's message can still get stuck

One last time, Alice's question, through the whole drawing. Her phone gives it an ID and sends it through the load balancer to a chat server. The server stores it as number 42 of the conversation, records its delivery job in the same transaction, and tells Alice “sent”. It publishes to Bob's channel; nobody is subscribed. A worker picks up the job, finds no entry for Bob in the registry, and hands a push to the notification service, and Apple passes it on once Bob's phone is back on, or doesn't. An hour later the phone reconnects, subscribes, and asks for everything after 40. The laptop, opened the next morning, asks for everything after 41. The message shows once on each device, in its place, and nothing in its route had to work the first time except the store.

Twelve boxes: three are Alice's phone and Bob's phone and laptop, and the two chat servers stand for a dozen. Here is what happens when each of the others breaks, and who notices:

  • A chat server dies → its users are disconnected, reconnect with jittered backoff to the other servers, subscribe again and catch up by sequence number. Messages sent meanwhile were stored; nobody loses one.
  • The load balancer fails → phones can no longer reach any chat server. Use a managed, redundant one, or run two.
  • The broker lags or drops messages → live delivery is late or patchy. Gap detection and periodic sync fetch the missing messages from the store.
  • The registry is stale → a notification can be missed. The registry is down → everyone is treated as offline: extra notifications, live delivery unaffected.
  • The delivery queue is down, or the workers fall behind or crash → jobs wait in the outbox or the queue, and a crashed worker's job runs again; notifications and group messages are late, one-to-one messages still arrive live, and Alice keeps sending.
  • The notification service or the push service behind it is slow or down → notifications are late or missing; everything is in the app when it opens.
  • The message store is down → sending stops. Alice's phone keeps the message and retries with the same ID. It is the one dependency the send path cannot work around, because every guarantee in this design starts with “the message is stored”. Run it replicated, with each write on a second machine before Alice sees “sent”, or a failover can lose messages she was told were sent.

Apart from the load balancer, which only has to be redundant, the store is the one box whose failure stops Alice from sending, and no failure on the list can lose a message she was told was sent. The broker and the push service are fast paths that are allowed to fail, because behind each of them the store still holds the message, and every device knows how to ask for it.

Not in this design

  • End-to-end encryption. The routing in this post survives it; what changes is what the server can read, which the worked chat design in Learn and the End-to-End Encrypted Messaging Challenge cover.
  • Read receipts and typing indicators. Both are short-lived state about a conversation rather than messages in it. A typing indicator can be lost without harm; a read receipt is closer to a message and usually stored.
  • Photos, video and files. Attachments go to object storage, and the message carries a reference to them, not the bytes.
  • Voice and video calls, which are realtime media with their own protocols.
  • Search over message history, a separate index fed from the store.
  • Several regions. Bob and Alice are on the same continent here.

Draw it yourself

The Chat System Challenge asks for this architecture. Its palette names products (MongoDB, Redis, Kafka, a push provider) where this post draws generic boxes; a Kafka topic can play the delivery queue. Draw it, then send Alice's question to a Bob whose phone is off.