Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

WebSockets at scale: gateways, routing & draining

Serving many long-lived WebSocket connections: a gateway tier where each connection stays on the instance it reached, a broker or a connection registry to reach a socket held by another instance, sharding by room so one instance owns it, the limits of one instance, and draining on deploy without a reconnect storm.

One process, every socket

A realtime server is easy while it is one process. Take a spreadsheet app whose server holds 40,000 open WebSockets, one per open tab. When someone types a figure into a workbook, the process finds every socket open on that workbook in its own memory and writes the change to each. That loop is the whole design, and it works only because every connection is in the same place.

That is also its weakness. A restart disconnects everyone, and one machine caps how many can connect. Adding servers fixes both and breaks the loop. Putting it back together takes a WebSocket Gateway tier, servers whose job is to accept and hold client connections, and a way for a message to reach the server holding the socket it is meant for. Choosing a WebSocket over a stream or long-polling is a separate decision, made in Realtime transport: long-polling vs SSE vs WebSocket.

Ana's edit never reaches Ben

A WebSocket starts as one HTTP request, the upgrade handshake, and the load balancer chooses an instance only then. After that the connection stays on that instance until it closes, which for a spreadsheet tab can be all day; the balancer cannot move it, and a gateway added later receives only new connections. Now run two gateways, with Ana and Ben both in the Q3 budget:

  1. Ana's handshake lands on gateway 1, Ben's on gateway 2.
  2. Ana changes cell C7, and her frame reaches gateway 1.
  3. Gateway 1 looks for its sockets on the Q3 budget and finds only Ana's.
  4. Ben keeps seeing the old figure until he reloads.

Each gateway knows only its own sockets. Either the instances get connected to each other, or the people who need to hear each other land on the same instance.

A broker between the gateways

The first answer puts a broker in the middle, a publish/subscribe service every gateway connects to. Each workbook gets a channel. A gateway subscribes to the channel of every workbook that has at least one socket open on it, and unsubscribes when the last one closes. When Ana's edit arrives, gateway 1 publishes it to the Q3 channel; the broker hands it to every subscribed gateway, and each writes it to its own sockets on that workbook.

Fan-out across gateways

Gateway 2 holds no socket of Ana's, but it subscribes to the Q3 channel because Ben has the workbook open.

The gateways stay interchangeable, and losing one disconnects only its share of the users. In exchange, every message takes an extra hop through a broker that every edit now depends on, and the broker decides nothing: if Ana and Ben change the same cell at the same moment, something else must choose which value stands, as Conflict resolution for concurrent writes explains. A plain broker such as Redis Pub/Sub also delivers only to the subscribers connected at that moment, so a gateway that loses its link for two seconds misses the edits published meanwhile, and nothing replays them; edits therefore carry version numbers, and when the link returns the gateway has its clients check theirs against the saved workbook and fetch the missing changes or a fresh copy (retention and replay, which a plain broker lacks, are in Pub/sub: topics, subscriptions & replay).

Messages addressed to a person

Some messages belong to no workbook. When Priya shares the payroll workbook with Ben, the notice is for Ben, wherever he is connected. That calls for a connection registry, a fast shared store, often Redis, mapping each user to the gateways that hold their sockets. A gateway writes the entry when a user's first socket on it opens and deletes it when the last one closes; the sharing service looks Ben up and sends the notice to gateway 2 alone. A crashed gateway deletes nothing, so each entry carries an expiry the gateway renews while the socket lives. The registry says where to deliver, not who is online; that has rules of its own, in Presence & ephemeral state, and Worked design: a realtime chat draws a registry and bus for chat.

Or put the whole workbook on one instance

The second answer removes the hop between instances instead of building it. Route each connection by its workbook rather than by load: the client opens /ws/workbooks/q3, and a layer-7 router hashes the workbook id in that URL to choose an instance, with Consistent hashing keeping most workbooks in place when instances come and go. Every socket on the Q3 budget lands on the same instance, which becomes the workbook's owner: the one-process loop again, one per workbook. No broker sits between instances, and because every edit passes through one process, it can put them in a single order before broadcasting.

Routed by workbook, not by load

Both handshakes name Q3, so both sockets land on the instance that owns it, which orders the edits and broadcasts them itself.

NoteRodGrid's own live collaboration takes this route: it shards by Grid, so one stateful edge object holds every connection to a given Grid and broadcasts to them directly. No cross-instance broker is needed, because a Grid's collaborators never span two objects.

The owner carries the costs. Because the router reads the workbook from the handshake, a tab with three workbooks open holds three sockets. A workbook with 3,000 viewers turns every edit into 3,000 writes on one machine, unless relays holding shares of the viewers subscribe to the owner, which is the broker shape again inside one room. And when an owner dies, every socket it held drops at once, and each of its workbooks must be loaded by its new owner before anyone in it is served again. Whenever a workbook moves while its old owner lives, because instances were added or one is draining, new handshakes reach the new owner while the old one still holds sockets. The handover must be explicit: the old owner stops accepting edits and closes its sockets, and the new owner accepts none until it has loaded every edit the old one accepted. A multiplayer match server is the same shape, one owner per match, with a simulation on top that Multiplayer game state: authoritative servers & tick rates describes.

What one instance can hold

Each socket is a file descriptor, and the per-process limit on Linux is often 1,024 by default, so a gateway must raise it or start refusing connections around its thousandth connection. Each socket also holds kernel buffers, TLS state and the application's objects: budget tens of kilobytes per idle connection, so 100,000 connections take a few gigabytes before any message moves, and every broadcast is written once per socket in the room, so send the changed cell, not the whole sheet. Refuse handshakes at a cap below what an instance can really hold: an instance killed for running out of memory loses every connection it holds, not only the newest. A slow client's unsent messages count against that memory too, so cap each socket's outbound queue; Backpressure & load shedding covers what to do when one fills.

Every deploy ends every socket

In an ordinary deploy, a new process does not inherit its predecessor's connections, so every socket on an old instance will close; the only choice is how. Taking gateway 3 and its 38,000 sockets out of service gracefully is called draining:

  1. The deploy marks gateway 3 as draining, and the load balancer stops sending it new handshakes.
  2. Its open sockets keep working.
  3. Gateway 3 closes them in batches, say 2,000 every ten seconds, each with close code 1001, going away, and closes whatever remains at a deadline, since a tab can stay open for days.
  4. The balancer sends each reconnecting client to another instance.

Skip the batches and you get a reconnect storm. Kill gateway 3 at once and 38,000 clients notice within the same second. Each opens a TLS connection, presents a token to check and asks for the state it missed, so the gateways, the auth service and the workbook store take tens of thousands of extra requests in one second. If they slow down, handshakes time out, clients try again, and the storm feeds itself. Rolling deploys add a twist: a client moved off the first instance can land on the second, which is drained next; sending new handshakes only to instances already on the new version moves most clients once.

So the fleet spreads the wave and caps it: batches, headroom for one instance's connections to land on the rest, and refused handshakes past the cap. The other half, how each client waits before reconnecting and resumes from what it last saw, belongs to the client and is worked through in Realtime transport: long-polling vs SSE vs WebSocket.

Size one whiteboard's owner

The Collaborative Whiteboard Challenge asks you to scale a single whiteboard to many concurrent editors and to reconnect after a disconnect. Choose how the connections to one whiteboard meet, a broker between interchangeable gateways or one owner per whiteboard (the Challenge allows a Cloudflare Durable Objects node for that), and put three numbers on the diagram: sockets per instance, instances for the largest whiteboard you plan for, and how long draining one instance takes at your batch size. Chat System poses the other routing problem, a message for one person whose socket lives on another gateway; Collaborative Document Editor is the one-owner-per-document shape, where every edit is ordered in one place.

Practice this