Two seconds behind the teacher
A language school moves its lessons online. The prototype sends each browser's camera and microphone, in small encoded chunks, over the WebSocket that already carries the lesson chat. In the first real lesson, with the student on a hotel's Wi-Fi, her picture freezes, catches up in a rush, and soon the two answer each other two seconds late.
A WebSocket runs over TCP, which delivers every byte, in order. One lost packet holds back everything behind it until the retransmission arrives, at least a round trip later, a stall called head-of-line blocking; Network protocols: TCP, UDP, HTTP/2 & HTTP/3 shows what else TCP's guarantees cost. And when the hotel's link briefly carries less than the stream needs, chunks queue in the sender's buffer, and TCP delivers each however late, so the call falls behind.
A conversation stays comfortable while one-way delay is under roughly 150 ms, and a frame that arrives after its moment to be shown is worth less than one that never arrives. For a call, late is the failure.
A transport that is allowed to lose
So live media usually travels over UDP, which sends each packet on its own, with no retransmission and no ordering. Browsers do it through WebRTC, a set of browser APIs and protocols for realtime audio, video and data, which numbers, timestamps and encrypts every media packet. The receiver holds packets briefly in a jitter buffer to reorder them and even out their timing, and does not wait past that for a missing one: audio conceals the gap, and video may ask for a lost packet again while it can still arrive in time, or for a fresh keyframe.
WebRTC leaves the introductions to you
First the browsers must agree on what to send and where, an exchange called signaling that WebRTC leaves to the application. The teacher's browser creates an offer, a session description in a text format called SDP: its codecs, the streams it will send, and the fingerprint of the certificate that will authenticate the encrypted connection. The school's server relays it, and the student's browser replies with an answer. The usual carrier is the WebSocket the app already holds to each participant (WebSockets at scale: gateways, routing & draining). Signaling also carries addresses, and the student's laptop does not know its own.
The address the laptop cannot see
The laptop has 192.168.1.37, a private address the hotel's router rewrites to its public one as packets leave (NAT, network address translation); nobody outside can send to 192.168.1.37. ICE (Interactive Connectivity Establishment), part of WebRTC, works around it: each browser collects the addresses it might be reached at, its candidates, sends them through signaling, and tests pairs until one works.
- Host candidates are the device's own addresses, useful when both sides share a network.
- Server-reflexive candidates come from a STUN server, which replies with the public address and port the request arrived from, an opening the other side can often send through.
- Relay candidates are addresses on a TURN server, which forwards packets for the client both ways.
ICE keeps a working pair, preferring direct over relayed. This hotel's router assigns a different public port per destination, so the port STUN reported is not the one the teacher would reach; the teacher's home router drops packets from addresses it never sent to, and every direct pair fails. The relay works. Some company and school networks block UDP outright; then a TURN server listening on TCP or TLS, often on port 443, still gets through, and that leg is back on TCP, with the stalls from the opening.
The offer, the answer and the candidates travel over the app's WebSocket. Direct pairs fail at the hotel's NAT, so the media goes through the TURN relay.
TURN carries every byte of a relayed participant's media, so its bandwidth bill grows with the calls it serves; run it in several regions, give clients short-lived credentials through signaling, and measure what share of your calls needs it.
Six people, five copies each
Then come conversation classes: a teacher and five students. Keeping the two-person shape means every browser connects to every other, a full mesh, and each sends its camera five times. At an assumed 1.5 Mbps per 720p stream, every laptop uploads 7.5 Mbps, more than many home or hotel uplinks sustain, and the browser usually encodes separately per connection, so CPU climbs too. Mesh suits two or three people; past a handful, uplinks and laptops usually give out.
Send once, let the server choose
A Selective Forwarding Unit (SFU) changes the shape. Each browser sends its streams once, to the SFU, which forwards each one to the participants who should get it. It does not decode the video: it reads packet headers and routes the packets, so a forwarding hop costs little CPU and adds little delay. Each upload falls from five copies to one.
Download still grows with the room, which is where selective matters: in a 50-student lecture, each screen shows the teacher and a few thumbnails, so the SFU forwards those, not 49 videos. Each browser still reaches the SFU through signaling and ICE, and with the SFU on a public address, the hotel problem eases: the laptop's outgoing packets usually open the path, and a relay is needed mainly where UDP is blocked. The SFU terminates each participant's encrypted connection, so it can read the media unless the app adds end-to-end encryption on top, with keys only the participants' devices hold (End-to-end encryption: keys on devices, a server that can't read). The older MCU (Multipoint Control Unit) instead decodes everything and mixes it into one stream per client, which spares weak clients at the cost of heavy server CPU, extra delay and a layout the server chooses.
Recording keeps that heavy work off the forwarding path too. The SFU sends a copy of the streams to a recorder, which encodes at its own pace into object storage, so a stalled recorder spoils the recording, not the class; turning that file into renditions for playback is Media processing & transcoding.
A phone on the bus in the same class
One student joins from a phone on a weak signal, and the teacher's single 1.5 Mbps stream stalls it. With simulcast, the teacher's browser encodes the camera two or three times, say at 720p, 360p and 180p, sends them all to the SFU, and the SFU forwards each receiver the one that fits its link and its tile; the sender pays in encoding and upload. SVC (scalable video coding) does it with one layered stream the SFU trims per receiver, in codecs such as VP9 and AV1.
What fits comes from bandwidth estimation. Receivers report when packets arrived, and packets arriving further apart than they were sent signal a queue growing on the path, often before any loss. The sender keeps its bitrate under its estimate, and the SFU, with an estimate per receiver, moves the phone to a lower layer at a point where the switch decodes cleanly, for simulcast one of that layer's keyframes, so the picture loses detail and is less likely to freeze. A segmented video player adapts too, a few seconds of video at a time; that adaptive bitrate is in Worked design: video streaming.
The teacher uploads three layers once. The SFU forwards one layer per receiver and a copy to the recorder, which writes files at its own pace.
A fleet of SFUs, and an ocean
An SFU forwards only between its own connections, so in a fleet signaling sends every participant of a class to the same SFU, or to SFUs linked for that class, never a round-robin spread. That one-owner-per-room routing is the subject of WebSockets at scale: gateways, routing & draining. What is particular to media is distance: with the teacher in Europe and students in Brazil, larger platforms cascade SFUs, each participant joining one in their region and the SFUs forwarding to each other, so the teacher's stream crosses the ocean once, not once per student.
Not every live video needs this. For one speaker and 20,000 viewers who never talk back, each viewer would need a stream forwarded by an SFU, while low-latency HLS serves the same segments to everyone from a CDN at roughly 2 to 5 seconds of delay; the Live Streaming Platform reference makes that choice.
What the Video Conferencing brief will test
Video Conferencing asks for all of this in at most 20 nodes, and the decisions that shape it are the ones above: signaling and media as separate paths out of the client, a WebSocket gateway for one and an SFU, drawn as an application service, for the other; the recorder off the media path; and an answer on the diagram for a participant whose network blocks UDP and a phone that cannot take the top layer.