Network protocols: TCP, UDP, HTTP/2 & HTTP/3

Thirty milliseconds in the logs, a second in the hand

A bird-sighting app keeps its API in Virginia, and a third of its users log sightings from Australia. The server's logs say a list of nearby sightings takes 30 ms to build; birders in Sydney say it takes most of a second to appear. Both are right. Between them is a round trip time (RTT), the time for a packet to reach Virginia and the answer to come back, here about 200 ms, and the number of round trips the phone makes before it can even ask.

Light in fibre covers about 200 km per millisecond and the cities are more than 15,000 km apart, so no round trip between them takes under about 150 ms. Protocols cannot shorten a round trip; they decide how many a request pays, and what one lost packet does to the rest.

Three round trips before the first byte

After the lookup that finds the server's address (DNS, proxies & the network edge), a fresh HTTPS request over TCP with TLS 1.3 goes like this:

1. The phone sends a TCP SYN to open a connection; the server answers SYN-ACK. One round trip, 200 ms.
2. The phone sends the last ACK and, right behind it, a TLS ClientHello with its key share; the server answers with its key share, its certificate and Finished. The second round trip.
3. The phone sends its Finished together with GET /sightings; the server builds the list in 30 ms and the response starts back. The third round trip.

About 630 ms before the first byte of an answer built in 30 ms. TLS agrees on keys and proves the server's identity (certificates are Secrets management & encryption). TLS 1.2 needs two round trips for a full handshake, so the same request pays four, about 830 ms.

Two things shrink the bill: fewer connections, and handshakes paid closer to the phone. An edge server in Sydney, about 10 ms away, ends the phone's handshakes and keeps its own connections to Virginia open, so a fresh request pays three short round trips and one long one, roughly 260 ms instead of 630. Where such edges sit is CDN & static delivery.

What TCP promises, and what one lost packet costs

TCP gives the app a reliable, ordered byte stream: every byte is numbered and acknowledged, whatever is not acknowledged in time is sent again, and bytes reach the app strictly in order. That last promise has a price. When one segment is lost, the segments after it may already sit in the phone's buffer, but the app gets none of them until the retransmission fills the gap: about a round trip at best, longer when the loss is noticed only by a timer. One missing piece holding up everything behind it, even unrelated data, is head-of-line blocking.

Congestion control cuts TCP's sending rate when packets are lost, and a new connection starts in slow start: with the common initial window of 10 segments, about 14 KB, it roughly doubles what it sends every round trip. If nothing is lost and nothing else caps the rate, a 1 MB photo on a fresh connection goes out in rounds of about 14, 28, 56, 112, 224 and 448 KB and finishes in the seventh, about 1.4 s on this path.

Reliable has a limit. An acknowledgement means the other side's TCP stack has the bytes, not that the application acted on them. When the connection drops halfway through Submit sighting, the app cannot tell whether the server saved it, and a blind retry can save it twice; Idempotency & exactly-once illusions covers making the retry safe.

UDP sends a packet and promises nothing

UDP sends each datagram on its own: no handshake, no acknowledgement, no retransmission, no ordering, no congestion control. A datagram may be lost, duplicated or overtaken, and the sender is not told; any reliability or pacing is the app's to build. That suits traffic where data that arrives late is useless and waiting for it would hold up newer data, a frame of voice or a player's position soon replaced by the next: Realtime media: WebRTC, SFUs & NAT traversal follows that through for calls, Multiplayer game state: authoritative servers & tick rates for games. For the bird app, UDP matters because HTTP/3 is built on it.

One connection, one response at a time

Opening the app sends about 30 requests: the sighting list, then thumbnails and species cards. HTTP/1.1 connections are persistent by default, keep-alive, so the phone sends request after request on one connection and pays the handshakes once. But each connection carries one response at a time, in order: a slow species card ahead of 20 thumbnails holds them all, head-of-line blocking again, this time in HTTP. Clients open more connections instead, browsers typically up to six per host, each paying its own handshakes and its own slow start.

HTTP/2: many streams, one TCP queue

HTTP/2 keeps HTTP's semantics and changes how they travel. Each request and response becomes a stream, cut into frames that interleave with other streams' frames on one connection, so the slow species card no longer holds up the thumbnails, and repeated headers are compressed. Client and server agree on HTTP/2 inside the TLS handshake, through an extension called ALPN, at no extra round trip. gRPC runs on HTTP/2; whether to use it is API styles: REST vs GraphQL vs gRPC.

The blocking moved; it did not go away. All the streams ride one TCP byte stream, and TCP knows nothing about them. On a coastal walk with a weak signal, one lost segment carrying a thumbnail stalls every stream, the sighting list included. Over six HTTP/1.1 connections, the same loss would have stalled one, which is why HTTP/2 helps least on the lossiest links.

HTTP/3 moves the streams into the transport

HTTP/3 is HTTP over QUIC, a transport on UDP that does in its own code what TCP does in the operating system, acknowledgements, retransmission and congestion control, but knows about streams. Each stream is still delivered in order, and a lost packet delays only the streams whose data it carried: the thumbnail waits, the sighting list keeps going. QUIC also folds the TLS 1.3 handshake into its own, so a fresh request from Sydney straight to Virginia pays two round trips, about 430 ms, instead of three.

A phone that has connected before can go further: with a ticket from the earlier session, it sends the request in its first packets as 0-RTT early data and gets the answer after one round trip. The catch is replay: whoever captures those packets can send them again, and the server cannot always tell. A replayed GET /sightings is harmless; a replayed Submit sighting is a duplicate. So early data should carry only requests safe to repeat, and a server can refuse it or answer 425 Too Early to have it resent after the handshake.

Then a birder walks out of a visitor centre's Wi-Fi onto mobile data mid-upload. A TCP connection is identified by the two addresses and ports, so with the phone's new address it is gone: the app reconnects, pays the handshakes again and restarts the upload unless it was sent in resumable pieces. A QUIC connection is identified by connection IDs the two ends choose, so when both support it, the connection carries on once the new path is checked: connection migration. A balancer that passes QUIC packets through to servers, rather than ending QUIC itself, has to keep sending a migrated connection's packets to the server that holds it, usually by routing on the connection ID instead of the address (Load balancing & horizontal scaling).

None of this skips slow start: QUIC backs off on loss as TCP does, for the whole connection, so the photo still ramps up over several round trips, and QUIC, running in user space, typically costs servers more CPU per byte than TCP in the kernel.

TCP stays as the floor

Some networks block or throttle UDP, such as firewalls that let out only TCP on ports 80 and 443, and a client usually learns that a server speaks HTTP/3 from an earlier answer over TCP, an Alt-Svc header. So browsers fall back to TCP when QUIC does not get through, and a deployment that adds HTTP/3 keeps serving HTTP/2 and HTTP/1.1 on the same names; turning TCP off would cut off exactly the users behind those networks.

Inside one data centre, with sub-millisecond round trips and little loss, pooled HTTP/2 connections leave HTTP/3 little to win; toward phones on long, lossy, changing paths it wins most, and the gain shows in the slowest requests (Latency percentiles & tail latency).

One participant on TCP

In the Video Conferencing Challenge, signaling crosses the WebSocket gateway over TCP and media reaches the SFU over UDP. Design for the participant whose office network blocks UDP, so their media reaches the SFU through a relay on TCP port 443. What does one lost packet do to their call that it does not do to anyone else's, and what in your design keeps the rest of the meeting from waiting for them? Multiplayer Game State Sync raises the first question for player inputs, which its allowed components can carry only over a WebSocket.