Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Worked design: video streaming

Upload to object storage, a transcode pipeline cut into per-rendition chunks, adaptive-bitrate manifests behind a CDN, and view counting off the hot path.

One upload, a million plays

A video is written once and, with luck, read a million times, and almost every decision in this design follows from that. At 18:00 a creator uploads a ten-minute, 300 MB video. Within minutes it should play on a phone on a train and on a television over fibre, and if it takes off, a million people may watch it by morning.

Scale that one video up to a platform (the method is in Back-of-envelope estimation) and the two moments become two very different numbers. Ingest is light in requests and heavy in bytes. Say 100,000 uploads a day at ~300 MB average — barely one upload a second, but 30 TB/day of raw bytes, and each video is transcoded into five renditions that add another ~1.5× — raw plus renditions, a fresh ~27 PB a year in object storage. Playback is the reverse. Say 100 million views a day, five minutes each at ~3 Mbps ≈ 110 MB per view, so ~11 PB/day of egress — roughly 1 Tbps on average. The estimate says two things: the serve side is a CDN problem (the origin must see a few percent of that), and the ingest side is a batch pipeline where a single video can take longer to process than to upload. The rest of this article follows the one video through both halves.

18:00: the bytes go straight to storage

Bytes never pass through an application server. The creator's app asks the API for a presigned URL and uploads straight to object storage in resumable parts; the API only records metadata in a state = uploading row, which moves to uploaded when storage confirms the object has landed. Sending 300 MB through the API instead would tie an app instance to one slow client for minutes. Large file upload covers presigned URLs, multipart uploads and the checks around them. When the object lands, the storage emits an event, and it is that event, put on a queue (Message queues & async workers), that starts the transcoding.

18:01: cut into chunks, one job each

Transcoding, which Media processing & transcoding describes as a building block, turns the upload into renditions: 1080p, 720p, 480p, 360p and audio-only. The obvious unit of work is one job per video, and it is the wrong one. A four-hour upload would hold one worker for hours, and a failure near the end would throw all of that away. So the unit of work is one job per rendition per time chunk (a few minutes of source each). The ten-minute video becomes about twenty short jobs (four chunks of two and a half minutes, times five renditions) that run in parallel; a four-hour upload becomes hundreds, and a failure costs one chunk, not hours of encoding.

Each worker writes its output as segments a few seconds long, to a deterministic path derived from the video ID, rendition and chunk, with keyframes at the same timestamps in every rendition so the player can switch between them at any segment boundary. A job that is retried therefore overwrites its own output rather than duplicating it, which is what makes redelivery from the queue harmless. No chunk job writes a playlist, since four chunks writing one file would race: when every chunk of every rendition has landed, one last job writes each rendition's playlist and the master manifest, the file that lists them, and the video flips to ready.

Ingest path: presigned upload → chunked transcode → manifest

18:20: the first play

The player first asks the API for the video. The rights and geo checks happen there, and the answer carries the manifest URL from the metadata store. Those checks only hold if the CDN enforces them too, because the segments are fetched from the CDN, not the API: for restricted content the API hands out short-lived signed URLs or cookies, and the CDN refuses requests without a valid one. The player fetches the master manifest, which lists the renditions, then requests short segments and can switch rendition at each segment boundary based on the bandwidth it measures. That is the core of adaptive bitrate: the phone on the train drops to 360p as the signal weakens and climbs back to 720p when the signal returns, without restarting the video, where a single 1080p file would simply stall. It is also why segments, not whole files, are what gets cached.

The next morning: the millionth play

Everything the player fetches — manifests, playlists, segments — is immutable once the video is ready — paths fixed per video and rendition, never rewritten after publish — so the CDN can keep it with a very long TTL. The origin is object storage, shielded (origin shielding is part of CDN & static delivery) so a cache miss on a newly viral video collapses into about one origin fetch per segment rather than one per edge: the shield caches the segment after its first fetch and coalesces the misses that arrive while that fetch is in flight.

A million plays also means a million view counts. The obvious version, UPDATE videos SET views = views + 1 per playback start, puts a database write on the path of every one of 100 million plays a day, and turns a viral video's row into a hot spot. Instead the player emits a heartbeat event to a queue and a counter aggregates those events in batches, the buffer-and-batch shape High-write ingestion describes, so playback never waits on analytics. The counts lag by seconds and a retried heartbeat can be counted twice; dedupe by (viewer, video, session) in the aggregator, never in the playback path.

Serve path: manifest and segments from the CDN, views async

Where it breaks, and what it costs

  • A transcode worker dies mid-rendition. The job is redelivered and rewrites the same deterministic path; every other chunk and rendition is unaffected. Idempotent output plus small jobs is what makes a multi-hour pipeline restartable. With one job per video, the same crash restarts hours of encoding from zero.
  • A video goes viral in a region with a cold cache. Origin shielding collapses the miss storm to about one fetch per segment. Without it, every edge's misses for that video reach object storage at once.
  • Bytes through the API servers, in either direction. Uploads via the app tier tie a worker to a 300 MB request, and serving video from it means the origin takes 1 Tbps instead of the CDN.
  • Eager transcoding of the long tail. Transcoding every rendition for every upload burns compute on the long tail nobody watches. The alternative — transcode a few renditions eagerly, a low one included so every connection can start, and the rest on first request — trades storage and compute for a slower first play, and is a defensible choice once you have measured the tail.

Two paths that meet at the bucket

Video Streaming Platform asks for both halves this article walks. Draw the ingest pipeline (presigned upload → chunked transcode → manifest) and the serve path (CDN in front of object storage, view events on a side edge) so that they touch only at the storage bucket and the metadata table, and label the transcode edge with its unit of work, "one job per rendition per chunk". Once that split is clean, Live Streaming Platform is the variation worth trying: the same serve path, with the transcoder producing segments in real time, so each rendition's playlist keeps growing, or slides as a window, instead of being written once. A call, where the audience talks back and a delay of seconds is too long, leaves segments behind altogether: Realtime media: WebRTC, SFUs & NAT traversal covers it.

Practice this