One photo, nine files
Eight photos and a 20-second clip, the usual listing on a secondhand marketplace, have to become 75 files before anyone can browse them. Each photo arrives from a phone as a 4032 × 3024 HEIC file of about 3 MB, and few clients can use it as it is. Many browsers cannot display HEIC at all, the search results need a 320-pixel thumbnail, the listing page a 1,080-pixel image, and the zoom view something larger. One photo becomes about nine files, three sizes in three formats; the clip becomes two video files and a poster frame.
Each of those files is a rendition, also called a derivative: a version of the original made for one kind of client. Producing them is media processing. For video, decoding the original and re-encoding it with a different codec, resolution or bitrate is transcoding, and the clip alone can take longer than all eight photos.
Not while the seller waits
The first version did the work inside the publish request, and it was fine on the developer's laptop with one photo. With a real listing, the request ran for 40 seconds: the load balancer gave up at its 30-second timeout and the seller saw an error, while the API instance went on encoding for a client that had gone, and other requests queued behind its busy CPU.
So the work leaves the request. The bytes land in object storage first, the subject of Large file upload. The publish request then only writes an assets row in state processing and one job per upload, in the same transaction, for a relay to put on a queue (a Transactional Outbox), and answers in milliseconds; the app shows the listing as processing and refreshes when it is ready. A pool of media workers takes jobs at the pace their CPUs allow, so when 2,000 sellers publish on a Sunday evening, the queue grows and drains over the following minutes instead of the API falling over. How a pool shares one queue is Competing Consumers.
Open it before trusting it
The first thing a worker does with an upload is find out what it really is, from the file's own bytes rather than the name and Content-Type the phone sent, and check whatever dimensions, frame count and duration the file declares against the service's limits before decoding anything. Declared values can be missing or false, so the decoder keeps enforcing those pixel, frame and duration limits while it reads. A PNG of about 10 MB, one colour across 50,000 × 50,000 pixels, decodes to 10 GB of memory at four bytes per pixel; a worker that decodes first and checks later dies of that decompression bomb. Decoders themselves have had exploitable bugs, so they run with memory and time limits and no credentials they do not need. Phone photos also carry EXIF metadata, often including the GPS position where they were taken, which for a bike photographed in a garage is the seller's home; some libraries copy it into the output by default, so strip it, after applying the orientation tag to the pixels. Anything served as uploaded, such as the original clip offered for download, is scanned for malware, and public media usually goes through content moderation too; nothing is copied where the CDN can read it until both pass. A file that fails these checks is not retried: the asset is marked rejected with a reason, and the seller is told which photo to send again.
Nine jobs, one ready flag
A photo that passes fans out: the worker puts nine rendition jobs on the queue, and nine free workers can run them at once. One after another, the renditions take the sum of their times; in parallel, roughly the time of the slowest.
Parallel jobs raise a question one job never had: when is the asset done? If the listing goes live when the first rendition lands, a phone asking for the 1,080-pixel WebP a second later gets a 404 for a size that is not there yet. The asset is ready only when every rendition made up front exists. Keep one row per rendition and a count of pending ones on the asset, and have each job, once the object store has confirmed its file, in one transaction:
- The job sets its rendition row to
done, only if it is stillpending. - If that changed a row, the job decrements the asset's pending count.
- If the count is now zero, the job sets the asset to
ready.
The database locks the asset row for each decrement, so one job, and only one, sees the count reach zero. The same guarded count one level up tells which job readies a listing's last asset; that job writes the seller's notice in the same transaction, through the same outbox, so a crash after the commit cannot lose it. From then on the renditions are served through a CDN (CDN & static delivery).
A worker checks and scans each upload, then queues one job per rendition. Each rendition is written to a fixed path, and the last job to finish flips the asset to ready.
Counters are enough for a listing. A two-hour film is cut into time chunks encoded in parallel, which Worked design: video streaming walks through, and a process of many dependent steps that must survive days of waiting belongs on an engine, as Durable workflows: orchestration that survives restarts explains.
Twice, or forever
Most job queues deliver at least once. A worker writes listings/8812/p3/1080.webp, then crashes before acknowledging the job, and the queue hands it to another worker, which does it again. A job that outlasts the queue's visibility timeout is handed out a second time with no crash at all, so the clip's jobs get a longer timeout than the thumbnails'. Every job must therefore be safe to run twice. Output paths come from the asset and the rendition, never from a random name, so a second run overwrites the same object instead of leaving an orphan beside it, and the first step of the transaction above stops a repeated job from counting twice. Calls to systems outside your own, a moderation service that bills per image for instance, need the tools Idempotency & exactly-once illusions compares.
The opposite failure is a job that never succeeds. Photo 6 was cut short in transit, and the decoder throws on it every time. A timeout reading from storage is worth retrying with backoff; this file is not, because it fails the same way on each attempt and, with no limit, circles the queue forever, holding a worker on every lap. So the worker tells the two apart. A decoder rejecting the file is a permanent error, and the rendition is marked failed at once; an unknown error gets a few attempts, then the job moves to a Dead Letter Channel where a person can look at it. Either way the asset moves to failed instead of sitting in processing, the listing says which photo to upload again, and a periodic sweep catches any asset still processing an hour later with no job left for it.
Paying for the CPU
Photos are cheap; video is where the bill is. Encoding time grows with duration, resolution, the number of renditions and the codec: AV1 files are smaller than H.264 at similar quality but take far more CPU to encode. Two choices cut the bill at little cost to sellers. Produce up front only the renditions every listing needs, and the rare ones, a 2,400-pixel zoom for instance, on first request, caching the result. And keep new listings ahead of backfills: re-encoding the whole catalogue into a new format can wait in its own queue, while a seller watching processing cannot.
Kill a worker on purpose
Video Streaming Platform asks for a transcoding pipeline that produces several resolutions; build its ingest half. Queue one job per rendition, give each output a fixed path, and write beside the asset table what flips it to ready. Then stop a worker halfway through a rendition and check that your design says where the job runs next, what the second run overwrites, and what keeps the video from going live early. File Upload & Processing Flow draws the same path as a flowchart: a decision that validates the file, parallel steps that rejoin, and an error path for the file that fails every time.