"Upload failed at 71%. Please try again."
A paralegal sees that message for the third time while uploading an 8 GB archive of scanned contracts to an e-discovery platform. The office uplink does 25 Mbps, so the file needs about 43 minutes. The upload endpoint is the obvious one: the browser posts the file to the API, and the API streams it on into object storage.
Each attempt dies somewhere different. A load balancer or gateway between the browser and the API enforces an idle timeout, often around 60 seconds, and a stretch where the uplink stalls is enough to trip it. A deploy restarts the instance holding the stream. Every failure starts again from byte zero. Meanwhile the API, sized to answer requests in milliseconds, holds a connection and a buffer per upload for most of an hour, so twenty firms uploading at once tie up a fleet built for page loads.
The fix is to take the API out of the byte path. The browser sends the bytes directly to object storage, the API only decides who may send what, and the platform learns that the file arrived from the storage itself, as an event.
The API signs, the bucket receives
The API's job shrinks to issuing permission. It mints a presigned URL: a URL to the store, signed with the platform's credentials, that authorises exactly one operation (PUT to uploads/7c41e9…) for one content type, for a few minutes. Valet Key is the pattern behind it, and Object storage & blobs covers the bucket on the other side. The API generates the key; the client never chooses it, so one firm cannot overwrite another's upload by guessing a name.
The bucket enforces only what the signature pins, so everything else is decided before signing: may this user upload to this case, does the declared size fit the firm's quota, is the content type on the allowed list. A bare presigned PUT accepts a body of any size. Sign the exact Content-Length too, or use a presigned POST policy with a content-length-range, and the bucket rejects a body that does not match what was approved.
One file, 125 parts
A single presigned PUT of 8 GB would still fail the way the old endpoint did: one request cannot be resumed or parallelised, and S3 caps a single PUT at 5 GB anyway. For anything past roughly 100 MB, the client uses a multipart upload:
- The client hashes every part and asks the API to start; the API validates the request, opens a multipart upload in the store and records a session row with those checksums.
- The API returns a presigned URL per part, each signed for that part's exact size, here about 125 parts of 64 MB (on S3 a part is 5 MB to 5 GB, the last one may be smaller, and an upload has at most 10,000 parts).
- The client sends several parts in parallel, each with its own checksum, and keeps the ETag the store returns for each.
- The client asks the API to complete; the API checks the parts' sizes against the approved total and tells the store to assemble the parts from the list of part numbers and ETags.
The session row, upload_id, case_id, owner, size, part_checksums, parts_done, status, expires_at, is what lets the platform list, resume and expire uploads. The checksum is there because a part can be damaged between the laptop and the store, and an archive with one bad megabyte can open fine until someone needs the page inside it. The client sends an MD5 or SHA-256 with each part, the store recomputes it and rejects a mismatch, and the bad part is resent alone at once instead of surfacing weeks later as a broken file.
The laptop that slept at part 90
The paralegal's laptop goes to sleep around part 90. Nothing already sent is lost: the store keeps every part it acknowledged until the upload is completed or aborted. When the page comes back, the client asks the API for the session; the API asks the store which parts it holds (S3's ListParts), signs fresh URLs for the missing ones, since the old ones have expired, and the upload carries on with whichever parts are still missing, which parallel sends can leave anywhere in the file. If the tab was closed, the user picks the file again; the client finds the open session by name and size, hashes the file part by part, and resumes only if every checksum matches the list recorded at the start. A different file with the same name and size starts a new upload instead. Resending a part with the same bytes is harmless, because a part number uploaded again replaces the earlier one.
So resumable is bookkeeping, not a new kind of server: the client, with the session row and the store's list of parts, knows where it stopped.
Done is not when the client says so
The complete call proves only that the client thinks it finished. Before anyone opens the archive, the platform needs to know that the object exists, that it is whole, and that it passed the platform's checks. Two decisions supply that.
First, uploads land in quarantine: a separate uploads/ bucket with no public access, from which nothing is served and which only the scanner reads (the Quarantine pattern). Second, the platform hears about the arrival from the store. When the multipart upload completes, the bucket emits an ObjectCreated event, and a queue carries it to a scanner; the event names the key instead of carrying the bytes, the move Claim-Check describes. The scanner checks the archive for malware and checks that the bytes match the declared type. Only then does it copy the object to the final cases/ bucket and mark the row ready; a failed scan marks it rejected, and until then the API answers the client with scanning. The same event can arrive twice, so the scanner skips a key it already handled (Idempotency & exactly-once illusions).
The API is on the path to start, sign and complete, and never carries file bytes. The storage event, not the client, moves the upload forward.
expired, since a bucket rule cannot touch your database. Quarantine objects that failed the scan need deleting too, by the scanner or an expiry rule of their own.Draw it without a byte through the API
The File Storage & Sync challenge asks for large-file uploads through chunking, and its deep dive for resumable uploads; draw this path there. No edge should carry file bytes into the API: the client writes parts to storage with presigned URLs, and the API signs, records the session and completes. Then add what the brief leaves to you: a quarantine bucket, a storage event that wakes a worker to scan, and an expiry for sessions nobody finished. Its chunks are also named by their hash for deduplication; Content-addressed storage & deduplication explains that half, and Media processing & transcoding what happens to a clean file next.