2,000 photos, 2,000 threads
A photo-sharing app resizes every upload into three sizes before showing it. A 12-megapixel JPEG is about 4 MB on disk; to resize it, the service decodes it into raw pixels, and at four bytes per pixel that is 48 MB of memory while the work runs. The first version did the obvious thing: write the upload to disk, start a new thread, resize, save the three files, let the thread end. With a few uploads a second and about two seconds of CPU per resize, it ran for months without trouble.
Then a push notification announced a photo contest, and 2,000 uploads arrived in ten seconds. The service started 2,000 threads. The machine has 8 cores, so at most 8 of those threads were running at any instant; the rest took turns, and resizes that should have taken two seconds stretched over minutes. Meanwhile every thread that got a turn began decoding, and 2,000 × 48 MB is about 96 GB of pixels on a machine with 16 GB. The process ran out of memory and died, taking with it the uploads that were nearly finished.
Creating a thread is not free either: the operating system allocates a stack for it, and the scheduler has one more thread to track. Next to a two-second resize that cost is small, and it is not what killed the service. What killed it was that the number of uploads arriving decided how much work ran at once.
A fixed crew and a line to wait in
A thread pool separates the two things the first version tied together: accepting a task and running it. In its usual form it has three parts:
- A task queue, where submitted tasks wait their turn.
- A fixed number of worker threads, kept alive and reused instead of one created per task. Each worker loops: take the next task from the queue, run it, come back for another. A worker with nothing to do waits on the queue.
- A submit call. The caller hands over a task and immediately gets back a handle to its eventual result, which it can wait on, check, or try to cancel. Java calls that handle a
Future, and so does Python'sconcurrent.futures.
With a pool of 8 workers, the contest burst plays out differently. The upload handler submits one resize task per photo, carrying only the path of the file already on disk, and keeps the handle so it can tell the user when the three sizes are ready. At most 8 resizes run at once, so at most 8 decoded originals, about 400 MB plus their smaller resized copies, sit in memory. Each worker that finishes a photo takes the next task without any thread being created or destroyed. The burst now costs time instead of the process.
Diagram components: Client, Queue, Thread Pool.
The queue is where the burst waits. Only the workers run tasks, so at most 8 resizes are in progress at once.
The name suggests Object Pool, and the reuse is the same idea, but the direction is reversed: from an object pool the caller borrows a connection and uses it itself, while the caller of a thread pool never touches a thread. It hands over work. And because the 8 workers run concurrently, any mutable state two tasks share, a counter of processed photos or an in-memory cache, needs the protection described in Mutex and semaphore.
How many workers?
Eight was not a random number. Resizing is CPU-bound: a resizing worker keeps a core busy, so a ninth one on 8 cores only takes time from the others, adding another 48 MB image and context switches for little or no throughput. Start CPU-bound pools near the core count. In standard CPython, threads run Python code one at a time, so this holds only for work done inside libraries that release the interpreter lock, as image libraries such as Pillow largely do. Then each task also uploads its three files to object storage, 600 ms during which it holds no core. Work that spends part of its time waiting, I/O-bound work, needs more threads than cores to keep the cores busy; a common first guess is cores × (1 + waiting time ÷ computing time), here about 10, which assumes alike tasks and CPU as the only limit.
The other limit sits downstream. If the storage client allows 16 connections, no more than 16 uploads run at once however many workers the pool has: a worker that reaches its upload while all 16 are busy waits for a connection, still holding its images. So three numbers shape the size: what keeps the cores busy, what fits in memory, and what the dependencies accept, since workers past that last one spend their upload step queuing for a connection. The queue's length and the workers' idle time tell you which one you guessed wrong.
Where the burst goes instead
The pool did not make the burst smaller; it moved it into the queue. Ten workers at about 2.6 seconds per task finish just under four uploads a second, so with a queue that accepts everything, the last of the 2,000 photos starts about eight and a half minutes after it arrived, long after its user gave up and uploaded it again. That is why the pool's queue needs a limit, and why a full pool must refuse new tasks or slow down whoever submits them, instead of letting the wait grow. The defaults do not do this for you: Java's Executors.newFixedThreadPool and Python's ThreadPoolExecutor both queue without a limit. The choices for a full queue are worked through in Producer-Consumer.
Ten workers, all waiting for each other
Someone noticed that the three storage uploads could run in parallel. The resize task now submits three upload tasks to the same pool and waits on their handles. On a quiet day that is faster. In the next burst:
- Ten resize tasks start, one on each of the ten workers.
- Each resize task finishes resizing, submits its three uploads and waits on their handles.
- The 30 upload tasks join the queue, behind the resize tasks already waiting there.
- No worker is free to run an upload, and none will be free until an upload finishes.
Every worker is blocked and the pool has stopped for good, with no lock anywhere in the application code and the CPU idle. This is a deadlock made by the pool itself: a task that waits for another task submitted to the same fixed-size pool can be holding the very worker the other task needs. In an ordinary pool like this one, the fix is to never let a worker wait on work queued in its own pool: let the resize task do the three uploads itself, one after another, or give the uploads a pool of their own.
The same exhaustion happens without any waiting between tasks when slow work shares workers with fast work: a few downloads from sluggish remote sites can occupy all ten workers while two-second resizes queue behind them. One pool per kind of work keeps that contained, which is the idea of Bulkhead.
Exceptions, and the last deploy of the day
A corrupt JPEG makes the decoder throw. Depending on the pool and how the task was submitted, that exception can end the worker thread, or sit unread inside the task's handle while the user waits for an answer that never comes; in a hand-written pool that does not catch it, each failure costs a worker for good, until none are left. So catch at the boundary of each task, log what identifies it, complete the handle with the error, and send the worker back to the queue. Anything stored per thread also outlives the task on a reused thread, so clear it before the worker moves on.
Stopping is the other end of the same lifecycle. A process that simply exits loses the queued tasks and the ones in progress. A graceful shutdown stops accepting new tasks, lets the workers drain the queue within a deadline, and only then stops them; whatever is left at the deadline has to go somewhere it can be retried from, because a queue in memory dies with the process.
The contest burst on a Grid
The Thread Pool challenge asks for exactly the mechanism in the first diagram: clients submitting tasks to a work queue, a fixed number of workers, idle workers pulling the next task, and threads reused across tasks. Draw a Client node for the upload handlers, a Queue node, and a Thread Pool node with its worker count in the title, and label the edges with what happens rather than with nouns. One thing worth trying within the six nodes: draw the first version beside it, a Client pointing straight at a Handler for every upload. Side by side, the bound the brief asks you to make clear comes down to one number written on one node.