One loop doing two jobs
An import job loads a customer's CSV export, 30 million rows, into the database. Downloading and parsing the file runs at about 100,000 rows a second; inserting into the database, in batches, runs at about 25,000 rows a second on one connection. The first version does both in one loop: parse a batch, insert it, parse the next. While the insert waits on the database, nobody parses, and while the parser waits on the network, nobody inserts.
The obvious split is two threads. A reader thread parses rows and hands them over; a writer thread takes them and inserts them. The thread that creates work is the producer, the thread that performs it is the consumer, and the place where handed-over work waits is a buffer, almost always a first-in, first-out queue. Neither side calls the other. Each only talks to the buffer, so each can run at its own speed.
The queue that ate the heap
The first try uses a plain growable list, protected so two threads can use it safely. The reader appends, the writer removes from the front, and a little under a minute in, the process dies with an out-of-memory error.
The arithmetic was there from the start. The reader adds 100,000 rows a second and the writer removes 25,000, so the list grows by 75,000 rows every second. At around 500 bytes per parsed row in memory, that is close to 40 MB a second, and a 2 GB heap lasts under a minute. With memory to spare, the backlog would have peaked at 22.5 million rows, more than 10 GB, the moment the reader finished, and the job would not have ended any sooner: the database still takes 25,000 rows a second, so 30 million rows need 20 minutes however fast the reader runs. The unbounded list moved most of the file from disk into RAM and bought nothing.
It also hid the problem. The reader never slowed down or failed, so from its side everything looked fast, and the first symptom anyone saw was the crash. Meanwhile a row parsed at second 10 sat behind 750,000 others, half a minute of waiting, and the wait grew every second.
Bounding it: the reader waits
The fix is to give the buffer a capacity, say 10,000 rows. When it is full, the producer's put blocks: the reader stops until the writer has taken a row and freed a slot. The queue fills in a fraction of a second, and from then on the reader, on average, parses at the writer's pace. The queue holds about 5 MB, and the import still takes 20 minutes, against about 25 for the single loop; it was never going to take less.
The other edge matters as much. When the download stalls, the writer drains the queue and finds it empty, and its take blocks too, so it sleeps until a row arrives instead of spinning in a loop that asks whether anything is there yet. A queue that blocks producers when full and consumers when empty is a blocking queue, and it is the usual way this pattern is built. Inside, it is usually a lock plus waiting for "not full" and "not empty", the tools Mutex and semaphore covers.
Diagram components: Client, Queue, Handler.
The queue is the only thing both sides touch. The reader waits when it is full; the writer waits when it is empty.
You rarely write one yourself. Java's ArrayBlockingQueue, Python's queue.Queue(maxsize=10000) and Go's buffered channel, make(chan Row, 10000), all block in the same two places. Watch the defaults: Python's queue.Queue() with no size and Java's LinkedBlockingQueue with no capacity are unbounded for practical purposes, which brings back the list that ate the heap.
The bound turns an invisible failure into a number you can graph: a reader that spends most of its time blocked on put says the writers are the bottleneck, long before anything crashes. It also caps how long a row waits, as long as the writers keep draining. In a steady state, the average number of items in a queue equals their arrival rate times the average time each spends in it (Little's law), so a full queue of 10,000 rows drained at 25,000 a second means about 0.4 seconds of waiting on average.
What the buffer buys is room for bursts and gaps. During a two-second download stall, a full 50,000-row buffer would keep the writer busy the whole time. What no buffer can do is fix a consumer that is slower on average: if rows keep arriving faster than they leave, any finite buffer eventually fills and stays full for as long as that lasts. So size it from the longest burst you want to ride out, keep it small enough that its memory and waiting time are acceptable, and remember that more capacity is not more throughput.
When waiting is the wrong answer
Blocking suits the import because every row matters and the reader has nothing better to do. Other producers cannot wait, and a full bounded queue gives them other choices. Blocking with a timeout fits a request thread that should return an error after 100 ms rather than hang. Dropping the new item, and counting the drops, fits debug logging on a request path, where losing a line is cheaper than slowing a request. Dropping the oldest item fits a stream of position readings, where only the latest value is worth anything. Rejecting at once fits a caller that can retry later or go elsewhere, because it learns now instead of after a timeout. Whichever you choose, choose it deliberately: an unbounded queue is also a policy, and its answer to "full" is the crash above.
Four writers, one queue
Inserting is the slow side, so the next step is four writer threads taking from the same queue. A take removes the item, so each row goes to exactly one writer. If the database can absorb four connections' worth of inserts, the job approaches a quarter of the time, at which point the four writers match the reader's 100,000 rows a second and a fifth writer would only wait on it. Producers multiply the same way: two files imported at once can have two readers putting into the same queue, and the standard blocking queues are built for any number on each side. Worker threads that take tasks from a shared queue and are reused from task to task have their own article, Thread Pool.
What several consumers give up is order. Rows still leave the queue in order, but one writer's batch can finish after a later one, so two updates to the same customer can land in the wrong order. Competing Consumers traces that failure.
Handing rows over, and the last row
Putting an item on the queue hands it over: from then on it belongs to the consumer. A reader that saves allocations by parsing every line into the same mutable row object will overwrite rows a writer is still inserting, so put a new object each time, or make items immutable.
At the end of the file the reader stops, the writers drain the queue, and then they block on an empty take, waiting for rows that will never come. A consumer only learns the work is over if someone tells it. Some queues can be closed, as Go's channels can; otherwise the usual way is a sentinel, a special item meaning "no more work" put after the last real row, one per consumer, so four writers need four. Because the queue is FIFO, every real row leaves before the first sentinel; with several producers, the sentinels go in only after the last producer has finished. Failure needs the same care in reverse: if all four writers die on a database error, the reader fills the queue and blocks on put forever, unless a timeout or a shared cancellation signal lets the failure reach it.
Beyond one process
This queue lives in one process's memory and disappears with it; between services, with a broker in the middle, the same idea is Queue-Based Load Leveling, and the broker itself is covered in Message queues & async workers. Carrying "full" upstream across services, or refusing work on purpose, is Backpressure & load shedding.
A buffer with a size, on a Grid
The Producer-Consumer challenge gives you up to six nodes for producers, the bounded buffer and consumers. Try two Client nodes as producers, a Queue node with its capacity written in the title, and Handler nodes as consumers, then label each edge with what happens at the limit: the producer's put waiting when the buffer is full, the consumer's take waiting when it is empty. If your drawing would still make sense with an unbounded queue, the number in the Queue's title is missing.