Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Bulkhead

Give each dependency its own limited share of threads or connections, so one slow dependency fills only its own compartment, where further calls are refused fast, while everything else keeps serving.

Recommendations got slow, so sign-in failed

At 14:05 a video streaming app's sign-in started timing out. Nothing was wrong with sign-in: its code had not changed and its database was idle. What had changed was the recommendations service, whose responses went from 80 ms to 8 seconds after a bad deploy.

The app's API backend handles every request on one of 200 threads, and home-screen requests, 150 a second, each call recommendations while their thread waits. At 8 seconds per call, 150 requests a second would need 1,200 threads waiting at once. There were 200, so within about a second and a third every thread was a home screen waiting on recommendations, and sign-in and playback requests sat in the queue behind them until they timed out. A 2-second Timeout on the call would not have saved it: 150 calls a second that each wait 2 seconds still need 300 threads.

Walls between the dependencies

Every outbound call borrowed from the same resource, the request threads, so the dependency that held them longest took them all. A Bulkhead divides the resource into compartments, one per dependency or per tenant, each with its own limit, so a flood in one fills only its own. The name comes from a ship's watertight compartments: a breach floods one, and the ship stays afloat.

Separate pools per dependency

The recommendations compartment has all 20 of its calls in flight and refuses the rest at once, before they reach the service. The other two compartments, and the rest of the backend's threads, are untouched.

With compartments, recommendations may have at most 20 calls in flight. At 14:05 those 20 fill with slow calls, and the 21st home screen's call is refused at once instead of waiting, and the screen renders without its recommendations row, which is that row's Fallback. At most 20 backend threads now wait on recommendations, every refused home screen holds its thread for milliseconds, and sign-in and playback never notice. Work that needs the slow dependency still fails or degrades unless it has a fallback; work that does not need it keeps serving.

The walls are usually built one of two ways: a semaphore per dependency, counting calls in flight on the caller's own thread, as the 14:05 numbers assume, or a thread pool per dependency with a short bounded queue, where the caller waits with a timeout and can walk away even from a call stuck in a blocking client, leaving only a pool thread blocked, at the price of extra threads and a handoff. How to size such a pool, and how it can deadlock itself, is in Thread Pool. An HTTP client whose connection pool is shared by every destination is the same trap, with the same fix: a limit per dependency.

A full compartment says no

The refusal is the point. Suppose the recommendations compartment, instead of refusing the 21st call, makes it wait for a permit with no time limit. The waiting request still holds its backend thread, all 200 fill up again, and the bulkhead has isolated nothing. A full compartment has to reject at once, or after a short bounded wait, and the caller needs something to do with the refusal: a fallback, a cached answer, or a clear error.

That is also what separates it from its neighbours. A Circuit Breaker stops calling a dependency once recent failures say it is sick, so before it trips, or when slow calls are not counted as failures, the bulkhead is what contains the slowdown. A Rate Limiter caps requests per unit of time, which does not bound concurrency: 20 calls a second that each take 8 seconds is still 160 calls in flight.

How big each compartment is

Recommendations normally need about 150 × 0.08 = 12 calls in flight on average, so a limit of 20 leaves room for a busy evening. Too small, and the compartment refuses calls on an ordinary day; too large, and it protects little, since a limit of 150 would let recommendations take three quarters of the threads. And compartments do not lend to each other: at 4 a.m. playback's 30 permits sit idle, and recommendations still may not go past 20 however many threads are free. That is the trade. Each limit is isolation paid for with headroom a busy dependency cannot borrow (and, with a pool per dependency, with idle threads as well), so size each compartment from its normal rate times its normal latency, add headroom, and watch its rejections.

Isolation covers only what you partition. These compartments split the backend's threads; they do not split its CPU, its memory, or a database that two services share, and any of those can still take everything down. The same walls can be built higher, with separate instances per kind of traffic or per tenant, up to whole copies of the stack that each serve a subset of customers, which is Deployment Stamps.

Break one compartment on purpose

The Bulkhead challenge asks for this backend in general terms: a service calling several dependencies, a separate resource pool for each, one dependency failing and exhausting only its own pool while the others keep serving. Its eight nodes are exactly enough for a Client, the Service, three Bulkhead nodes with their limits in the titles, and three dependencies. Then make one dependency hang on the canvas and check that nothing outside its compartment has a reason to change.

Practice this