Why does the autoscaler stop at 60%?
A company that sells a spell-checking API keeps its fleet at about 60% CPU: past that, the autoscaler adds instances. In a cost review, finance asks why. At 90% the same traffic would need a third fewer machines. The engineers say latency would suffer, and they are half right. Which half depends on something the CPU graph does not show: how many requests each instance works on at once.
The company also runs /suggest, which calls an old dictionary library that is not safe to use from two threads, so every call takes a lock and each process serves one request at a time. At 90% busy, on the assumptions below, the spell-checker adds about 3 ms to a 20 ms request, while /suggest makes a request wait 180 ms on average before it starts. Same utilization, more than fifty times the wait.
Where waiting comes from
Utilization is the share of the time the servers are busy; CPU is a fair stand-in only when CPU is what requests wait for. A request waits when it arrives and every server is busy. One /suggest process takes 20 ms a call, so it can finish 50 a second; at 45 a second it is 90% utilized. Had those 45 arrived exactly 22.2 ms apart, nobody would wait. But users act independently, so requests bunch, and a gap with nobody to serve is capacity lost for good while a burst becomes a line. With random arrivals, a request finds the process busy 90% of the time.
The textbook model of one server, the M/M/1 queue, assumes random, independent arrivals and service times as variable as their mean (the simulator's default). Its mean wait in line is ρ / (1 − ρ) times the service time, ρ being the utilization: 2.3x at 70%, 4x at 80%, 9x at 90% (a 20 ms call takes 200 ms in all), 19x at 95%, 99x at 99%. With constant service times the waits are about half as long. There is no knee, just a curve that steepens, and these are means: what the slowest requests see is the subject of Latency percentiles & tail latency.
Counting the servers
So count what can be in progress at once, not the boxes. Little's law says that, on average, in a system that keeps up, the requests inside it equal their arrival rate times the time each spends there. A node's servers are therefore its maximum throughput times the time one request takes to serve, wait excluded: how many requests it is serving at once when full. A spell-checker instance serving 1,500 requests a second at 20 ms holds about 30 in flight when full, so it behaves like 30 servers. A relational database at 10,000 queries a second and 8 ms is about 80. A matching engine, a single-threaded event loop doing CPU work, or a lock that serialises every caller is one, however big the machine.
Both endpoints run at 90% of their capacity. The spell-checker's 30 slots barely queue; /suggest, one call at a time per process, waits nine times its service time.
More servers help twice. Fewer arrivals find them all busy: with 200 servers at 90%, fewer than one in ten do, and most requests walk straight in. And the line drains faster: a full spell-checker instance finishes a request every two-thirds of a millisecond on average, not every 20 ms. At 90%, about half its requests find all 30 slots taken, but they wait about 7 ms on average, so the mean over all requests is about 3 ms. Under the same assumptions:
- 30 servers at 90%: the wait is about 0.16x the service time. It only reaches several times the service time very close to full: about 3x at 99%.
- 200 servers at 90%: the wait is under 1% of the service time. A wide pool can run very hot before a queue shows.
- Any number of servers at 100% or more: requests arrive at least as fast as they can be served, so the queue has no steady state and, on average, keeps growing for as long as the load lasts. More servers move that point; they do not remove it.
More cores do nothing for a step that does one thing at a time; a faster core, more processes, or splitting the work do. And one server can hide in a wide design: a global lock or a sequence number handed out in one place makes that step a one-server queue for every request that needs it, and a single primary taking every write caps writes at one machine, however many replicas serve reads.
One line or four
/suggest cannot use more threads, but it can run four processes, at 90% between them: 180 requests a second against 200. If the balancer picks a process at random, without knowing which one is free, each process keeps its own line and is a single server at 90%: 9x, 180 ms, as before, even while a neighbour sits idle; round-robin, which spaces arrivals out, only brings that down to about 5x. If requests wait in one shared queue and each process takes the next as soon as it finishes, a request waits only when all four are busy, and the wait drops to about 2x, roughly 40 ms. That is pooling, the same reason one spell-checker instance does so well: one line in front of many servers. Twenty instances behind a balancer, by contrast, are twenty lines of 30 slots, not one line of 600.
The same four processes at 90%. With a line each, a request can wait while another process sits idle; with one shared line, it waits only when all four are busy.
Headroom is for the minutes before help arrives
So finance's 90% would be fine for the spell-checker's queue, about 3 ms, and wrong for the fleet. A new instance takes minutes to start and pass its health checks, and until then the fleet carries any rise on what it has. At 60% it can absorb two-thirds more traffic before reaching 100%; at 90%, a ninth more. A failure is the same arithmetic: spread over three zones at 60%, losing one leaves the rest at 90%, while at 90% it leaves them at 135%, with a queue that grows until the zone returns. The target follows from how fast you add capacity and what you must survive, and it has to hold at the peak, not the daily average. "Fine at 90%" describes today's wait, not headroom; past capacity the question becomes which requests to refuse (Backpressure & load shedding).
/suggest is the opposite case: its target comes from the queue itself, 2.3x (about 47 ms) at 70% against 180 ms at 90%. That is why one alert threshold for every service fails: "page at 80% CPU" is too late for the single-server step and far too early for the wide pool. Set each from the wait you can tolerate, which a latency objective in SLOs & error budgets writes down.
Push a design over the edge
In the Rate Limiter challenge, simulate the design with one replica of everything: the busiest node goes past 100%, requests go unanswered and the modelled tail reads unbounded. Add replicas until the busiest node is comfortably below 100%, not just under it, and watch the wait almost disappear; beyond that, each extra replica buys almost nothing. Stock Exchange Matching Engine asks the other question: matching is sequential per symbol on purpose, and in a real exchange the answer is to partition by symbol, many one-server queues side by side, rather than to add capacity to one queue (Single-writer sequencing & matching engines explains why one book keeps one writer). The simulator models the engine node as an ordinary service pool, so it will not show that difference; the design is where you make it.