Why not just buy a bigger server?
It is the first question asked when an API starts running hot, and often the right first answer. Vertical scaling, a bigger box, needs no change to the code. But it has a ceiling, the largest machine you can get, and the bigger box is still one box: when it crashes, or restarts for a patch, the whole API is down.
Horizontal scaling takes the other route: run several identical instances of the service and put a Load Balancer in front to distribute requests. Take a photo-sharing app's API that peaks at 1,200 requests a second. Three instances at about 400 requests a second each carry that peak, a fourth is how you add capacity, and losing one costs a third of the capacity instead of all of it. At the peak, though, two survivors at 800 cannot carry 1,200, which is why a pool is usually sized with one instance to spare. The bet only pays off if a few things are true, and the rest of this article is those things.
Any instance, any request
The first thing to break is usually login. The API kept each user's session in its own memory, which was fine on one server. Now a user logs in on App #2, the balancer sends their next request to App #3, and App #3 has never heard of them.
The fix is to make the instances stateless: keep anything that exists in only one place — sessions, counters — in a shared store, so any instance can serve any request; what the session itself carries, and how a logout reaches every instance, is Sessions vs tokens: JWT, refresh & revocation. A per-instance cache is still fine, because it only holds copies: a miss falls back to the shared store, and a short TTL bounds how long a changed entry can stay stale there, while a shared cache raises the hit rate across instances; Caching layers weighs the two.
The tempting shortcut is sticky sessions, where the balancer keeps sending each user to the instance that holds their session. It hides the problem until that instance dies, and those users' state dies with it. It can also leave one instance busier than the others, because the balancer is no longer free to choose.
Noticing that an instance died
Spreading requests is half the balancer's job; the other half is not sending them to an instance that cannot answer. It runs health checks: every few seconds it calls each instance, often on an endpoint such as /health (what that endpoint should check is the subject of Health Endpoint Monitoring), takes an instance out of rotation after a few failed checks, and puts it back once it passes again. Skip them and the balancer keeps sending traffic to a dead or degraded instance.
Health checks route new requests around a failure; they do not save the requests it catches. Requests already in flight on an instance that dies can fail, and so can any sent to it before the checks notice. Callers need what Timeout and Retry describe: a timeout, and a retry for the requests that are safe to repeat.
Which instance gets the next request
The photo API's requests are not alike. Listing an album takes about 20 ms; building a zip export of one takes several seconds. Under round-robin, each request to the next instance in turn, an instance that happens to draw three exports keeps being handed its full share of everything else and falls behind. Least connections sends each request to the instance with the fewest active connections, a stand-in for how busy it is, so traffic flows around the busy one. Later the team adds one large machine beside two of the original small ones. Plain round-robin still gives each a third and overloads the small ones first; weighted routing (weighted round-robin or weighted least-connections) gives each instance traffic in proportion to a weight you set for what it can take.
Adding boxes without a person
Traffic at 04:00 is a fraction of the evening peak, and paying for the peak all night is waste. An Autoscaler adds or removes instances as load changes, by watching a metric such as CPU or request rate and keeping it near a target. A new instance takes time to start and pass its health checks, so autoscaling follows a surge rather than meeting it. For a peak you know is coming, a launch or a televised event, add the instances before it starts.
What does not multiply
The balancer itself is one more box in the request path. Run it redundantly too, or it becomes the new single point of failure. Managed load balancers from cloud providers are usually run that way for you.
Then look behind the app servers. Scaling them while the database or cache behind them stays a single instance only moves the bottleneck: ten instances sending queries to one database are limited by that database. Growing that tier is its own subject: Replication & read replicas spreads reads, Sharding & partitioning spreads writes, and Caching layers keeps reads from reaching it at all.
Take a zone away
The Diagnose: Sessions in a Single Zone challenge hands you this article's starting point gone wrong: one app service behind a load balancer, every session in one unreplicated in-memory store, all in a single zone, and a zone outage about to take it all down. The app tier is yours to fix: run it as several interchangeable instances behind the balancer by raising the app service's replicas on its stepper, since the simulator spreads one node's replicas across zones but takes down any node with a single replica, and keep sessions in the shared store, so any surviving instance can serve any logged-in user. Done right, and with the store replicated as well, the outage costs capacity, not users. Replicating the store is the brief's other half, and belongs to Replication & read replicas.
URL Shortener asks the same of a read-heavy redirect service, and Distributed Session Store has you design the shared store itself.