Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Rate limiting

Cap how often each client may call you: deciding what counts as one client, counting with a fixed window, a sliding window or a token bucket, sharing the count across instances, and telling a refused caller when to come back.

One line in the price list

A company-registry API sells lookups by API key: send a company number, get back the company's name, registered address and directors. The Starter plan's price list says "600 requests per minute". That line protects three things: the capacity of the lookup servers and their database, every other customer's sign-up form from one customer's overnight script, and a registry worth selling from being copied record by record at full speed.

Enforcing the line is rate limiting: keep a count of each client's recent requests and refuse the ones above its allowance, before doing the work. The line hides three decisions: who counts as one client, how "per minute" is counted, and where the count lives when the API runs on eight servers.

A per-client limit stops one client from taking more than its share. When thousands of clients, each within its allowance, add up to more than the servers can do, it has nothing to refuse: that overload belongs to Backpressure & load shedding. And a limit refuses without putting a crowd in order, which is what Virtual waiting rooms: admitting a crowd fairly do for a sale that opens at a known minute.

Who is one client?

On the paid plans it is the API key, which identifies a customer who signed a contract, so the limit follows the contract. A client can also be one of your own services; Rate Limiter looks at limits between internal services, where one caller with no limit can still overload a dependency.

The free search page on the registry's website has no key, only the caller's IP address, and that is where per-IP limits mislead. Many people share one address behind an office's or a mobile carrier's NAT gateway, so a limit tight enough to slow a scraper blocks the office, while a scraper renting thousands of addresses from a botnet keeps each one under the limit. A per-IP limit stops one loud source; floods from many addresses need the layers in Edge protection: WAF, DDoS & defence in depth.

Six hundred a minute, by the clock

The simplest counter is a fixed window: one counter per key per calendar minute, named after both, such as acme:10:04. Each request increments it; once it passes 600, the rest of that minute is refused; the next minute starts a new counter at zero, and old ones expire.

Then a customer reports, correctly, that its script sent 1,150 requests in one minute without a single refusal: 590 between 10:04:30 and 10:04:59, and 560 between 10:05:00 and 10:05:29. Each calendar minute stayed under 600; the 60 seconds from 10:04:30 held 1,150. A fixed window lets a client send up to twice its limit in a short span around the boundary. For a monthly quota that hardly matters; for a database sized for 600 a minute from each customer, it can.

Counting the last sixty seconds

A sliding window counts the 60 seconds that end now instead. The common way, a sliding window counter, keeps only this minute's count and the previous one's, and weights the previous count by how much of that minute still falls inside the last 60 seconds. Replay the script against it, sending at an even pace: left unrefused, by 10:05:15 it would have sent 280 requests this minute, and three quarters of last minute's 590 still count, about 722 in all. In fact the limiter starts refusing it within the first two seconds of 10:05. It assumes last minute's requests were spread evenly, so the count is an estimate, bought with two counters: it can refuse early, and when last minute's requests came in a burst near its end, it can let well over 600 through in 60 seconds.

Neither window has a notion of a burst you mean to allow, and the registry's customers have them: a bank's onboarding batch at 09:00 wants 200 lookups at once, then little for an hour. A token bucket fits that shape. Each key's bucket holds up to 200 tokens and refills at 10 a second, the 600 a minute of the price list; each request takes a token, and a request that finds the bucket empty is refused. A quiet customer can spend all 200 in one second, yet over a long stretch no customer averages much more than the refill rate. The plan now publishes two numbers, the rate and the burst.

Eight gateways, one count

The check usually runs in the gateway, ahead of every service (API gateway & the edge), and the registry runs eight gateway instances behind a load balancer. If each keeps its counters in its own memory, each sees about an eighth of a customer's traffic and allows 600 of it, so up to 4,800 requests a minute get through. The effective limit can multiply by up to the instance count. Diagnose: Rate Limiter That Isn't Shared starts from exactly this design, under a retry storm.

The usual fix is a shared, fast store for the counters, often Redis, so every instance reads and updates the same number. The update has to be atomic: two gateways that each read 599 and each write 600 have let two requests through on one remaining slot. For a fixed window an atomic increment is enough, as long as the gateway decides from the count the increment returns. The sliding-window estimate and the token bucket read their state, compute and update it together, so that whole step must run atomically, for example as a script the store runs as one operation, such as a Lua script in Redis.

Every gateway asks the same counter

A request over the limit is answered by the limiter with 429 and Retry-After; the registry service never sees it.

The shared count is paid for on every request, allowed or refused: a network round trip to the store before the lookup starts, which is why the store sits close to the gateways. It is also a new dependency: if Redis stops answering, the gateways either fail open, serving unlimited and leaving the database exposed, or fail closed, turning a counter outage into an API outage; many teams fail open behind a coarser per-instance backstop. One customer that dwarfs the rest also turns its counter into a hot spot, the problem Counting at scale takes on.

What the refusal says

A bare 429 Too Many Requests invites the client to try again at once and be refused again, so the answer also says when to come back: Retry-After: 3, in seconds or as an HTTP date. A token bucket can compute it from its refill rate; a fixed window can only point at the end of the minute. Clients that all return in the same second still arrive as a wave, which jittered backoff in Retry spreads out. Better still, the client sees its allowance before running out: many APIs report it on every response in headers such as X-RateLimit-Remaining, names that vary by provider while an IETF draft proposes standard RateLimit fields. And because the refusal is decided from the counter alone, a refused lookup costs a counter check, never a database query.

Not every request weighs the same

The registry's bulk endpoint returns up to 100 records per call; counted as one request, it would let a Starter customer take 60,000 records a minute, so a bulk call takes one token per company from the customer's own bucket, inside the same 600 a minute. The same reasoning separates plans, Starter at 10 a second and Business at 100. And it separates endpoints: limit a cheap read as tightly as an expensive write and you refuse capacity customers paid for; limit the write as loosely as the read and a few callers can saturate what it touches.

Put the price list on the limiter

The Rate Limiter Challenge asks for a distributed limiter in front of a given API, with two algorithms, token bucket and sliding window, instances that agree on the count, the quota and Retry-After headers, and little added latency. Draw the gateway, the limiter and a shared counter store, and write on the limiter the decisions this article took one at a time: the key, the algorithm with its rate and burst, and what the gateways do when the store does not answer. Then label the refused path with its status and headers.

Practice this