Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Rate Limiter

Where a rate limiter sits and whose limit it enforces: at the public edge, in front of an internal dependency, or on your own side of a third party's quota, and how many workers share one quota.

Behind the gateway, nobody counted

Nothing outside the company can reach the Contacts service of a notification platform. It answers one question for the rest of the platform: for this user, which phone number, and have they opted out of text messages? Every SMS the platform sends asks it first. The public API had limits per customer API key, enforced at the gateway, and everything behind the gateway was trusted, so nobody gave Contacts a limit.

Then the marketing team's new Campaign service assembled the audience for a promotional text: 1.8 million users, looked up as fast as its 40 threads could go, about 6,000 lookups a second against a service sized for 1,500. Contacts' database saturated, the lookups for one-time sign-in codes timed out alongside the campaign's, and for twenty minutes nobody could sign in with a text code. No customer had sent too many requests; one of the platform's own services had. Limiting only at the public edge and skipping service-to-service calls leaves exactly this hole: an internal caller with no limit can still overload a dependency.

A rate limiter is the component that caps how many requests per unit of time pass from callers to what it protects. How it counts, with windows or a token bucket, who counts as one client and how instances share one count are all in Rate limiting. This article is about where the component sits and whose limit it enforces. Its limit is a number set by policy, from a contract, a load test or a price list; shaping how much of a shared resource each consumer uses, and deciding what happens to the excess, is Throttling.

One component, three places

By the end of that week the platform had three limiters. The component was the same each time; what differed was where it sat and whose limit it enforced:

  • At the public edge, in the gateway: each customer's API key gets the allowance its plan pays for. The platform protects itself from its customers; Rate limiting follows this one in depth.
  • In front of an internal dependency, Contacts: each calling service gets its own allowance. One service is protected from its colleagues.
  • On the caller's side of a third party, before the sender workers call the SMS provider. The platform enforces someone else's limit on itself.
Two limiters behind the gateway

Contacts' limiter gives each calling service its own allowance. The SMS limiter sits on the platform's side and keeps all twelve sender workers under the provider's quota.

A limit for each colleague

Contacts' allowances are a policy decision: 1,000 lookups a second for the sender workers, whose lookups are for sign-in codes and order updates, and 300 for Campaign, together under the 1,500 Contacts was sized for, so a new caller's allowance comes out of what is left. Paced to its 300 a second, Campaign's audience of 1.8 million now takes 100 minutes instead of five, and the promotion did not need it in five.

The limiter sits in front of Contacts, not inside Campaign's code. A limit the caller enforces on itself protects Contacts only from callers that remember to have one, and the next new service may not. In front, keyed by the calling service's identity, it covers every caller, including the ones not written yet, as long as internal calls carry an identity it can trust and nothing reaches Contacts around it.

A rate is all it caps. If Contacts' database slows down, 300 lookups a second can still pile up hundreds of calls in flight, and capping those is Bulkhead. Nor does the limit drop when Contacts struggles, because it is policy rather than a reading of Contacts' health; carrying the consumer's actual capacity upstream is Back-Pressure.

Someone else's quota

The SMS provider allows the platform's account 100 messages a second and refuses the rest. It enforces that on its own side, so why limit anything before calling it? Because a refused message still costs a round trip and has to be sent again, because this provider's terms let it suspend an account that keeps exceeding its quota, and because only the platform knows which message can wait: the provider refuses whatever happens to arrive over the line, sign-in code or promotion.

The first version gave each of the twelve sender workers its own limiter at 100 a second. Each one was right about itself, and together, at peak, they could send 1,200 a second. One quota shared by many workers has two common designs. Split the quota: each worker gets a twelfth and nothing is shared, but the share is tied to the worker count, so scaling to 20 workers without recomputing it overshoots, and an idle worker wastes its share while a busy one waits. Share one bucket: every worker takes a token from one count in a shared store before each send, so the whole quota goes to whichever workers have work and adding workers changes nothing, at the price of a round trip to the store per send and a new dependency, whose correctness and failure modes are in Rate limiting. Either way the platform sets its own limit a little under the provider's, since the two sides do not count over the same instants, and allows no burst larger than the provider's way of counting accepts (Rate limiting covers bursts).

A send that finds no token still has to go somewhere: it can wait in the queue, be dropped once it is no longer worth sending, or fail fast so the app can offer the code another way. Which one depends on who is waiting for that message, and that choice belongs to Throttling.

The edge, under bursty load

The Rate Limiter Challenge builds the first of the three places: a given API to protect at scale, with a limiter in front of it and a shared counter store. Where it sits is the first decision to get right there. In front of the API, nothing can reach it around the limiter, but the limiter is also on the path of every allowed request, so its check is a cost that legitimate callers pay, the latency the brief asks you to keep small. And because the callers at that edge are waiting on their answers, a request over the limit is refused cleanly rather than queued indefinitely, which is what lets a good design hold a steady allowed rate under bursty load, provided the limiter and its store have room for the refused requests too.

Practice this