Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Throttling

Meter what each consumer's work costs a shared resource and, as one nears its budget, delay or degrade that consumer's work, or refuse it with a Retry-After the server can stand behind, so one tenant cannot take capacity everyone shares.

Slow dashboards, quiet API

"Is it just us, or have the dashboards been slow all morning?" Three shops sent support the same question within an hour. They are tenants of an analytics service that serves 900 online shops from one shared analytical database with 32 cores: sales dashboards for the shop owners, scheduled CSV exports, and an API their own scripts pull data from. Tiles that usually load in a second were taking thirty.

The API's request graphs showed nothing unusual. One tenant, a large marketplace, had scheduled 40 exports of its full order history, each scanning about 200 million rows. Forty requests in an hour is nowhere near the 600 a minute the API allows each key. But at the peak, eleven of those exports were running at once on two cores each, about two thirds of the database, and the other 899 shops' dashboard queries waited behind them. That is a noisy neighbour: one tenant whose heavy use of a shared resource degrades the service for everyone else.

Throttling is the server's answer: meter each consumer's use of a shared resource against its budget and, as it nears the budget, slow, defer or reduce that consumer's work, so one spike or one abusive caller cannot exhaust capacity everyone needs. The service degrades gracefully for one consumer instead of collapsing for all. The two terms overlap in practice; in this catalogue, counting requests per client, with its algorithms, is Rate limiting, and this article starts from the resource and asks how much of it each consumer takes. Refusing work because the whole server is full, whoever sent it, is Load Shedding.

Count what the work costs

Requests are the wrong unit when one request can cost a million times another. A dashboard tile showing yesterday's revenue reads a few thousand rows of a pre-computed table; an export reads 200 million. So the service meters the resource that ran out, database CPU time, and gives each plan a budget in CPU-seconds: the marketplace's plan allows 120 a minute, refilled at 2 a second. Rows scanned and bytes exported are other common meters; the right one is whatever runs out first.

The catch is that cost is only known once the work has run. So the throttle charges an estimate on admission, from the query planner's row estimate or the export's date range, and corrects it to the measured CPU time when the work finishes. A tenant that used more than its estimate ends up below zero and pays it back by waiting while the budget refills. A single runaway query still needs its own ceiling, such as a time limit enforced in the database, because the throttle decides only when work starts.

Delay, degrade or refuse

Once the marketplace nears its budget, what happens to its next piece of work depends on who is waiting for it. All three answers below are legitimate; the context picks one.

  • Delay the exports. Nobody watches a spinner for an export; the file is emailed when it is ready. An export over budget joins a queue for that tenant and starts when the budget has room. The queue is bounded at 20 waiting exports and the next one is refused, because a queue that only grows turns today's spike into tomorrow's backlog. The tenant is not asked to slow down, as with Back-Pressure: its work waits on the service's side.
  • Degrade the dashboards. A shop owner looking at a dashboard is waiting now, so a throttled tenant's tiles are served from hourly rollups instead of raw orders, and auto-refresh drops from every 30 seconds to every 5 minutes. The numbers are up to an hour old, and the page still loads. Degrading needs a cheaper path built in advance; work with no cheaper version can only be delayed or refused.
  • Refuse the API pulls. The marketplace's own scripts get 429 Too Many Requests with a Retry-After header. A script can wait and come back, and the header tells it when.
Three answers to one tenant over budget

Over budget, an export waits in the queue until the budget refills; an API pull gets 429 with Retry-After from the throttle.

A refusal that says when

The throttle's first version answered over-budget pulls with a bare 503 and no header. With nothing telling them to wait, the marketplace's scripts retried at once, and their retries multiplied in the way Retry describes. Each retry was refused again, but each one still cost the API servers authentication and a budget lookup, plus a TLS handshake on a new connection, during exactly the minutes the service was short of capacity. A throttle that refuses with no signal to back off, no 429 and no Retry-After, turns a load spike into a worse outage when its clients retry at once.

The value in Retry-After is the server's estimate of when the request would be admitted, and with a budget that refills at a known rate the server can compute it. The marketplace is 90 CPU-seconds below zero, its next pull is estimated at 6, and the budget refills at 2 a second: if nothing else draws on the budget meanwhile, 96 ÷ 2 is 48 seconds, so Retry-After: 48. For work that waits in a queue, the estimate comes from the queue instead: the export's position in the queue divided by how fast the tenant's exports drain, a rough figure when export costs vary. Too short a value brings the client back while it is still over budget, to be refused again; too long leaves budget unused while the client sleeps.

Tenants throttled together and told to return when the same capacity frees come back together, as a wave, so the server adds a few random seconds to each value; clients add their own jitter, which is Retry's side. The header is advice, not enforcement: a client that ignores it is refused again by the same cheap check.

Who gives way first

Per-tenant budgets add up to far more than the database has, so several heavy tenants, each within budget, can still fill it. So admission also checks the database's total load, and when that nears its limit even tenants with budget left wait, in an order: background work gives way before interactive work, and lower plans before higher ones, so a Standard tenant's exports are delayed first and an Enterprise tenant's dashboards are the last thing touched. Each plan, and background work within it, keeps a minimum share, because under strict priority a lower plan's exports could wait forever. Ordering the waiting work itself is Priority Queue's job.

Standing policy, and a bridge

Throttling stays on as policy, limiting each tenant's share, and it also buys time while capacity catches up, for example while a read replica for exports copies its data and warms up. A throttle that fires every afternoon may be rationing a service that is too small, so the count of throttled work per tenant belongs on a dashboard as a capacity signal. Moving exports to their own pool is Bulkhead, and the wider choices for a tenant that outgrows shared capacity are in Multi-tenancy.

Allow, delay or refuse, as a flow

The API Rate-Limit Decision Flow Challenge turns this article's decision into a flowchart: a Decision checks whether the client is within its limit, an exceeded request is either delayed before processing or rejected with an error response, and the paths merge before the final response. Model the marketplace's API there, with the Decision asking about the tenant's CPU-second budget rather than a count of requests. The interesting part is the over-budget branch: it needs a second question, whether someone is waiting on this answer and whether the queue has room, before it can choose between the Delay and a rejection whose Retry-After comes from the refill rate. The degraded dashboard has no branch in the brief; on the real service it is the third answer to that same question.

Practice this