Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

SLOs & error budgets

Turn "reliable" into a number you can measure, alert on and spend: SLIs, SLOs, the budget they imply, and burn-rate alerts that page on the trend rather than the blip.

“Can we run the migration on Thursday?”

The checkout team wants to run a database migration with a small chance of causing a few minutes of errors. The on-call engineer, paged twice this week at 3 a.m. for error spikes that were over before the laptop opened, says the service has been flaky lately and now is not the time. Product points out that no customer has complained. Three people are arguing about one word, reliable, and none of them can win, because it is not a number.

"The API should be reliable" is not a requirement; "99.9% of requests succeed within 300 ms, measured over 30 days" is. Three terms carry that sentence. An SLI (service level indicator) is the measurement — the fraction of good events over total events. An SLO (service level objective) is the target for that SLI — 99.9%. An SLA (service level agreement) is the contract with consequences attached, usually service credits, and normally looser than the SLO so an internal miss is not a refund. Engineering runs on SLOs; sales signs SLAs; nobody should be paged for an SLA.

Where the SLI is measured decides whether anyone trusts it. Checkout's first draft counted CPU and pod restarts, which are causes, not experiences: a box can run hot while every customer is served, and every box can look healthy while a misrouted request fails. The version the team kept measures success ratio and latency at the load balancer, where customer requests arrive, with latency read as a percentile (Latency percentiles & tail latency). A good SLI is one a customer would agree with.

A budget for failure

Here is the idea that settles Thursday's argument: the SLO is a budget for failure. 99.9% over 30 days means you are allowed to fail 0.1% of requests, where failing means an error or an answer slower than 300 ms. That is your error budget, and it is a resource to spend, not a shame to hide. For checkout, the arithmetic looks like this:

  • 99.9% = 0.1% bad = 43 minutes of full outage per 30-day month at steady traffic, or 7.2 hours at 10% errors, or 43,000 bad requests out of 43 million.
  • 99.99% = 4.3 minutes/month — beyond what a human on-call can respond to, so it forces automation. 99.5% = 3.6 hours/month, often plenty for an internal batch service.

Choosing the target is a cost decision. Every extra nine cuts the budget tenfold, usually costs far more engineering than the last one, and buys nothing users notice past their own network, and 100% is not a target at all: it leaves no budget, so the first failed request is already a miss and nothing risky can ever be approved. 99.99% for a typical blog is money burned.

Checkout also depends on payments, inventory and fraud scoring, and each of those promises 99.9%. A request that touches three 99.9% services is roughly 99.7% if they fail independently — the budget of a chain is roughly the sum of its parts' failure fractions. So the order matters: set the customer-facing SLO first, then derive what each dependency must hit. For checkout to keep 99.9% across three dependencies in a chain, splitting the budget evenly and leaving none for checkout's own code, each one needs roughly 99.97%. That sum assumes every dependency failure reaches the customer; timeouts, retries and fallbacks can stop some of them, which Cascading failures: how one dependency takes down its callers covers.

TipThe budget is a release policy. Budget healthy: ship daily, run the risky migration. Budget half-spent with most of the month left: keep shipping, but no experiments. Budget exhausted: freeze feature releases and spend the sprint on the top causes of burn — the number decides, so nobody has to argue.

Thursday's question becomes a lookup: how much of this month's budget is left? And if the answer never changes a decision, if no release is ever paused and no risky migration is ever approved on the strength of a full budget, the SLO is a dashboard, not an SLO. A rollout tool can read the same number on its own, pausing or rolling back a release that spends budget too fast; Progressive delivery: blue-green, canary & rollback covers that loop.

Why the pager kept firing

Now the on-call engineer's complaint. Checkout's alert was a threshold: page when the error rate is above 1%. A 90-second spike to 6% errors crosses it and pages at 3 a.m., yet it costs about 0.2% of the month's budget. Meanwhile a steady 0.3% error rate never crosses the line, and it spends the entire month's budget in ten days. The alert fires on what does not matter and stays silent on what does.

Burn rate fixes both: it is how many times faster than "exactly on budget" you are failing: the bad fraction in the window divided by the fraction the SLO allows, so 0.3% bad against a 0.1% budget is a burn of 3×. Burn rate 1 spends the budget in exactly 30 days; 2 spends it in 15. The alert rules ask how fast the budget is going, measured over a window:

  • 14.4× over one hour spends 2% of the month's budget in an hour — page now.
  • 6× over six hours has spent 5% of the budget — a slower leak, but still worth a page.
  • 1× over three days is a ticket, for working hours.

Pair each long window with a short one, about a twelfth of it (5 min for the 1 h rule, 30 min for 6 h, 6 h for 3 days), and alert only while both exceed the burn rate, so the alert clears soon after the burn stops. This is the standard multi-window, multi-burn-rate design and it is worth copying rather than inventing.

SLI at the edge, budget computed from it, page on burn rate

The alert asks 'how fast is the month's budget disappearing?', not 'is error rate above X right now?'.

Run the two incidents through the new rules. The 90-second spike averages out to a burn of 1.5× over the hour, far below 14.4×, so nobody is woken. The steady 0.3% is a burn of 3×: too slow to page, and the three-day rule opens a ticket within about a day, with most of the budget still left to fix it. How the metrics get from the load balancer into the store that evaluates these rules is the subject of Observability: metrics, logs & alerting.

Burn-rate rules on a real pipeline

Open the Metrics Monitoring & Alerting challenge. It is the pipeline this article's alerts run on, and it carries an SLO badge of its own: 99.9% availability and a 350 ms tail-latency target (the simulator models the tail as about 3x the mean), with the Simulation panel reporting your design's error-budget burn at that load, where anything above 1x means it fails faster than the target allows. Focus on one thing: write the three burn-rate rules on the edge where alerts are evaluated, with their windows, and route the two pages and the ticket separately. Uptime Monitoring & Status Page practises the same instinct from the outside, confirming an outage from a second region before anyone is paged, and Alert Evaluation & Auto-Remediation turns the fast-burn page into an automated response.

Practice this