Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Ad serving & real-time auctions

Pick a winning ad inside a 50 ms budget: what each stage returns when its slice runs out, candidate retrieval, ranking on fresh features, the auction's pricing rule, and budget caps that hold across a distributed fleet.

Fifty milliseconds, spent line by line

A reader opens an article on a regional news site, and the page asks for the ad above the fold. The ad server's share of the wait is 50 ms, with the network on either side not counted. Here is its ledger for this one request:

  • 2 ms gone, 48 left. The request is parsed, and the page and the reader are identified.
  • 8 ms gone, 42 left. Retrieval returns 300 candidates out of the platform's 2 million active ads.
  • 19 ms gone, 31 left. Features for the reader and the candidates are back, 3 ms later than usual.
  • 45 ms gone, 5 left. The click-prediction model has finished four of its six batches of 50, because a slow replica is taking 6.5 ms a batch instead of 4. A hundred candidates have no score.

Those last 5 ms are spoken for: 2 for the auction, 3 to write the response. So the server stops waiting. It drops the 100 unscored candidates, runs the auction among the 200 it has, and answers at exactly 50 ms. On a normal request the model is done at 40 ms and the answer leaves at 45.

The reader got an ad on time, possibly not the best one, and nobody logged an error. The whole system is built on that trade: the page is already rendering, a late answer finds the slot empty and the revenue gone, so the deadline is fixed and the quality is what flexes.

Why the model only ever sees 300 ads

The normal 24 ms for 300 candidates works out at 0.08 ms each. At that rate, scoring all 2 million would take 160 seconds, more than 3,000 times the entire budget. So the work is arranged by what it costs: retrieve wide, rank narrow, decide once. A cheap filter cuts millions of ads to a few hundred using targeting rules only, an expensive model scores those few hundred, and the auction picks one. The general shape, without the auction, is in Recommendation systems: retrieval, ranking & cold start.

Retrieval is a set intersection: geography, device, audience segment, campaign still active, budget not exhausted. A furniture store's campaign survives it because this reader is on a phone in its metro area. This happens on every impression, so it reads an in-memory index, not a relational database. Retrieval hands over its candidates sorted by a cheap estimate, the bid times the ad's cached average click-through rate, which is why the 100 left unscored were the least promising by that estimate.

Ranking multiplies predicted click-through by the per-click bid to get expected value per impression. The prediction needs features fresher than any nightly job can provide, because the reader's last few actions often matter more than their profile. A feature store aims to serve those in single-digit milliseconds, inside the 12 ms this call has, computed the same way for training and for serving; compute them differently and you get training/serving skew, a model that mysteriously underperforms in production. Feature stores: online, offline & training-serving skew covers that, and Model serving: inference, versions & safe rollout covers keeping the model itself fast.

One request, with a deadline on every call

Each label on a call out of the ad server is that stage's slice; the model's is a clock time, counted from the request's arrival. Impression and click events leave on a stream; nothing on the serving path waits for them.

A written answer for running out of time

A slice is only useful if the code knows what to do when it ends. The answer is a smaller or staler result, in the last resort a cached default, never an error:

  • Retrieval, 6 ms. Return the candidates found so far.
  • Features, 12 ms. Run the model with default values or an older profile in place of whatever did not arrive. Predictions get worse, and they stay usable only if the model was trained to tolerate gaps.
  • Model, until 45 ms. Keep the finished batches and cancel the rest. If no batch finished, rank on the cheap estimate retrieval already computed.
  • Auction and response, 5 ms. These call nothing over the network, so a slow dependency cannot eat the reservation. With no candidate left, return a house ad, a cached creative promoting the site itself.

A house ad earns nothing, but it costs almost nothing to serve, and an empty slot earns nothing and can break the layout. None of this helps against a long pause in the ad server's own process: that request misses its deadline.

The awkward part of a working degraded path is that it hides. The slow replica in the ledger produced no errors and no missed deadlines, only auctions on fewer candidates and revenue that drifts down. So count how often each stage falls back and watch that next to latency, or a sick model can look healthy for days.

Same winner, two prices

Among the 200 scored ads, the furniture store bids $2.00 a click at a predicted 1.5%, an expected value of 3 cents per impression. A mattress shop bids $1.50 at 1.6%, which comes to 2.4 cents. The furniture store wins either way. The pricing rule decides what it pays.

  • Under first-price, which most real-time exchanges use today (usually quoting per thousand impressions, where the rules work the same way), it pays its bid: $2.00 a click.
  • Under a second-price (GSP) rule, it pays the minimum bid that would still have beaten the runner-up's expected value: 2.4 cents divided by 1.5%, or $1.60 a click.

Then the decision is final: one winner per request, written to a log. That log is not the bill. What a campaign pays for must be deduplicated and validated before anyone is charged: for this per-click ad, the clicks (bots, double clicks); for a campaign priced per thousand impressions, the impressions (bots, double-fired pixels, and out-of-viewport ones where the contract requires viewability).

Spending a budget nobody can check in time

The furniture store has $600 a day, and its next impression may land on any of the region's 40 ad servers. A shared counter checked on every impression puts a round trip into a ledger with 5 ms to spare on a good day, and makes all serving depend on one counter staying fast and available.

A common answer is to let each server spend part of the budget locally and reconcile on a cycle. One illustrative design is local budget leases: a coordinator hands each server a slice, say 24 cents, and subtracts it from what remains. The server spends it locally with no network call, and every 30 seconds it returns what is unspent and asks for more. At most 40 × 24 cents, or $9.60, is out at once. A server cut off from the coordinator stops when its slice or its lease runs out, and a crashed one strands its slice until the lease expires and actual spend shows how much of it was used, so these failures tend to underspend.

If the leases never exceed what remains, overspending comes from the price not being known yet. This ad pays per click, so the server deducts the expected cost of each impression, $1.60 × 1.5% or 2.4 cents, and the real click arrives later. The last $9.60 of leases therefore covers 400 impressions and an expected 6 clicks. If those impressions draw 9 clicks, the store owes $14.40 against that $9.60: $4.80 over, or 0.8% of the day's budget, and clicks still due from earlier leases can add to that. Actual spend flowing back from the aggregates corrects the balance, but it cannot unshow an impression already served.

So overdelivery is bounded, by the clicks that impressions already served can still draw, rather than impossible, and someone must decide who eats the overspend, because the answer is in a contract somewhere. Frequency caps ("no more than 3 times per user per day") are the same shape of problem with a per-user key instead of a per-campaign one, so without a check on every impression a cap can overshoot too. Counting all those events at volume is the subject of High-write ingestion.

Slices on your own Grid

On the Ad Serving & Real-Time Auctions Challenge, label every edge that leaves the ad server with its slice, check that the slices fit inside the deadline with the auction and the response reserved, and draw what comes back when each one fires. Without an explicit degraded path, the deadline is only a wish. Ad Click Aggregation then takes the measurement side, where clicks become money, and Recommendation Engine drills retrieve-then-rank with no auction on top.

Practice this