Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Usage metering & billing pipelines

Turning raw product usage into an invoice finance will sign: usage events with stable ids, deduplication, totals per customer, meter and period, late events and closed periods, rating against the plan, and reconciliation before the invoice goes out.

Two counts of one March

A podcast network's upload logs add up to 41,850 minutes of audio for March. The March invoice from the speech-to-text API it uses bills 44,300. The API is priced per audio minute: the first 1,000 minutes of each month are free, the next 49,000 cost 4 cents each, and every minute beyond 50,000 costs 2 cents. On April 6th the network's engineering lead asks which number is wrong, and the gap is $98 on a bill of $1,732.

The customer's count is right. March's raw usage events show two separate faults, neither of which would have stood out on a dashboard: 1,190 minutes were counted twice, and another 1,260 came from rounding. A usage chart 6% high is a curiosity; an invoice 6% high is a refund, an apology and, if systematic, a compliance problem. The same events often feed both, and only one can be approximately right.

Usage-based pricing gives an analytics pipeline the correctness requirements of an accounting system. It runs in four stages, meter, aggregate, rate and invoice, each re-runnable from the output of the one before, so a pricing bug found in month three can be fixed without replaying everything.

What a usage event has to carry

The transcription service cuts each upload into chunks of up to 20 minutes, split at pauses, and for each transcribed chunk it emits a meter event: a record that one customer used some quantity of one meter at one moment. Five fields do the work:

  • Who: the customer account the usage belongs to.
  • What: the meter, here transcription_seconds.
  • How much: an integer quantity in the meter's smallest unit, seconds here rather than minutes.
  • When: the time the usage happened, which decides the month it bills in, not the time the event arrived.
  • An id assigned by the producer and built from what the event describes, such as job_8812:chunk_3.

The last field is where the 1,190 minutes came from. The client SDK resends a chunk when the response times out, often after the first copy has finished. Each copy emitted an event with a random UUID minted at send time, so one piece of audio was counted twice, an overcharge: the customer loses money, and the provider loses trust. An id derived from the job and the chunk travels with the chunk and comes out the same on every attempt, and the pipeline can then deduplicate at ingestion, keeping the ids it has seen and dropping an event whose id is already there. Recording the id and adding the quantity to the total have to be one atomic step: done separately, a crash between them either loses the event or counts it twice.

The record of seen ids has to reach back further than the worst retry, and a backfill run twice by someone unsure the first one finished is a retry too. A send timestamp in the id breaks deduplication as surely as a random UUID: the id describes the usage, not the attempt. For the general problem of making a repeated operation count once, read Idempotency & exactly-once illusions.

One total per customer, meter and month

Behind ingestion, a stream processor keeps running sums keyed by customer, meter and billing period. The raw events stay stored, so a total computed wrongly is rebuilt from them. Absorbing the event volume itself is a throughput problem, the subject of High-write ingestion.

From a finished chunk to a reconciled invoice

Usage is deduplicated and summed before it is rated. Reconciliation checks the totals against a count that never passed through the meter.

The event that arrives on April 3rd

On the evening of March 31st one transcription node loses its connection to the event stream and holds its usage events on local disk until it is repaired, so usage from March 31st arrives on April 3rd. Without a stated rule, somebody's code makes one by accident, usually by dropping the event, which undercharges and leaves no trace.

A common rule has two parts. A period stays open for a grace window after it ends, say three days: late events for March still join March, and March's invoice waits until the window closes. After that the period is closed, and a March event arriving on April 9th becomes an adjustment, a line on the April invoice that names the period it corrects. The sent March invoice is never edited, since the customer may have paid and booked it. A longer window means fewer adjustments and later invoices. Stream processing: windows, watermarks & state explains how a stream processor decides that a window's data is complete.

Corrections take the same path: a closed period with a pricing bug is re-rated from its stored totals, and the difference is booked as an adjustment against it. Editing rows in place would lose the record of why the number changed, the first thing an auditor asks for; keeping such a record trustworthy over years belongs to Audit logs & tamper evidence. The network's $98 goes back the same way, as a credit on April's invoice.

From seconds to cents

Rating turns a quantity into money using the plan: unit prices, tiers, included allowances, minimums, discounts, proration on a mid-cycle plan change. This plan uses graduated tiers, where each minute is priced by the tier it falls in.

The other 1,260 minutes came from a rating rule applied in the wrong place. The plan measures usage by the second and rounds the month's total up to a whole minute, once. The meter, written before that rule, rounded each chunk up to a whole minute before emitting it. Chunks end at pauses, so almost every one of March's 2,520 chunks was rounded, by 30 seconds on average. Keeping quantities in the smallest unit until rating, and rounding once, at the end, gives the customer's number: 2,511,000 seconds are 41,850 minutes, and after the 1,000 free ones, 40,850 minutes at 4 cents is $1,634. Re-aggregating March's events would repeat the error, since they already carried rounded minutes; the corrected total comes from the audio lengths in the jobs database, which never passed through the meter.

Two more rules keep rating explainable. Rate against the price in effect when the usage happened: prices are versioned with effective dates, so an April price change does not reprice March; overwrite a unit price instead and any invoice recomputed from it silently changes. A price or plan change in the middle of a period needs the totals split at that date, or a re-rating from the stored events, since one monthly total cannot say how much usage fell on each side. And store the rated line items, not just the total: 1,000 minutes included, 40,850 minutes at 4 cents answers a customer's question, and $1,634 alone does not. Rating at read time, from raw events whenever the billing page opens, gives an answer that shifts as late events arrive.

NoteShow the customer the current month's usage in near real time, and say on the page that it is an estimate while the period is open. A customer who watches a live counter and then receives a different invoice will trust neither number again.

At close, the rating engine posts the rated lines to the billing ledger, an append-only record of what was billed (ledgers on their own are the Digital Wallet & Ledger Challenge), and asks the payment provider to charge, with retries and failed payments handled as Payments & billing integration describes. Stopping a free-tier account at its 1,000th minute while it uploads is enforcement rather than metering, and that live quota is a question for Multi-tenancy and Rate limiting.

Before the invoice leaves

None of these rules announces its own failure. What catches a fault like the network's is reconciliation, comparing the pipeline's numbers with counts that did not come through it, every day while the period is open and once more before the invoices go out. Three comparisons cover the path:

  • Metered against an independent count. The transcription service records every finished job and its audio length in its own jobs database, through code the meter never touches. In March it said 41,850 minutes and the aggregate 44,300: the whole dispute, before the invoice went out.
  • Billed against rated. Every rated line item has exactly one ledger line, for the same customer and amount, and the ledger has no others, so nothing was lost or doubled on the way to the invoice.
  • Collected against billed, counting what was charged before the provider's fees. After settlement, a difference is either a bug or a business event such as a failed charge or a chargeback.

A comparison needs a tolerance, because a few events may still be in flight when it runs; above it, that customer's invoice is held for a person instead of sent. Skip the run, and the difference is still found, by the customer, as it was on April 6th.

The Usage Metering & Billing Challenge asks for this path, down to a closed period recomputed after a pricing fix. Draw the four stages as separate boxes, label the edge into each with what makes it safe to re-run (event id, stored totals, price version), and give reconciliation its own box reading one count from outside the pipeline. If every number it compares came through the meter, it cannot catch a mistake the meter made.

Practice this