Every graph was green
The support ticket says: "Downloading an invoice spins for half a minute, then fails. Third time this week." The engineer who picks it up opens the invoicing service's dashboard: six instances, CPU at 35%, 0.2% of requests failing, no alert in a month. Nothing on the screen is false, and nothing on it can say which customer, which invoice or which step failed.
Observability is how well you can answer a question like this one from what a running system emits, including questions nobody thought of when the dashboard was built. Following the ticket to its cause takes three kinds of record.
Three questions, three kinds of record
- Is something wrong, and how much? Metrics: numbers aggregated over time, cheap to keep for months and quick to chart. They can say that one route has failed 4% of the time since Monday, not which requests.
- What exactly happened to this request? Logs: one record per event, with whatever detail the code wrote down. Costly by volume, and usually where the specific customer and error live.
- Where did the time go? Traces follow one request through every service it touches, as a tree of timed steps. Distributed tracing covers how a trace is built, carried between services and sampled.
Counting without remembering
A metric keeps a number, not the events behind it, so it costs the same whether it counted ten requests or ten million. A counter only goes up, like requests served or errors returned, and the store turns it into a rate such as errors per second over the last five minutes; it drops to zero when the process restarts, and rate functions are written to absorb that. A gauge goes up and down: queue depth, memory in use, open connections. A histogram counts observations into latency buckets, under 100 ms, under 250 ms and so on, so a percentile such as the 99th can still be estimated across all instances; why buckets add up where percentiles do not is in Latency percentiles & tail latency.
Two habits decide what to measure. For every service, RED: the rate of requests, the errors, and the duration. For every resource, such as a disk, a pool or a queue, USE: utilization, saturation (work waiting for it) and errors. The invoicing service had USE on its machines and RED only for the service as a whole, so a 4% failure on its PDF route, a small share of traffic, showed up as 0.2% overall.
The label that ate the metrics store
The fix looks obvious: split the request metrics by label, here a route label with the route template as its value. Each distinct combination of metric name and label values is its own time series, and the store pays memory and index space for every one of them. The latency histogram is 14 series per combination (twelve buckets, a sum and a count); with 8 routes and 5 status codes on 6 instances, that is 3,360 series, a light load for a metrics store.
Then someone, chasing which customers are affected, adds customer_id. Thirty thousand active customers, each seen on about three route-and-status pairs, turn 3,360 series into as many as 30,000 × 3 × 14 × 6, about 7.5 million. The store slows down or runs out of memory, and the dashboards go with it. That is cardinality, the number of distinct series, and it is the cost that explodes, because one careless label multiplies everything already there. Labelling with the raw path, /invoices/88231/pdf instead of the template, does the same thing more quietly.
So a label takes values from a small, fixed set: route template, status code, region, instance. A detail with an unbounded set of values, such as a customer id or an invoice number, belongs on the event, which is the next record type.
One line per thing that happened
The invoicing service did log, in prose: Render failed for invoice 88231 after 30001ms. Finding every failure in a week of those means grepping raw text, one pattern per wording, under pressure. A structured log writes the same event as fields:
{"level":"error","request_id":"r-7f3a","customer":"c-4410","route":"/invoices/:id/pdf","line_items":1240,"duration_ms":30001,"msg":"render timed out"}
Now the question is a query: errors on the PDF route this week, grouped by customer. Thirty-seven customers, and every failing invoice has more than a thousand line items and times out at 30,000 ms. A log line is an event, not a series, so the unbounded customer and line_items add bytes to the line, not a series per customer, provided the log store does not turn them into labels.
One field earns a place on every line: a request id, created where the request enters and passed along to every service it calls, so all the lines one request produced can be pulled out together. With tracing in place, the trace id plays that role, and each line links to the trace that shows which hop spent the 30 seconds. Crashes get their own treatment: an error tracker such as Sentry groups thousands of reports of the same stack trace into one issue.
Events cost by volume. Six instances at 300 requests a second each, five lines per request and about 400 bytes per line come to roughly 310 GB a day to ship, index and store, more during an incident, so keep logs searchable only as far back as investigations reach, use levels that actually filter something, and sample debug lines per request id while keeping every warning and error.
Getting it out of the process
Prometheus pulls metrics from the service; a log agent pushes the service's lines to Loki. Neither backend sits on the request path.
Neither signal should make a request wait on monitoring: incrementing a counter is a write to memory, and a log line is a quick write to standard output, as long as whatever reads it keeps up. Metrics are then pulled: Prometheus fetches each instance's /metrics page on an interval, often 15 or 30 seconds, and records a target that stops answering as down, though a scrape that succeeds proves only that the process can serve that page (Health Endpoint Monitoring covers what a real check tests). A job that ends between two scrapes pushes its numbers to a gateway instead, and a mobile app nobody can scrape sends them to an ingestion endpoint. Logs go out through an agent on each host, Fluentd here, which batches lines and buffers while the store is slow, and at larger scale a durable buffer such as Kafka absorbs an incident's burst. The agent's buffer cannot keep a write inside the service from waiting, though, so the logger itself should queue lines and, when that queue fills, drop them, debug first, rather than block the request.
Page on what the customer feels
The ticket arrived through support because the service's two alerts watched CPU above 80% and the overall error rate above 1%. CPU is a cause: it can run hot while every customer is served, or stay cool while one route fails for everyone who calls it. Alerts on causes tend to be noisy alerts nobody trusts, and this one was silent when it mattered.
Page on symptoms: the error ratio and latency of the routes customers actually call, measured where their requests arrive, at the load balancer or gateway. Causes belong on dashboards, where they explain a symptom, and in tickets, except one about to become a symptom with near certainty, such as a disk that will be full in four hours at its current rate. A fixed threshold on the error ratio has its own flaw, paging on a ninety-second blip or staying silent through a slow leak, depending on where it is set; SLOs & error budgets replaces it with burn-rate alerts. Keep dashboards few, one per service with its RED panels, before a hundred nobody opens hide the one that matters.
At full size
Metrics Monitoring & Alerting asks for this pipeline at the scale of millions of active series, so its hardest question is the one the invoicing team met by accident: which labels will you refuse? Its log counterpart is Centralized Log Aggregation, and the error tracker is Crash Reporting & Symbolication.