Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

High-write ingestion

Absorbing a write rate far above the read rate: a log in front of storage, batched writes, partition keys and hot keys, stores built for appends, expiry and downsampling, and deciding in advance which data may be lost.

Sixty readings, once a second

A wind-farm operator runs about 4,000 turbines across 60 sites. Every turbine sends one message a second with 60 readings in it: vibration on the main bearing and the gearbox, temperatures, generator power, wind speed, blade pitch. Two lines of arithmetic:

  • 4,000 messages a second × 60 readings = 240,000 points a second.
  • 240,000 points × 86,400 seconds = about 21 billion points a day, roughly 330 GB at 16 bytes a point before compression.

The reads, against that, are tiny: control-room dashboards for 60 sites, an alerting job, a reliability engineer pulling one gearbox's history every few days. Almost nothing is read one row at a time. Telemetry, metrics, location pings and clickstreams all have this shape, and a design built for ordinary request-and-response traffic fails on it at the database. Doing these two lines first is what Back-of-envelope estimation teaches: whether the answer is megabytes or gigabytes a minute changes every box that follows.

240,000 INSERTs a second

The pilot, on two sites, had each site gateway insert every reading into a relational table as it arrived. One row per statement means every point pays for a network round trip, a transaction and an index update, overhead many times the size of a 16-byte reading. At 20 sites the database fell behind. Then it paused for a failover, the gateways' inserts timed out, and those readings were gone, because there was nowhere for them to wait. The redesign lowers what each point costs at every stage, and gives the writes a place to wait.

A log to wait in

The gateways now append to a log: a durable, append-only sequence of records, such as a Kafka topic, that producers add to at the end and consumers read at their own pace. Appending is cheap, and the log keeps records for a set retention, three days here, whether anyone has read them or not. When storage slows down or pauses, the consumers fall behind and the log grows; while the log itself is up and has room, the gateways do not notice. Consumers working through a backlog at their own speed are the subject of Message queues & async workers, and flattening a burst that way has a pattern name, Queue-Based Load Leveling.

The log is split into partitions, and each record's key decides its partition. Partitions are the unit of parallelism: within a consumer group, each partition is read by one consumer at a time, so their number caps how many consumers can share the work. The operator keys by turbine id, which keeps each turbine's readings in one partition, in the order they were appended, and spreads 4,000 turbines roughly evenly over 48 partitions. Keying by site would have looked tidy and spread the load badly. The largest site has 300 turbines, so its whole stream, 18,000 points a second, would land on a single partition and its one consumer, more than three times the 5,000 a partition averages. If that is more than one consumer can write, it is a hot key: a single key value with more traffic than its partition can take, which adding consumers cannot fix, because only one of them reads that partition. For key choice beyond this one log, see Sharding & partitioning.

Thousands of rows per write

The 48 consumers do the second half. Each one handles about 5,000 points a second and, instead of one INSERT per point, writes them as one bulk statement once a second. The round trip and the transaction are then paid once per batch rather than once per point (index updates still happen per row). How much throughput that buys depends on the store, the row size and the network; for small rows it is commonly ten times or more. The price is latency: a point can wait up to a second before it is stored, a delay the operator accepted for its dashboards and its bearing-temperature alerts. A failed batch is retried from the log and may land twice, and any count, sum or mean over those points can then come out wrong unless the write is idempotent (Idempotency & exactly-once illusions); anything that ends up on an invoice needs the per-event care described in Usage metering & billing pipelines.

The batches land in a time-series database, built, like many of them, so that a write is a sequential append rather than an update in place; Data structures behind big systems opens up the LSM tree that makes that cheap.

Seven days of raw, years of rollups

Keeping every point forever is a cost curve that only bends up, and nobody reads year-old per-second points. So retention is decided when the data is written, not later. Raw points go into tables partitioned by day and expire after seven days: dropping a whole day's partition is a cheap metadata operation, while deleting a billion expired rows one by one, months later, can become an outage of its own. Stores with a TTL on records do the same job.

What lives longer is downsampled: a stream job rolls the raw data into 1-minute aggregates, then 1-hour aggregates, kept for a year and for the turbine's life respectively. Dashboards read the rollups, never the raw table, because a chart pointed at raw data is a large scan competing with the incoming stream for the same disks. The alerting job is the exception: it queries only the last few minutes of raw points, a small read that sees each point as soon as its batch is stored. Windowed aggregation, late points included, is covered in Stream processing: windows, watermarks & state. Rebuilding a rollup after a bug means recomputing it from raw, a batch job (Batch processing: partitions, shuffles & backfills) that can reach back only as far as raw is kept.

The IoT Fleet Telemetry Challenge asks for this pipeline: an IoT gateway, downsampling before storage, a time-series database with retention tiers, dashboards and alerts. Metrics Monitoring & Alerting puts the same split under alert rules that need fresh data.

Gateways, a log, batches, two tiers

Readings wait in the log, partitioned by turbine, until consumers write them in batches. Raw points expire after seven days; dashboards read the rollups.

When the log stops being a cushion

The log buys time, not capacity. If storage writes fewer than 240,000 points a second for long enough, the backlog grows until the log's retention deletes records nobody consumed, so the operator watches consumer lag in hours against those three days. Pushing back on producers and shedding load belong to Backpressure & load shedding; what this design decides, in advance and per kind of data, is what may be lost:

  • Per-second power and wind readings may be thinned to one in ten; the 1-minute mean, built from six points, gets noisier but still shows the trend, and its maximum can miss a short peak.
  • Temperatures may wait.
  • Alarm and fault events have their own topic and consumers, and are never thinned.
  • Each site gateway buffers a few hours locally, so a cut link repaired within those hours delays readings instead of losing them.

What the rollups threw away

Each of those choices discards something, and an honest design says what. A 1-minute vibration rollup keeps the mean, minimum, maximum and count; it cannot say how the signal moved within the minute, and a spike seen in one second's reading survives only as the minute's maximum, with no record of when in the minute it came. Seven days of raw means a gearbox fault noticed on day ten cannot be studied at full resolution. Thinned power readings mean some minutes rest on six points instead of sixty.

Before cutting raw retention from 30 days to seven, the reliability team checked that this was safe. They took the past year's 41 confirmed component failures and asked whether each investigation could have been done with seven days of raw plus the rollups. Thirty-nine could. The other two had drifted for weeks before anyone looked, and the answer was a rule rather than more retention: when a turbine's vibration rollup trends upward, its raw points are copied to cheaper long-term storage before they expire. Since then, every request for data that was already deleted is logged, and the count is reviewed each quarter; a rising count means a retention choice needs revisiting.

Practice this