A question that keeps being asked
A batch job asks its question once, over data that has stopped changing. Some questions cannot wait for the data to stop. A mobile game wants to know, minute by minute, how many matches each region completed, because when that number collapses the matchmaking servers in that region have usually failed, and players start leaving within minutes. Its clients publish every event, from match starts to disconnects, to a Kafka topic, tens of thousands a second, and storing them all to query a slice every minute rereads what the last query already read.
So keep the question running and push each event through it as it arrives. That is a stream processor: a program that runs continuously over an unbounded stream, a sequence of events with no last one, and updates its results as events come in. The stream it reads here is a durable log; the log itself belongs to Pub/sub: topics, subscriptions & replay, and a database's committed changes can feed the same kind of job once Change data capture has turned them into events.
The job is a chain of operators. The first ones parse each event, drop test devices and keep match_completed and match_abandoned. Each looks at one event and forgets it: they are stateless, so any instance can handle any event and a crash loses nothing that rereading the input will not rebuild. Counting matches per region is different. That operator carries eu-west's running number from one event to the next, which makes it stateful, like a join between streams, a deduplication or a session. State is kept per key, so the processor repartitions the stream by region and every eu-west event reaches the instance holding eu-west's count.
Cutting an endless stream into minutes
A count over an endless stream never finishes, so "matches per region" needs a boundary. A window groups events by time, and its shape is the question being asked:
- Tumbling windows have a fixed length and do not overlap: 19:41:00 to 19:42:00, then 19:42:00 to 19:43:00. Each event falls in exactly one. Matches per region per minute is a tumbling window.
- Sliding windows have a fixed length and start at a shorter, regular step, so they overlap: ten-minute windows starting every minute put each event in ten of them, which suits "abandoned matches in the last ten minutes, updated every minute".
- Session windows have no fixed length. A session per player stays open while their events keep arriving and closes after a gap of inactivity, say 15 minutes. Session length per player is a question no fixed window can ask.
The outage alert walks straight into a trap here. A window exists for a key only once an event for that key arrives. If eu-west's matchmaking fails completely, no match completes there, the job emits no eu-west window for that minute, and a rule that fires on "fewer than 50" has nothing to fire on: the outage the alert was built for is the one it cannot see. Count something that keeps arriving during the failure, such as match requests and their errors, and alert on the ratio, or set a timer per region that fires when a minute goes by without a result.
Two clocks on every event
Two times matter for every event. Event time is when it happened, stamped by the phone. Processing time is when the job sees it. Usually they are a second or two apart. Then a player on the metro finishes a match with no signal:
- At 19:41:10 the phone records
match_completedand holds it, offline. - At 19:42:00 a job that windows by processing time closes eu-west's 19:41 minute without it.
- At 19:45:30 the train surfaces, the event arrives, and that job counts it in the 19:45 minute.
One misplaced match is noise. But when the job comes back after twenty minutes down and works through the backlog in three, processing-time windows pour twenty minutes of matches into three: a hole, then a spike, neither of which happened, and a replay of last week gives different numbers from the first run. Windows by event time assign each event to the minute it happened, whenever it arrives, so the same input belongs to the same windows. They also raise a question processing time never had to answer.
When is 19:41 over?
By event time, an event for 19:41 can turn up at any moment, so the job needs a rule for when to stop waiting and publish the count. That rule is a watermark: the job's running estimate, carried along with the data, that no more events with timestamps up to some time T are expected. When the watermark passes 19:42:00, the 19:41 window fires.
Few sources can know that for certain, so the watermark is an estimate. A common form trails the largest event time seen so far by a fixed bound. With a two-minute bound, an event stamped 19:44:05 moves the watermark to 19:42:05, and the 19:41 window closes. With several input partitions the job follows the lowest of their watermarks, so a partition that goes quiet can hold every window open. The bound is a trade-off you choose: a tighter one publishes sooner and leaves out more stragglers, a looser one publishes counts that are more complete and older. For the alert, a tight bound has an unpleasant side: if a mobile network starts delaying events, windows close with many matches still in transit, and a network problem looks like a server outage.
An event whose timestamp the watermark has already passed is late data. There is always some, from the player on the metro to a phone that was switched off for a day, so the job needs a stated policy for it:
- Drop it. Many engines do this by default once a window has closed, often with only a metric to show for it, so a count that is always slightly low goes unnoticed.
- Keep the window open for an allowed lateness after it fires, and emit a corrected count when a late event arrives. Whatever reads the results must then accept updates, replacing eu-west's 19:41 row rather than adding a second one; in session windows a late event can even merge two sessions into one.
- Route late events to a separate output, for a correction job or a reconciliation to pick up later.
A crash in the middle of a minute
At 19:41:40 the job holds eu-west's count for the open minute, 211 so far, and a session for every active player, in memory and on local disk. One machine dies. The events are still in the log, but where should rereading start? From the newest offset, the 211 matches are lost; from the start of the topic, days are recounted.
The answer is a checkpoint: a periodic snapshot of every operator's state, taken together with the input offsets that state reflects, and written to durable storage off the machine. Flink takes them at a configured interval, often seconds to a minute, while the job keeps running. After a failure the job restores the last completed checkpoint and rewinds its input to the offsets recorded in it, so every event after the checkpoint is processed again, against state that has not seen it yet. Saving the two independently would not work: a crash between the writes leaves state that has counted events the offsets say were never read, or the reverse, and the replay counts them twice or not at all.
Each checkpoint stores the operators' state and the Kafka offsets it reflects, together. A restore rewinds both to the same point.
This is what a stream processor means by exactly-once: after any number of failures, its state reflects each input event once. The events themselves may be processed more than once; work done after the last checkpoint is discarded with the state and redone. Two things stay outside that promise.
- What the job sent out. Output emitted between the checkpoint and the crash goes out again: the 19:41 count reaches ClickHouse twice, and the on-call engineer is paged twice. End to end, exactly-once needs the sink's help: a transactional sink commits output only when the checkpoint completes; Flink does this with Kafka transactions, and consumers that read only committed data see each result once, after waiting for that checkpoint. The other route is a write that lands the same however often it runs, keyed by region and window, as built in Counting at scale.
- Duplicates already in the log. A client that retried a send puts the same match in the topic twice, and the processor counts both, each exactly once; an event id and the Idempotent Consumer pattern remove them, and the wider argument is in Idempotency & exactly-once illusions.
The report that can wait until morning
The studio also wants matches and revenue per region per day by 08:00. A stream job would carry a watermark, a lateness policy and checkpointed state, and would have to be kept running and upgraded without losing that state, for a number nobody reads before morning. A batch job at 03:00 over the finished day needs none of it: most stragglers have had hours to arrive, a later rerun can pick up the rest, nothing survives between runs, and a bug is fixed by running it again; Batch processing: partitions, shuffles & backfills shows how such a job runs and reruns without double counting. Scheduled loads are the business of ETL/ELT & data orchestration, and the store they fill that of Analytics data: OLTP vs OLAP & the warehouse. What decides is who acts on the number, and how soon; many systems run both over the same log, streaming for decisions now and batch for the number of record.
Streaming Analytics on GCP asks for this job on Google Cloud, Dataflow in the stream processor's place, with exactly-once windows, watermarks, and late events kept rather than dropped. Settle three things before placing a node, which clock the windows follow, how long they wait and where a straggler goes, and Ad Click Aggregation will ask them again at 100,000 clicks a second.