Concepts
Event time, windows, and watermarks
Intuition
Imagine counting how many letters were written each day, but the letters arrive at the post office with different delays: one written on Monday only shows up on Thursday. If you group by the day the letter arrived, the count is wrong; the right way is to group by the day it was written. But you can never be sure that all of Monday's letters have already arrived, so you need to decide when to publish a provisional result and what to do when a late letter shows up afterward.
How it works
The Dataflow model separates two time domains: event time (when the event actually occurred, which essentially never changes) and processing time (when the pipeline observes it, according to the system clock). Communication delays, scheduling and serialization create a dynamic skew between the two. For unbounded data, windowing slices the stream into finite pieces in event time: fixed (tumbling) windows of static size, sliding windows with a size and a slide period (fixed is the case where the two coincide), and sessions, delimited by an inactivity timeout, usually per key. To generalize, the model treats all windows as unaligned and uses two operations: AssignWindows assigns each element to zero or more windows, and MergeWindows merges windows at grouping time (GroupByKeyAndWindow), which allows sessions to join when intermediate events arrive. The watermark is a lower bound on the event times already processed; in most real cases it is heuristic, since the system does not know, for example, when a phone with no connection will start sending data again. Because it is sometimes too fast (leaving late data behind) and sometimes too slow (being global, a single slow datum holds back the pipeline), triggers come in; they define when, in processing time, to emit a window's result and can produce multiple panes per window; there are triggers based on the watermark (including percentile), on processing time, on data count, and compositions of these. Accumulation modes relate successive panes: discarding (each pane is independent), accumulating (each pane refines the previous one, keeping state) and accumulating and retracting (emits the retraction of the previous value before the new one, needed when serial groupings spread partial results across different keys). All of this boils down to four questions: what is computed, where in event time, when in processing time, and how refinements relate.
In practice
It is the reasoning behind billing pipelines, aggregate statistics, abuse and anomaly detection, recommendations, and user sessions in streaming: the paper associates billing with triggers, accumulation and retractions, and abuse detection with percentile watermarks. Because the model separates the pipeline logic from its physical execution, choosing batch, micro-batch or streaming becomes a tradeoff among correctness, latency and cost, without rewriting the logic. The authors present it as an alternative to the Lambda architecture, which requires maintaining two systems and is only correct if the input is complete when the batch runs; the design rule is to never depend on a notion of completeness.
Use it in a design
Practice the cases that rely on this concept in the System Design school.