Streaming Systems learning path: everything to study, topic by topic

Understanding the false dichotomy between batch and streaming: streaming is a superset that processes unbounded data with low latency, while batch is streaming over a bounded dataset

The nature of unbounded (infinite) datasets: data that arrives continuously with no known end, requiring incremental processing and a strategy for emitting results before all data has arrived

The fundamental duality: a stream is the changelog of a table, and a table is the accumulated snapshot of a stream. Grouping converts a stream to a table; ungrouping converts a table back to a stream

The two domains of time: event-time (when the event actually occurred) vs processing-time (when the system observes it), and the variable, unpredictable skew between them

The four questions that structure any out-of-order data processing pipeline: WHAT results are computed (transforms), WHERE in event-time they are computed (windowing), WHEN in processing-time they are materialized (triggers/watermarks), and HOW refinements relate (accumulation)

Fixed (tumbling) windows partition event-time into contiguous, non-overlapping intervals of equal size, with each element assigned to exactly one window

Sliding (hopping) windows defined by a size and a period; when the period is smaller than the size, windows overlap and a single element belongs to multiple windows

Data-driven, dynamic windows that group bursts of activity separated by gaps larger than a configured gap duration; window extents depend on the data itself, not a fixed schedule

The merging mechanics that make session windows work: each element initially proto-windows, then overlapping windows merge; merging interacts with triggers, watermarks, and state

A watermark is a notion of input completeness in event-time: a monotonic assertion that no more data with event-time earlier than the watermark is expected to arrive

Perfect watermarks guarantee no late data but require perfect knowledge of the input; heuristic watermarks estimate completeness and may be wrong, producing late data that must be handled

How watermarks flow through a multi-stage pipeline: input/output watermarks per stage, the min across inputs and across keys/shards, and how the slowest source bounds overall event-time progress

Diagnosing pipeline health via watermark skew (how far the watermark lags real time) and the interplay between completeness, latency, and stuck watermarks caused by stragglers or slow sources

Triggers that fire based on the watermark passing the end of a window, materializing a single on-time result when event-time completeness is reached

Triggers that fire on processing-time intervals (e.g., every N seconds) to deliver low-latency, regularly-updated partial results regardless of event-time completeness

Triggers that fire based on properties of the data accumulated in a window, such as a count of elements or a size threshold, rather than on time

Combining triggers to express realistic policies: early speculative firings on processing-time, an on-time firing at the watermark, and late firings as late data arrives (the early/on-time/late pattern)

How successive pane firings relate: discarding (deltas), accumulating (cumulative overwrites), and accumulating-and-retracting (emits a retraction of the prior value plus the new value)

Handling data that arrives after the watermark has passed its window's end (late data), which occurs whenever a heuristic watermark advances too aggressively

The allowed-lateness horizon bounds how long past the watermark window state is retained so late data can still be processed; beyond it, state is garbage-collected and later data is dropped

Achieving effectively-once output semantics despite retries and failures via deterministic operations, checkpointing, idempotent or transactional sinks, and deduplication

General stateful processing primitives — persistent per-key state and event-time/processing-time timers — that underpin windowing, joins, and custom incremental aggregations

The unified Beam/Dataflow model that separates the WHAT/WHERE/WHEN/HOW questions from the runner, letting one pipeline run in batch or streaming across engines with consistent semantics

Lambda architecture runs parallel batch and speed layers and reconciles them; Kappa serves all queries from a single replayable streaming layer. The semantic argument for a single streaming system

The deeper streams-and-tables view that recasts the whole model: grouping/triggers turn a stream into a table and ungrouping turns a table into a stream, unifying batch, streaming, SQL, and joins