Stream Processing: Flink, Kafka Streams & Windowing

Q: Is exactly-once processing real?

Exactly-once effect is real and is what engines provide, via consistent checkpoints plus transactional or idempotent sinks. Exactly-once delivery over a network is impossible — acknowledgments can be lost. In practice, at-least-once with idempotent sinks gives the same correctness with lower latency and is the more common production choice.

Stream Processing

Stream processing computes results continuously over data that never ends. Instead of a job that wakes at 2am, scans yesterday's table, and writes a summary, a streaming job stays running, consumes events as they arrive, maintains state, and emits updated results within seconds.

The conceptual shift is that a stream and a table are two views of the same thing: a table is the current state produced by a stream of changes, and a stream is the log of changes that produced a table. Once that clicks, "keep a running 5-minute count per user" and "maintain a materialized view" become the same operation — which is why modern engines let you express streaming logic in SQL.

The hard part isn't throughput. It's that events arrive late and out of order, and every interesting question ("how many orders in the 10:00–10:05 window?") requires deciding how long to wait for stragglers.

TL;DR

Streaming processes unbounded data continuously; batch processes bounded data on a schedule. Batch is a special case of streaming.
Event time (when it happened) is almost always what you want; processing time (when you saw it) is easy but produces wrong answers under delay.
Windows bucket infinite streams into finite chunks: tumbling, hopping/sliding, and session windows.
Watermarks are the engine's estimate of event-time progress — they decide when a window can be closed and how late is too late.
State is what makes streaming hard: aggregations and joins hold state that must be checkpointed to survive failures.
Exactly-once means exactly-once effect — checkpointed state plus transactional sinks, not magic in the network.
Pick Flink for complex event-time logic, Kafka Streams for a library inside a JVM service, Spark Structured Streaming when you already run Spark.

Quick Example

A five-minute tumbling count per product, in Flink SQL:

The equivalent in Kafka Streams, as a library inside your own service:

Core Concepts

Streams and tables are dual

An aggregation turns a stream into a table; observing a table's changes turns it back into a stream. This duality is why change data capture integrates so naturally with streaming — CDC is the changelog of a database table.

Event time vs. processing time

A mobile client goes offline for ten minutes and uploads its buffered events at 10:12. Under processing time, a purchase that happened at 10:03 is counted in the 10:10 window — the 10:00 window's numbers are simply wrong, and rerunning the job tomorrow gives a different answer than it gave today. Under event time, it lands in the 10:00 window, and the result is reproducible.

Use processing time only for genuinely "now"-shaped questions like rate limiting or liveness monitoring.

Windows

Watermarks

A watermark is an assertion: "no event with a timestamp earlier than T will arrive from here on." It's a heuristic, and its lateness allowance is a direct latency/completeness tradeoff.

Three knobs, each with real consequences:

Bounded out-of-orderness — how long to wait. Larger = more complete, more latency, more retained state.
Allowed lateness — how long after closing to keep a window's state and re-emit updated results for stragglers.
Side output for late data — where events past the deadline go. Route them somewhere; don't drop them silently.

⚠️ Warning: The watermark is the minimum across all input partitions. One idle or dead partition freezes it, and every window stops closing while the job looks perfectly healthy. Always configure idleness detection.

State

Any operator that remembers something across events is stateful: aggregations, joins, deduplication, pattern matching. State lives in an embedded key-value store (RocksDB, typically) next to the operator, and is checkpointed to durable storage periodically.

State TTL is not optional for any keyed state whose key space grows forever (session IDs, request IDs, device IDs). Unbounded state is the most common way a streaming job dies at 3am.

Delivery Semantics

"Exactly-once" is exactly-once effect, not exactly-once delivery — the latter is impossible over an unreliable network. The mechanism is:

The engine takes consistent checkpoints of all operator state plus source offsets (Flink uses the Chandy–Lamport barrier algorithm).
On failure it restores the whole job to the last checkpoint and rewinds the sources.
Output is made atomic with that checkpoint via a transactional sink — Kafka transactions, or a two-phase-commit sink.

The catch: with transactional Kafka output, downstream consumers reading read_committed see records only when a checkpoint commits. Checkpoint every 10 seconds and your end-to-end latency floor is ~10 seconds. If your sink is naturally idempotent (an upsert keyed by primary key), at-least-once plus idempotent writes gives the same correctness with much lower latency — and is what most production pipelines actually run.

Stream Joins

Joining two unbounded streams requires bounding the state somehow.

Interval join — match events within a time window of each other:

State is retained for the interval width. A 30-day interval join is a 30-day state buffer — usually the wrong design.

Stream-table (temporal) join — enrich a stream from a slowly-changing dimension, matching the dimension's value as of the event's time:

This is the most common production join, and CDC is the usual source for the dimension side.

Windowed join — both streams bucketed into the same windows and joined per window.

Comparison

How to choose: if you already run Spark for batch, Structured Streaming avoids a second system. If your logic lives inside one Kafka-consuming service, Kafka Streams needs no cluster at all. If you have genuinely hard event-time semantics — sessionization, complex windowing, pattern detection, exactly-once across heterogeneous sinks — Flink is the reference implementation. If the requirement is "a SQL view that stays fresh," a streaming SQL engine may replace the whole pipeline.

Do you need streaming at all?

Streaming is a real operational commitment: a stateful, always-on job whose failure modes (state growth, watermark stalls, checkpoint duration, backpressure) are unfamiliar to most teams. Before committing, check whether micro-batch every 1–5 minutes (Airflow or dbt on a short schedule) meets the requirement. It very often does, and it's dramatically easier to reason about, test, and backfill. Reserve streaming for cases where seconds genuinely matter: fraud detection, real-time personalization, operational alerting, live dashboards, and dynamic pricing.

Best Practices

Use event time, and put the timestamp in the event

Producers should stamp events at creation with a clock you trust. Deriving event time from ingestion order makes every result irreproducible on replay.

Bound every piece of state

Any keyed state on an unbounded key space needs a TTL, or the job runs until the disk fills.

Key by something with even cardinality

Streaming parallelism is per-key. If 40% of events share one key, one subtask does 40% of the work and backpressures the whole job. Salt hot keys (key + "#" + hash % 8) and re-aggregate — the same reasoning as database sharding.

Tune checkpoints against your latency target

Frequent checkpoints mean faster recovery and, with transactional sinks, lower output latency — but more overhead. Start at 10–60 seconds, enable incremental checkpoints with RocksDB, and alert on checkpoint duration: a checkpoint that takes longer than the interval means the job can never keep up.

Capture late data instead of dropping it

Silent data loss is much worse than a small reconciliation job.

Plan for reprocessing from day one

You will need to replay history after a logic bug. Keep source retention long enough to rewind (or tier to object storage), make sinks idempotent so a replay overwrites rather than duplicates, and version your output so a rebuild can run alongside the live job before you swap.

Monitor the streaming-specific signals

Consumer lag, watermark lag, checkpoint duration and failure count, state size per operator, and backpressure. Ordinary CPU/memory dashboards will show a healthy job that is silently hours behind. See Monitoring and Alerting.

Common Mistakes

Using processing time by default

Unbounded keyed state

Ignoring idle partitions

An unbounded stream-stream join

Non-idempotent sinks under at-least-once

FAQ

What's the difference between stream processing and batch processing?

Batch runs over a bounded dataset and terminates; streaming runs over an unbounded one and doesn't. Modern engines unify them — the same Flink or Spark SQL runs in both modes — but streaming adds event-time semantics, watermarks, and persistent state, which is where the real complexity lives.

Flink or Kafka Streams?

Kafka Streams if your logic is a library inside an existing JVM service reading and writing Kafka, and you don't want to operate another cluster. Flink if you need non-Kafka sources or sinks, non-JVM languages, sophisticated event-time handling, or exactly-once across heterogeneous systems.

Is exactly-once processing real?

Exactly-once effect is real and is what engines provide, via consistent checkpoints plus transactional or idempotent sinks. Exactly-once delivery over a network is impossible — acknowledgments can be lost. In practice, at-least-once with idempotent sinks gives the same correctness with lower latency and is the more common production choice.

How late is "too late" for an event?

A product decision, not a technical one. Ask: how much accuracy does an extra minute of waiting buy, and what does that minute of latency cost? Common practice is a small watermark delay (seconds) plus allowed lateness (minutes) to emit updates, plus a side output feeding a nightly batch reconciliation for the long tail.

Do I need streaming, or is a 5-minute batch enough?

Be honest about the requirement. If the consumer is a dashboard a human reads, a 5-minute dbt run is almost certainly enough and vastly simpler. Streaming earns its complexity when a machine acts on the result in seconds — fraud blocking, real-time bidding, live personalization, operational alerting.

How do I test a streaming job?

With a test harness that lets you inject events with explicit timestamps and advance the watermark manually (TestHarness in Flink, TopologyTestDriver in Kafka Streams). Test the out-of-order and late-arrival paths specifically — they're where the bugs are, and they never show up under in-order test data.