Apache Beam: Portable Batch & Streaming Pipelines

Apache Beam

Apache Beam is an open source, unified programming model for batch and streaming data pipelines. You describe a pipeline once — reading data, transforming it, grouping and aggregating it, writing results — and run it on different execution engines called runners: Google Cloud Dataflow, Apache Flink, Apache Spark, and local runners for development. Beam grew out of Google's internal data processing systems and the Dataflow model, which formalized how to handle unbounded, out-of-order data correctly.

Beam's main contribution is a precise vocabulary for streaming: event time versus processing time, windows, watermarks, triggers, and accumulation. These concepts answer the questions every streaming system faces — what results are computed, where in event time, when they are emitted, and how later data refines them — and they apply even if you never write a Beam pipeline.

TL;DR

A Pipeline applies PTransforms to PCollections (bounded for batch, unbounded for streaming).
Core transforms: ParDo (per-element processing with DoFns), GroupByKey, CoGroupByKey, Combine, Flatten, Partition.
Streaming correctness comes from event-time windows, watermarks, triggers, allowed lateness, and accumulation modes.
Pipelines run on runners — Dataflow, Flink, Spark, or local runners — with differing capabilities and guarantees.
SDKs for Java, Python, and Go, plus Beam YAML; cross-language transforms share connectors between SDKs.
Runners retry bundles, so DoFn side effects must be idempotent; test with TestPipeline and assertions.

Quick Example

A streaming Python pipeline: read order events from Pub/Sub, window by event time, sum revenue per store each minute, and allow late data for ten minutes:

A unit test for the composite transform using bounded test data:

Core Concepts

The Programming Model

Event Time, Windows, and Watermarks

Streaming data arrives late and out of order: a phone uploads yesterday's events this morning. Beam computes over event time (when things happened) rather than processing time (when they arrived), by assigning elements to windows:

The watermark is the runner's estimate of how far event time has progressed — "we have probably seen all data up to 12:05". When the watermark passes the end of a window, results can be emitted.

Triggers, Lateness, and Accumulation

Triggers decide when to emit results for a window: at the watermark, early on a processing-time schedule for speculative results, and again when late data arrives within the allowed lateness. Accumulation mode decides whether later firings include earlier data (accumulating) or only new data (discarding). Together they answer the Dataflow model's four questions: what is computed, where in event time, when in processing time results are emitted, and how refinements relate.

Runners and SDKs

The capability matrix documents which features each runner supports. SDKs exist for Java (most complete), Python (popular for ML and data science), and Go, plus Beam YAML for declarative pipelines. Cross-language transforms let Python and Go pipelines use Java connectors such as Kafka I/O.

State, Timers, and Splittable DoFns

Stateful DoFns keep per-key, per-window state (values, bags, maps) and set timers in event or processing time — useful for deduplication, sessionization with custom logic, and buffering. Splittable DoFns let sources and expensive transforms split work dynamically for parallelism.

Execution Semantics

Runners process elements in bundles and retry failed bundles, so a DoFn may see the same element more than once. Exactly-once guarantees apply to Beam's own state and shuffles on runners that provide them — not to external side effects. Write to sinks idempotently (deterministic keys, upserts, transactional sinks) and avoid side effects inside DoFns that cannot be repeated safely. Dead-letter outputs route unparseable or failing elements to a separate sink instead of failing the pipeline. See Stream Processing.

Testing and Operations

Unit test DoFns and composite transforms with TestPipeline, Create, and assert_that.
TestStream simulates watermarks and late data to test windowing and triggers.
Integration test connectors against emulators (Pub/Sub, Kafka) or sandboxes.
Monitor watermark lag, backlog, throughput, error outputs, and per-step latency in the runner's UI.
Update and drain streaming jobs deliberately: in-place updates need compatible transforms; draining finishes in-flight windows before stopping.
Backfill by running the same pipeline in batch mode over historical data. See Pipeline Backfills and Historical Corrections.

Best Practices

Use Event Time and Explicit Lateness

Window on event timestamps, set allowed lateness from observed delays, and decide what happens to data later than that.

Make Sinks Idempotent

Assume bundles are retried and design writes so duplicates are harmless.

Build Composite Transforms

Package logic into named, tested PTransforms and keep I/O at the edges so the same logic runs in batch, streaming, and tests.

Check Runner Capabilities

Verify that your runner supports the features you rely on — triggers, state, timers, update semantics — before committing to it.

Common Mistakes

Processing-Time Aggregation

Non-Idempotent Side Effects in a DoFn

One Bad Record Stops the Pipeline

FAQ

What is Apache Beam used for?

Building batch and streaming data pipelines with one programming model that runs on multiple execution engines, such as Google Cloud Dataflow, Apache Flink, and Apache Spark.

What is the difference between Beam and Dataflow?

Beam is the open source programming model and SDKs. Dataflow is Google Cloud's managed service that executes Beam pipelines as one of several runners.

What is a watermark in Beam?

The runner's estimate of event-time progress — the point before which it believes all data has arrived. Windows are typically finalized when the watermark passes their end.

How does Beam handle late data?

Windows can allow lateness for a set period; triggers fire again when late data arrives, and accumulation modes determine whether updated results include earlier data.

Does Beam guarantee exactly-once processing?

Some runners, such as Dataflow, provide exactly-once processing for Beam's internal state and shuffles. Side effects to external systems still need idempotent or transactional writes.