Apache Beam Tutorial: Unified Batch and Stream Processing (2026)
Apache Beam provides a unified programming model for both batch and stream processing, and after deploying Beam pipelines across Google Cloud Dataflow, Apache Flink, and Apache Spark runners, I appreciate how it decouples pipeline logic from execution engine. Write once, run anywhere is not just a slogan — it genuinely works when your infrastructure changes.
This tutorial covers Beam's model, PTransforms, windowing, triggers, side outputs, and runner selection for building portable data processing pipelines.
Beam Programming Model
Beam programs are built from PCollections (datasets) and PTransforms (operations on datasets). A PCollection is immutable and distributed — it represents a finite (bounded) or infinite (unbounded) collection of elements. PTransforms consume one or more PCollections and produce new PCollections. The pipeline object manages the dataflow graph.
The Beam model handles two critical aspects: windowing (grouping elements by time) and triggering (determining when to emit results). This separation lets you control when results are produced independently from how elements are grouped.
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
options = PipelineOptions([
'runner=DataflowRunner',
'project=my-gcp-project',
'region=us-central1',
'staging_location=gs://bucket/staging',
'temp_location=gs://bucket/temp'
])
with beam.Pipeline(options=options) as p:
lines = p | 'Read' >> beam.io.ReadFromText('gs://bucket/input/*.txt')
words = lines | 'Split' >> beam.FlatMap(lambda line: line.split())
counts = (
words
| 'Pair' >> beam.Map(lambda word: (word, 1))
| 'Count' >> beam.CombinePerKey(sum)
| 'Format' >> beam.MapTuple(lambda word, count: f'{word}: {count}')
)
counts | 'Write' >> beam.io.WriteToText('gs://bucket/output/counts')Windowing Strategies
Beam supports fixed (tumbling), sliding, session, and global windows. Fixed windows divide the timeline into non-overlapping intervals. Sliding windows overlap, providing smoothed aggregations. Session windows group elements by activity gaps — no fixed time boundary. Global windows apply to all elements but are useful only with custom triggers.
Windowing is essential for unbounded PCollections because you cannot aggregate infinite data. Each element is assigned to one or more windows based on its event timestamp.
from apache_beam import window
# Fixed windows (tumbling) — 1 minute
words | beam.WindowInto(window.FixedWindows(60))
| beam.CombinePerKey(sum)
# Sliding windows — 1 minute window, 10 second slide
words | beam.WindowInto(window.SlidingWindows(60, 10))
| beam.CombinePerKey(sum)
# Session windows — 5 minute gap threshold
words | beam.WindowInto(window.Sessions(300))
| beam.CombinePerKey(sum)
# With timestamp extractor
words | beam.WindowInto(
window.FixedWindows(60),
timestamp_extractor=lambda event: event['timestamp']
)Triggers and Watermarks
Triggers determine when results are emitted from a window. The default trigger fires when the watermark passes the window end. Beam provides composite triggers: AfterWatermark (emit when watermark passes), AfterProcessingTime (emit after wall-clock delay), AfterCount (emit after N elements), and Repeat (fire repeatedly).
Watermarks estimate progress through event time. For unbounded sources, Beam generates watermarks based on the source's capabilities. For bounded sources, the watermark jumps to infinity at the end of the input.
from apache_beam import window, trigger
# Custom trigger: fire early and late
words | beam.WindowInto(
window.FixedWindows(60),
trigger=trigger.AfterWatermark(
early=trigger.AfterProcessingTime(10), # early results every 10s
late=trigger.AfterCount(1) # fire on each late element
),
accumulation_mode=trigger.AccumulationMode.DISCARDING,
allowed_lateness=300 # 5 minutes
) | beam.CombinePerKey(sum)
# Accumulation modes:
# DISCARDING — each firing only has new elements
# ACCUMULATING — each firing has all elements since window startSide Inputs and Outputs
Side inputs let a PTransform access additional data beyond its main input. Use cases include lookup tables, configuration data, and enriched reference data. Beam serializes side inputs and ships them to workers — keep them small for performance.
Side outputs route elements from a single transform to multiple downstream PCollections. This is useful for filtering, branching, and error handling patterns.
from apache_beam import pvalue
def enrich_event(event, lookup_table):
enriched = lookup_table.get(event['user_id'], {})
return {**event, **enriched}
# Side input as dict
lookup = p | 'Load Lookup' >> beam.io.ReadFromText('gs://bucket/lookup.json')
lookup_dict = p | beam.Map(json.loads) | beam.combiners.ToDict()
result = events | beam.Map(enrich_event, beam.pvalue.AsDict(lookup_dict))
# Side output
small, large = (
events
| beam.Partition(lambda e, _: 0 if e['size'] < 1000 else 1, 2)
)
small | 'Small Events' >> beam.io.WriteToText('gs://bucket/small')
large | 'Large Events' >> beam.io.WriteToText('gs://bucket/large')I/O and Connectors
Beam provides IO connectors for reading and writing data across systems. TextIO reads/writes text files. ParquetIO handles Parquet format. BigQueryIO integrates with Google BigQuery. KafkaIO reads from/writes to Kafka topics. JdbcIO connects to relational databases.
Each connector handles splitting (for parallel reads), serialization, and error handling. For unbounded sources, connectors produce watermarks that feed into Beam's windowing system.
from apache_beam.io.kafka import ReadFromKafka, WriteToKafka
from apache_beam.io.parquetio import ReadFromParquet, WriteToParquet
# Kafka read
messages = p | 'Read Kafka' >> ReadFromKafka(
consumer_config={'bootstrap.servers': 'kafka:9092'},
topics=['events'],
with_metadata=True,
timestamp_policy=ReadFromKafka.create_time_policy
)
# Parquet write
messages | 'Write Parquet' >> WriteToParquet(
file_path_prefix='gs://bucket/output/events',
file_name_suffix='.parquet',
schema='event_type: string, user_id: string, timestamp: long'
)
# BigQuery write
messages | 'Write BQ' >> beam.io.WriteToBigQuery(
table='project:dataset.events',
schema=beam.io.WriteToBigQuery.SCHEMA_AUTODETECT,
write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED
)Runner Selection and Deployment
Beam runners determine where and how your pipeline executes. Google Cloud Dataflow is the reference runner with autoscaling and streaming support. Apache Flink runner provides excellent state management. Apache Spark runner leverages existing Spark clusters. DirectRunner is for local testing.
Choose Dataflow for managed infrastructure. Choose Flink for self-managed clusters with large state. Choose Spark when your team already operates Spark and you want batch-focused processing.
# DataflowRunner (managed)
python my_pipeline.py \
--runner DataflowRunner \
--project my-project \
--region us-central1 \
--temp_location gs://bucket/temp \
--streaming
# FlinkRunner (self-managed)
python my_pipeline.py \
--runner FlinkRunner \
--flink_master flink-jobmanager:8081 \
--parallelism 10 \
--max_num_workers 20 \
--streaming
# DirectRunner (local testing)
python my_pipeline.py --runner DirectRunnerFrequently Asked Questions
Is Apache Beam a framework or a library?
Beam is a programming model and SDK. It defines how to write pipelines but does not execute them — runners like Dataflow, Flink, or Spark do the execution. This separation is Beam's core value proposition.
How does Beam handle exactly-once processing?
Exactly-once depends on the runner. Dataflow provides exactly-once via windmill. Flink provides exactly-once via checkpointing. Beam's model ensures deterministic processing semantics regardless of runner.
When should I use Beam instead of Flink directly?
Use Beam when you need portability across runners or when you want a language-agnostic pipeline definition. Use Flink directly when you need advanced Flink-specific features like complex CEP or custom network protocols.
What languages does Beam support?
Python, Java, and Go SDKs are available. Java SDK is the most complete. Python SDK is popular for data science workflows. Go SDK has limited runner support.
Originally published on Ayodhyyya. Last updated June 1, 2026.