Apache Storm Tutorial: Learn Stream Processing from Scratch (2026)
Apache Storm was the first real-time stream processing system I used in production. Storm processed events with latencies under a second, using a topology of spouts (data sources) and bolts (processing steps) connected by stream groupings. Storm claim to fame was guaranteed message processing — every tuple is replayed until fully processed.
This tutorial covers building Storm topologies: spout and bolt design, stream groupings for data distribution, reliability guarantees, and the Trident API for micro-batching.
Topology Design: Spouts and Bolts
A Storm topology is a directed graph of spouts and bolts. Spouts read from external sources — Kafka, Kestrel, or custom generators — and emit tuples. Bolts process tuples, performing transformations, aggregations, or interactions with external systems. Each node runs in parallel across multiple executors.
A well-tuned topology processes millions of tuples per second.
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("sentences", new SentenceSpout(), 4);
builder.setBolt("split", new SplitBolt(), 8).shuffleGrouping("sentences");
builder.setBolt("count", new CountBolt(), 4).fieldsGrouping("split", new Fields("word"));
Config conf = new Config();
conf.setNumWorkers(3);
StormSubmitter.submitTopology("wordcount", conf, builder.createTopology());Stream Groupings: How Tuples Flow
Stream groupings define how tuples are distributed. Shuffle grouping randomly distributes for load balancing. Fields grouping hashes on field values — all tuples with the same value go to the same executor. All grouping broadcasts to every bolt instance.
Fields grouping on the join key enables efficient bolt-level joins.
builder.setBolt("split", new SplitBolt(), 8).shuffleGrouping("sentences");
builder.setBolt("count", new CountBolt(), 4).fieldsGrouping("split", new Fields("word"));
builder.setBolt("printer", new PrinterBolt(), 2).globalGrouping("count");Reliability and Acker Mechanism
Storm guarantees tuple processing through the acker system. Every tuple tree is tracked via XOR checksum. When the checksum reaches zero, the spout ack() is called. If not acknowledged within the timeout, the spout fail() is called and the tuple is replayed.
The timeout is configurable per topology. I set it to 2-3x the observed tuple latency during peak load.
public class ReliableSplitBolt extends BaseRichBolt {
public void execute(Tuple input) {
String sentence = input.getString(0);
for (String word : sentence.split(" ")) {
collector.emit(input, new Values(word));
}
collector.ack(input);
}
}
Config conf = new Config();
conf.setNumAckers(4);Trident: Micro-Batching on Storm
Trident processes tuples in micro-batches, providing exactly-once semantics through transactional spouts and state persistence. Trident operations mirror Spark Streaming: filter, project, groupBy, aggregate, and stateQuery. The trade-off is latency — Trident processes batches every few seconds.
I use Trident for use cases requiring exactly-once without custom deduplication logic.
TridentTopology trident = new TridentTopology();
trident.newStream("sentences", new FixedBatchSpout(5))
.each(new Fields("sentence"), new SplitFunction(), new Fields("word"))
.groupBy(new Fields("word"))
.persistentAggregate(CassandraState.nonTransactional(new CassandraStateFactory()), new Count(), new Fields("count"));KafkaSpout Integration
The KafkaSpout brings Kafka data into Storm topologies. It consumes from Kafka partitions and emits tuples to the topology. The spout tracks offset positions, providing at-least-once delivery. When a tuple fails, the spout replays from the last committed offset.
The spout parallelism should match the Kafka partition count for optimal consumption.
BrokerHosts hosts = new ZkHosts("zk1:2181,zk2:2181");
SpoutConfig spoutConfig = new SpoutConfig(hosts, "transactions", "/storm-kafka", "txn-id");
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("kafka-spout", new KafkaSpout(spoutConfig), 6);
builder.setBolt("transform", new TransformBolt(), 8).shuffleGrouping("kafka-spout");Monitoring and Debugging Topologies
Storm UI provides per-topology metrics: latency, throughput, capacity, and error rates. The capacity metric (executor busy time) is most important — if a bolt capacity approaches 1.0, increase parallelism or optimize the bolt logic.
Logviewer aggregates worker logs for debugging. Drpc is useful for querying real-time results from external applications.
# CLI monitoring:
storm list
storm kill wordcount
# storm.yaml:
storm.messaging.transport: "org.apache.storm.messaging.netty.Context"
storm.messaging.netty.buffer_size: 5242880Frequently Asked Questions
What is the difference between Storm and Spark Streaming?
Storm processes each tuple as it arrives (true streaming). Spark Streaming processes micro-batches. Storm offers lower latency (sub-second).
How does Storm guarantee message processing?
Through the acker mechanism. Each tuple tree is tracked; the spout is notified when fully processed or partially failed. Failed tuples are replayed.
What is the role of ZooKeeper in Storm?
ZooKeeper coordinates cluster state: supervisor heartbeats, topology assignments, and Kafka offset tracking. Storm cannot function without ZooKeeper.
Can Storm run without Nimbus?
Nimbus is required for topology submission but is stateless. If Nimbus fails, running topologies continue working.
Originally published on Ayodhyyya. Last updated June 1, 2026.