Apache Flume Tutorial: Learn Data Collection from Scratch (2026)
Apache Flume is a distributed service for ingesting streaming data into HDFS. I have used it to collect logs from thousands of application servers and route them into Hadoop for batch processing. Flume architecture is simple: agents with sources (ingest), channels (buffer), and sinks (deliver). The agent-based model makes it easy to set up multi-hop data flows that scale horizontally.
This tutorial covers the practical patterns for deploying Flume in production: configuring reliable channels, building fan-in topologies, using interceptors for event enrichment, and monitoring pipeline health.
Agent Architecture: Source, Channel, Sink
A Flume agent runs as a JVM process with three components. The Source receives data from a tailed file, network socket, or system log. The Channel buffers data in memory (fast but lossy) or on disk (durable). The Sink delivers to HDFS, Kafka, or the next agent.
The channel is the reliability bottleneck. I use file channels for mission-critical data and memory channels when throughput is prioritized over durability.
# Single agent config
agent1.sources = src1
agent1.channels = ch1
agent1.sinks = sink1
agent1.sources.src1.type = spooldir
agent1.sources.src1.spoolDir = /var/log/app
agent1.channels.ch1.type = file
agent1.channels.ch1.checkpointDir = /flume/checkpoint
agent1.channels.ch1.dataDirs = /flume/data
agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.hdfs.path = /flume/logs/%Y/%m/%dInterceptors for Event Enrichment
Interceptors modify or tag events between the source and the channel. The Timestamp interceptor inserts current time. Host interceptor adds the agent hostname. Custom interceptors can parse logs and extract fields like log level or service name.
Interceptors execute in order. Multiple interceptors can chain: add a timestamp, parse the log line, then route based on the parsed field.
agent1.sources.src1.interceptors = i1 i2 i3
agent1.sources.src1.interceptors.i1.type = timestamp
agent1.sources.src1.interceptors.i2.type = host
agent1.sources.src1.interceptors.i3.type = regex_extractor
agent1.sources.src1.interceptors.i3.regex = "level":"([A-Z]+)"
agent1.sources.src1.interceptors.i3.serializers = s1
agent1.sources.src1.interceptors.i3.serializers.s1.name = log_levelFan-In Topology with Tiered Agents
Tiered ingestion uses first-tier agents on app servers to forward via Avro Sink to second-tier aggregator agents. The aggregators receive from multiple upstream agents and write to HDFS. This reduces HDFS connections and centralizes configuration.
The second-tier agents use Avro Source to receive multiplexed data.
# Tier-1 agent:
app.sinks.agg1.type = avro
app.sinks.agg1.hostname = aggregator-host
app.sinks.agg1.port = 4141
# Tier-2 aggregator:
agg.sources.upstream.type = avro
agg.sources.upstream.bind = 0.0.0.0
agg.sources.upstream.port = 4141Channel Selectors and Event Routing
The Channel Selector determines which channel an event goes to. The Replicating Channel Selector sends every event to all channels. The Multiplexing Channel Selector routes events based on header values — ERROR goes to alert channel, others go to main channel.
I use multiplexing to separate critical events from routine logs.
agent1.sources.src1.selector.type = multiplexing
agent1.sources.src1.selector.header = log_level
agent1.sources.src1.selector.mapping.ERROR = error_ch
agent1.sources.src1.selector.default = main_ch
agent1.channels = error_ch warn_ch main_chHDFS Sink Configuration and File Rotation
The HDFS Sink writes events to HDFS files, rolling based on time, file size, or event count. Rolling on time (rollInterval=600) creates predictable file sizes. Rolling on size (rollSize=134217728) creates files aligned with HDFS blocks.
I set hdfs.fileType=CompressedStream with gzip to reduce storage by 70-80% on text log data.
agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.hdfs.path = /flume/logs/%Y/%m/%d/%H
agent1.sinks.sink1.hdfs.filePrefix = app_log
agent1.sinks.sink1.hdfs.rollInterval = 600
agent1.sinks.sink1.hdfs.rollSize = 134217728
agent1.sinks.sink1.hdfs.fileType = CompressedStream
agent1.sinks.sink1.hdfs.codeC = gzipMonitoring Flume Agents
Flume exposes JMX metrics for every component: source accepted bytes, channel size, sink batch duration. I monitor the channel fill percentage as the primary health metric — if a channel fills up, the source blocks and data ingestion stops upstream.
Log shipping latency is the second key metric — a growing gap indicates backpressure from sinks.
# Enable Flume monitoring:
FLUME_JAVA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445"
echo "getChannelFillPercentage(ch1)" | flume-ng monitor -agent agent1Frequently Asked Questions
What is the difference between Flume and Logstash?
Flume is purpose-built for HDFS ingestion with strong reliability via file-based channels. Logstash is more versatile with richer parsing and Elasticsearch output.
How does Flume handle backpressure?
When a channel reaches capacity, the source blocks incoming data. This pressure propagates upstream, throttling the data producers.
Can Flume process events in real-time?
Flume is designed for near-real-time ingestion with latencies of seconds to minutes. For true real-time, use Kafka as the sink.
What is the best channel type for production?
Use file channels for data that cannot be lost. Use memory channels for high-throughput pipelines where some data loss is acceptable.
Originally published on Ayodhyyya. Last updated June 1, 2026.