Home › Categories › big-data › Apache NiFi Tutorial: Data Flow Automation and I…
big-data5 min read

Apache NiFi Tutorial: Data Flow Automation and Integration (2026)

Sandip Mhaske • Ayodhyya • big-data

Apache NiFi Tutorial: Data Flow Automation and Integration (2026)

Published:  |  Category: Big Data  |  Reading time: ~15 min
Apache NiFi Tutorial: Data Flow Automation and Integration (2026)

Apache NiFi provides a visual, flow-based programming model for automating data movement between systems. After deploying NiFi for real-time data ingestion pipelines processing millions of events per hour, I appreciate how its back pressure, prioritization, and provenance tracking eliminate the fragility of custom scripts and cron jobs.

This tutorial covers NiFi's processor model, flow design, content and flow file concepts, scheduling, provenance, and production deployment for reliable data flow automation.

📖 Table of Contents
  1. NiFi Architecture and Core Concepts
  2. Processors and Controller Services
  3. Scheduling and Flow Control
  4. Provenance and Data Lineage
  5. Clustering and High Availability
  6. Production Deployment and Monitoring

NiFi Architecture and Core Concepts

NiFi operates as a single-node or clustered application. Each node runs a Java process with a web UI, flow controller, repository, and processor instances. The flow controller manages processor scheduling, flow file routing, and resource allocation. FlowFiles are the unit of data transfer — they carry attributes (metadata) and content (the actual data).

Processors are the building blocks. Each processor has input ports, output ports, and relationships. When a FlowFile is processed, it is routed to a relationship (success, failure, retry). The flow controller handles routing based on the relationship configuration.

# NiFi flow configuration — processor chain

# 1. GetFile — reads files from a directory
Processor: GetFile
Properties:
  Input Directory: /data/incoming
  Batch Size: 100
  Poll Interval: 5 sec
  File Filter: .*\.csv$

# 2. UpdateAttribute — tag flow files
Processor: UpdateAttribute
Properties:
  pipeline.name: daily_etl
  processing.tier: production

# 3. ConvertRecord — CSV to JSON
Processor: ConvertRecord
Properties:
  Record Reader: CSVReader
  Record Writer: JsonRecordSetWriter
  Schema Access: Embedded Schema

Processors and Controller Services

Processors implement data movement and transformation logic. NiFi provides 300+ processors for reading, writing, routing, and transforming data. Controller services provide shared resources: database connections, SSL contexts, record readers/writers, and distributed cache access.

Record-based processors (ConvertRecord, MergeContent, SplitContent) operate on structured data without writing custom code. Define a schema once, and all record processors use it consistently.

# Controller Services — configure in NiFi UI

# Database Connection Pool
Controller Service: DBCPConnectionPool
  Database Driver Path: /opt/nifi/lib/postgresql-42.2.0.jar
  Database Driver Class Name: org.postgresql.Driver
  Database URL: jdbc:postgresql://db:5432/analytics
  Username: nifi_user
  Max Total Connections: 20

# Avro Reader
Controller Service: AvroReader
  Schema Access Strategy: Embedded Schema

# CSV Reader
Controller Service: CSVReader
  Schema Access Strategy: Header-Defined
  Use First Line as Header: true
  Date Format: yyyy-MM-dd HH:mm:ss

# Record Writer
Controller Service: JsonRecordSetWriter
  Pretty Print: true
  Schema Write Strategy: No Output

Scheduling and Flow Control

Processors run on schedules: timer-driven (fixed interval) or event-driven (on trigger). The flow controller manages scheduling with configurable concurrent tasks. Back pressure prevents downstream processors from being overwhelmed by limiting queued FlowFiles and total queue size.

Prioritizers control FlowFile processing order: FIFO, priority by attribute, last modified, or custom logic. This ensures critical data is processed first when queues back up.

# Processor scheduling configuration

# Timer-driven (every 5 seconds)
Scheduling Strategy: Timer Driven
Run Duration: 5 sec
Concurrent Tasks: 4

# Event-driven (triggered by incoming FlowFiles)
Scheduling Strategy: Event Driven

# Back pressure configuration
# Queue: Success
  Back Pressure Object Threshold: 10000
  Back Pressure Size Threshold: 1 GB
  Drop All: false (route to failure)

# Prioritizers
  Priority 1: FirstInFirstOutQueueComparator
  # Or: PriorityAttributeQueueComparator (uses 'priority' attribute)
  # Or: LatestMostRecentLastQueueComparator

Provenance and Data Lineage

NiFi tracks the complete lineage of every FlowFile through its Provenance Repository. Every event — fork, clone, merge, content-modify, route, receive, send — is recorded with timestamps, relationships, and attributes. This provides complete audit trails for compliance and debugging.

Provenance events support event-time queries, lineage graphs, and impact analysis. For high-throughput flows, configure the provenance repository for memory-based or write-ahead logging to handle millions of events per second.

# Provenance Repository configuration
# nifi.properties
nifi.provenance.repository.directory.default=/data/provenance
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=8 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB

# Query provenance via REST API
curl 'http://localhost:8080/nifi-api/provenance?searchTerm=event_type=CONTENT_MODIFIED&maxResults=10'

# Query by FlowFile UUID
curl 'http://localhost:8080/nifi-api/provenance?searchTerm=flowFileUuid=abc-123&maxResults=10'

# NiFi CLI provenance query
$ nifi provenance query --searchTerm "eventType = ROUTE" --maxResults 50

Clustering and High Availability

NiFi clusters use a primary/secondary model for flow configuration. The primary node manages the DGM (cluster coordinator) and flow configuration. Secondary nodes receive flow configuration from the primary and execute processors. All nodes process data in parallel.

Cluster membership is managed via ZooKeeper. Nodes register themselves and report health. If a node fails, its FlowFiles are not lost — the provenance repository and content repository persist to disk. When the node recovers, it resumes processing.

# Cluster configuration
# nifi.properties
nifi.cluster.is.node=true
nifi.cluster.node.address=node1:8080
nifi.cluster.node.protocol.port=8389
nifi.cluster.load.balance.port=6367
nifi.cluster.load.balance.host=node1

# ZooKeeper for cluster coordination
nifi.zookeeper.connect.string=zk1:2181,zk2:2181,zk3:2181
nifi.zookeeper.root.node=/nifi
nifi.zookeeper.auth.type=default

# Site-to-Site for multi-cluster communication
# Remote Process Group: site-to-site
# Transport: Input Port on receiving cluster
# Input Port: cluster-receive
# Target: primary-node:8080

Production Deployment and Monitoring

Deploy NiFi with sufficient heap (8-16 GB) and content repository storage (SSD for performance). Monitor queue depths, processor throughput, JVM metrics, and repository usage. Set alerts for queue back pressure, processor failures, and repository disk space.

Use NiFi Reporting Tasks to emit metrics to Prometheus, Ambari, or custom monitoring systems. Flow status includes active threads, FlowFiles processed, bytes read/written, and processing time per processor.

# NiFi JVM configuration
# bootstrap.conf
java.arg.1=-Xms8g
java.arg.2=-Xmx16g
java.arg.3=-XX:+UseG1GC
java.arg.4=-XX:MaxGCPauseMillis=200
java.arg.5=-Dcom.sun.management.jmxremote
java.arg.6=-Dcom.sun.management.jmxremote.port=9010

# Content repository on SSD
nifi.content.repository.directory.default=/ssd/nifi/content
nifi.content.repository.archive.max.space=50 GB

# Prometheus reporting
nifi.reporting.task.prometheus.host=0.0.0.0
nifi.reporting.task.prometheus.port=9092

# Key metrics to monitor
# - Processor: BytesRead, BytesWritten, FlowFilesReceived
# - JVM: HeapUsed, GCCount, GCTime
# - Repository: ContentRepoSize, ProvenanceRepoSize
# - Cluster: ActiveNodes, HeartbeatsLost

Frequently Asked Questions

When should I use NiFi vs. Kafka for data movement?

NiFi excels at data flow with transformations, routing, and protocol conversion. Kafka excels at high-throughput event streaming between decoupled consumers. Use NiFi for ingestion and transformation, Kafka for distribution and buffering.

How does NiFi handle back pressure?

NiFi queues have configurable thresholds for FlowFile count and total size. When a queue reaches its threshold, upstream processors stop producing to it, creating back pressure that propagates through the flow.

Can NiFi process data in real-time?

NiFi processes data in near-real-time with sub-second latency for event-driven processors. For true real-time streaming, combine NiFi with Kafka — NiFi ingests and transforms, Kafka provides the streaming backbone.

How do I version-control NiFi flows?

Use NiFi Registry to version flows. Flows are stored in a Git-backed registry. Deploy flows from the registry, and version control is automatic. You can roll back to previous versions via the UI.

Originally published on Ayodhyyya. Last updated June 1, 2026.