Hadoop Tutorial: Learn Big Data from Scratch (2026)
I've spent the better part of a decade working with Hadoop clusters in production, and the one thing I can tell you is this: Hadoop is not dead. Despite the rise of cloud-native alternatives, Hadoop's architectural lessons around data locality, fault tolerance, and rack-aware storage remain foundational to distributed computing. Whether you run an on-prem cluster or deploy to a cloud EMR environment, understanding Hadoop's internals will make you a better engineer.
This tutorial walks through the core subsystems that make Hadoop tick: HDFS for storage, MapReduce for computation, and YARN for resource management. By the end, you'll understand how data flows through a cluster, how failures are handled, and why certain design decisions were made.
HDFS Architecture and Data Blocks
HDFS splits files into blocks (default 128 MB) and distributes them across DataNodes. The NameNode keeps the entire filesystem metadata in memory — this is both its strength and its single-point-of-failure risk. Each block is replicated (default factor 3) across different racks to survive node and rack failures. When you write a file, the NameNode assigns a pipeline of DataNodes; data flows through the pipeline in a chain of replicas rather than being written in parallel.
The choice of 128 MB blocks is not arbitrary. Larger blocks minimize seek overhead relative to transfer time, which is critical when streaming through terabytes of data. If you are working with many small files, HDFS will struggle because the NameNode memory is consumed by metadata, not data. This is why HDFS is designed for bulk throughput, not low-latency access.
hdfs dfs -mkdir /user/tutorial
echo "Hello Hadoop" | hdfs dfs -put - /user/tutorial/sample.txt
hdfs dfs -ls /user/tutorial
hdfs fsck /user/tutorial/sample.txt -files -blocks -locationsMapReduce: Data Processing at Scale
MapReduce forces you to think in two phases: the map phase reads input splits and emits key-value pairs; the reduce phase groups those pairs by key and aggregates them. The shuffle — the data transfer between mappers and reducers — is where most performance issues surface. Partitioner logic determines which keys go to which reducer, and an uneven distribution causes a straggler problem where one reducer lags behind all others.
A common mistake is emitting too many keys from the mapper, forcing an expensive shuffle. Combiners act as mini-reducers on the mapper side, cutting down data transfer. In production, I have seen jobs speed up 3x simply by adding a well-placed combiner.
public class WordCount {
public static class TokenizerMapper extends Mapper {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context) throws IOException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
} YARN Resource Management
YARN decouples resource management from the processing framework. The ResourceManager runs on the master and allocates containers — CPU and memory bundles — across NodeManagers. ApplicationMasters negotiate containers for individual jobs. This separation is what allowed Spark, Flink, and Tez to run on the same cluster that previously only ran MapReduce.
Memory tuning in YARN is a balancing act. Container size, heap overhead, and the ratio between physical and virtual memory all need calibration. A misconfigured cluster will either underutilize resources or kill containers aggressively. I typically start with a 1:2 vCore-to-memory ratio and adjust from observed utilization.
yarn.scheduler.minimum-allocation-mb
1024
yarn.scheduler.maximum-allocation-mb
8192
yarn.nodemanager.resource.cpu-vcores
8
Data Replication and Rack Awareness
HDFS replica placement policy is what makes it resilient. The first replica goes on the node writing the data. The second goes on a node in a different rack. The third goes on a different node in the same rack as the second. This ensures that a single rack failure does not wipe out all replicas. The rack awareness script maps IP addresses to network topologies so HDFS knows where nodes are physically located.
When a DataNode fails, the NameNode detects it through heartbeat loss and immediately schedules replication of the blocks that were under-replicated. The replication priority queue ensures critical blocks with only one replica are copied before those with two.
topology.script.file.name
/etc/hadoop/conf/rack-awareness.sh
dfs.replication
3
NameNode High Availability
In Hadoop 1.x, the NameNode was a single point of failure. Hadoop 2.x introduced NameNode HA using a shared edits directory on NFS or Quorum Journal Manager (QJM). The Active NameNode writes edits to journal nodes, and the Standby NameNode reads them to keep its state current. A ZooKeeper-based failover controller detects Active failure and promotes the Standby.
QJM is strongly preferred over NFS in production. It requires an odd number of journal nodes (typically 3 or 5) and uses a quorum-based write protocol.
dfs.nameservices
mycluster
dfs.ha.namenodes.mycluster
nn1,nn2
dfs.namenode.shared.edits.dir
qjournal://node1:8485;node2:8485;node3:8485/mycluster
Cluster Provisioning and Capacity Planning
Building a Hadoop cluster requires thinking about the ratio of storage to compute. A data-heavy workload needs more spindles; a compute-heavy workload needs more CPU. A balanced node typically has 12-24 disks at 4-8 TB each, 256-512 GB RAM, and 16-32 vCores.
For cloud deployments, ephemeral EMR clusters are popular because you pay only for the runtime. The key is to right-size your core and task node instance families.
## Rough formula:
## Total storage = (Raw disk x Nodes x RF) / Compression ratio
## Example: 10 nodes x 12 x 4TB x 3x / 1.5 = 960 TB usable
hdfs balancer -threshold 10Frequently Asked Questions
What is the ideal HDFS block size?
128 MB is the default and works well for most workloads. Larger blocks (256 MB) help with massive sequential reads.
How does Hadoop handle node failure?
The NameNode detects DataNode failure via heartbeat loss within 10 seconds. Blocks on the dead node become under-replicated, and the NameNode schedules copies from remaining replicas.
Can I run Hadoop without YARN?
Yes, in standalone mode MapReduce uses local filesystem and JobTracker runs in-process. This is useful for testing but not for production.
What is the difference between HDFS and S3?
HDFS provides strong consistency and data locality for compute, while S3 is object storage with eventual consistency on some operations. HDFS is ideal for on-prem clusters.
Originally published on Ayodhyyya. Last updated June 1, 2026.