Apache Spark Tutorial: Learn Distributed Computing from Scratch (2026)
I remember the first time I watched a Spark job complete a terabyte sort in under 10 minutes on a cluster that would have taken MapReduce half an hour. That performance gap comes down to one thing: in-memory computation with DAG optimization. Spark does not force you to write rigid map-then-reduce pipelines; it builds a directed acyclic graph of transformations and optimizes the execution plan before running anything.
This tutorial covers the building blocks you will reach for daily: RDDs, DataFrames, Spark SQL for structured queries, and structured streaming for real-time pipelines. Understanding Spark's execution model will help you write efficient, debuggable distributed code.
RDDs: The Foundation
Resilient Distributed Datasets are immutable collections of objects partitioned across a cluster. Each RDD tracks its lineage — the sequence of transformations that produced it — so any partition can be recomputed if a node fails. Transformations are lazy; they build the DAG without executing. Actions like count() or collect() trigger actual computation.
In practice, you rarely use RDDs directly anymore. DataFrames provide a higher-level API with Catalyst optimizer under the hood. But understanding RDDs is essential for debugging stage boundaries and detecting shuffle points.
val rdd = sc.textFile("hdfs:///logs/2024/*.gz")
.flatMap(_.split("\\s+"))
.map(word => (word, 1))
.reduceByKey(_ + _)
rdd.saveAsTextFile("hdfs:///output/wordcount")DataFrames and the Catalyst Optimizer
DataFrames bring relational operations to Spark with automatic optimization. When you write a filter followed by an aggregation, Catalyst builds a tree of logical transformations, applies rule-based optimizations like predicate pushdown and constant folding, and then generates physical plans. It picks the cheapest plan based on cost estimates.
The Tungsten engine further accelerates execution by generating optimized Java bytecode and managing memory off-heap. The key is to let Catalyst do its job: avoid breaking the chain with inefficient UDFs that force serialization.
val df = spark.read.parquet("hdfs:///sales/")
.filter($"amount" > 1000)
.groupBy("region")
.agg(sum("amount").alias("total"))
df.explain("cost")Spark SQL and Hive Integration
Spark SQL lets you query structured data using standard SQL while leveraging Spark's distributed engine. It connects to Hive Metastore to read table schemas and partitions, making migration from Hive seamless. The thrift server provides JDBC/ODBC access so BI tools can query Spark as if it were a database.
I have migrated petabyte-scale Hive pipelines to Spark SQL by simply changing the execution engine. Use bucketing and partitioning hints to guide Spark toward better shuffle strategies.
spark.sql("""
SELECT region, SUM(amount) as total
FROM sales_view
WHERE year = 2024
GROUP BY region
ORDER BY total DESC
""").show()Structured Streaming for Real-Time Pipelines
Structured Streaming treats streams as unbounded tables. Every batch of data appended to the stream is like a new row inserted into the table. You write transformations using the same DataFrame API, and Spark incrementally executes them on new data. The key concept is the output mode: append for new rows only, update for updated keys, and complete for full result recomputation.
Checkpointing with WAL guarantees exactly-once semantics. The most common gotcha is watermark misconfiguration.
val stream = spark.readStream
.format("kafka")
.option("subscribe", "transactions")
.load()
.selectExpr("CAST(value AS STRING)")
val query = stream.writeStream
.outputMode("append")
.format("parquet")
.option("path", "/output/stream")
.option("checkpointLocation", "/checkpoints")
.start()MLlib and GraphX for Advanced Analytics
MLlib provides distributed implementations of standard machine learning algorithms: linear regression, random forests, k-means, and ALS for recommendation. The pipeline API allows chaining transformers and estimators. Cross-validation and hyperparameter tuning are built in.
GraphX extends RDDs with a graph abstraction for vertex-centric computation. The Pregel API is useful for PageRank, connected components, and shortest paths.
import org.apache.spark.ml.clustering.KMeans
val kmeans = new KMeans()
.setK(5)
.setFeaturesCol("features")
val model = kmeans.fit(df)
val silhouette = model.summary.trainingCostPerformance Tuning and Shuffle Optimization
Shuffles are the most expensive operation in Spark. They involve writing data to disk, transferring across the network, and reading on the reducer side. The number of shuffle partitions defaults to 200, which is too low for large datasets and too high for small ones. I tune it to 2-4 partitions per core.
Enable adaptive query execution (AQE) in Spark 3+ — it coalesces shuffle partitions dynamically, switches join strategies, and optimizes skew joins at runtime. AQE has been the single biggest performance improvement in recent Spark releases.
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.shuffle.partitions", "100")
df_sales.join(broadcast(df_dim), "product_id")Frequently Asked Questions
What is the difference between transformation and action in Spark?
Transformations (map, filter, join) are lazy — they build the DAG. Actions (count, collect, save) trigger execution.
When should I use RDDs instead of DataFrames?
Almost never. DataFrames offer Catalyst optimization, Tungsten execution, and a richer API. Use RDDs only when you need fine-grained control over partition data.
How does Spark handle straggler tasks?
Spark's speculative execution detects tasks running slower than the median and launches duplicate copies on other executors. Enable spark.speculation for large jobs.
What is the spark.executor.memory overhead?
It is extra memory allocated for JVM overhead, string serialization, and native buffers. Set spark.executor.memoryOverhead to 10-15% of executor memory.
Originally published on Ayodhyyya. Last updated June 1, 2026.