PySpark Tutorial: Learn Big Data Processing from Scratch (2026)
I first used PySpark to process 50GB of server logs that choked Pandas completely. Spark's approach — distribute data across a cluster and process it in memory with lazy evaluation — felt foreign at first but made sense once I understood the DataFrame API and the concept of transformations vs. actions. PySpark gives you Python access to Spark's distributed engine, letting you scale from a laptop to a hundred-node cluster without rewriting your code.
We'll process a multi-GB dataset of taxi trip records. You'll learn Spark DataFrames, transformations and actions, UDFs, window functions, and how to write efficient queries that minimize shuffle.
SparkSession and Reading Data
SparkSession is the entry point for any Spark application. It manages the SparkContext, configuration, and catalog. Data is read into a DataFrame — a distributed collection of rows with named columns, partitioned across executors. Spark supports CSV, Parquet, JSON, Avro, ORC, and direct database connections via JDBC.
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("TaxiAnalysis") \
.config("spark.sql.adaptive.enabled", "true") \
.getOrCreate()
df = spark.read \
.option("header", "true") \
.option("inferSchema", "true") \
.csv("taxi_trips/*.csv")
print(f"Partitions: {df.rdd.getNumPartitions()}")
df.show(5)
df.printSchema()Transformations, Actions, and Lazy Evaluation
Spark builds a DAG of transformations — filter, select, groupBy — but nothing executes until an action (show, count, write) is called. This lazy evaluation lets Spark optimize the entire pipeline, reordering operations and pushing filters down to the data source.
# Transformations (lazy)
filtered = df.filter(df['fare_amount'] > 0) \
.select('pickup_datetime', 'fare_amount', 'trip_distance') \
.withColumn('fare_per_mile', df['fare_amount'] / df['trip_distance'])
# Action (triggers computation)
result = filtered.count()
print(f"Valid trips: {result}")
filtered.cache()
filtered.write.mode('overwrite').parquet('clean_trips/')GroupBy and Aggregation
Spark's groupBy and agg functions mirror SQL's GROUP BY. Use agg() with multiple aggregation functions from pyspark.sql.functions — sum, avg, count, min, max, stddev. These operations trigger a shuffle (data movement between partitions), which is the most expensive part of a Spark job.
from pyspark.sql import functions as F
daily_stats = df \
.filter(df['fare_amount'] > 0) \
.groupBy(F.to_date('pickup_datetime').alias('date')) \
.agg(
F.count('*').alias('trip_count'),
F.sum('fare_amount').alias('total_fare'),
F.avg('trip_distance').alias('avg_distance'),
F.max('passenger_count').alias('max_passengers')
) \
.orderBy('date')
daily_stats.show(10)Window Functions for Rank and Running Totals
Window functions operate on a group of rows (the window) without collapsing them. You partition by one column, order within the partition, and apply functions like row_number(), rank(), lag(), lead(), or sum(). I use window functions for deduplication and computing running totals.
from pyspark.sql.window import Window
window_spec = Window \
.partitionBy(F.month('pickup_datetime').alias('month')) \
.orderBy(F.sum('fare_amount').desc())
vendor_ranking = df \
.groupBy(F.month('pickup_datetime').alias('month'), 'vendor_id') \
.agg(F.sum('fare_amount').alias('total_fare')) \
.withColumn('rank', F.row_number().over(window_spec)) \
.filter(F.col('rank') <= 3)
vendor_ranking.show()
running_window = Window \
.partitionBy(F.to_date('pickup_datetime')) \
.orderBy('pickup_datetime') \
.rowsBetween(Window.unboundedPreceding, Window.currentRow)
df_with_running = df.withColumn('running_total_fare', F.sum('fare_amount').over(running_window))User-Defined Functions (UDFs) and Pandas UDFs
When built-in functions aren't enough, define a UDF. Regular Python UDFs serialize each row to Python, process it, and return — this is slow. Pandas UDFs (vectorized UDFs) operate on batches of rows using Apache Arrow for zero-copy data transfer, offering 10-100x speedup.
from pyspark.sql.functions import pandas_udf
from pyspark.sql.types import DoubleType
import pandas as pd
@pandas_udf(returnType=DoubleType())
def fare_category(fare_series: pd.Series) -> pd.Series:
return pd.cut(
fare_series,
bins=[0, 10, 25, 50, float('inf')],
labels=[1.0, 2.0, 3.0, 4.0]
)
df_with_category = df.withColumn('fare_category', fare_category(df['fare_amount']))
df_with_category.groupBy('fare_category').count().show()Writing Efficient Spark Jobs and Optimization Tips
Performance in Spark is about minimizing shuffle. Partition your data appropriately, use broadcast joins for small tables, and avoid wide dependencies when possible. Enable adaptive query execution for automatic coalescing and join strategy selection.
optimized_df = df.repartition(100)
df.write.bucketBy(50, 'vendor_id').sortBy('pickup_datetime').saveAsTable('trips_bucketed')
from pyspark.sql.functions import broadcast
lookup = spark.read.parquet('vendor_lookup/')
result = df.join(broadcast(lookup), 'vendor_id')Frequently Asked Questions
How does PySpark compare to Pandas for large datasets?
Pandas holds data in memory on a single machine. PySpark distributes data across a cluster. For datasets under 10GB, Pandas is simpler and faster. Beyond that, PySpark scales.
Do I need a Hadoop cluster to run PySpark?
No. PySpark runs locally with local[*] as the master URL. For production, deploy on a cluster (EMR, Databricks, or your own Spark cluster with YARN or Kubernetes).
What is the difference between DataFrame and RDD?
DataFrames have a schema, are optimized with Catalyst, and use Tungsten for efficient memory management. RDDs are lower-level, untyped, and slower. Always use DataFrames unless you need low-level control.
How do I avoid 'Out of Memory' errors in Spark?
Increase the number of partitions, use Kryo serialization, reduce shuffle data by filtering early, and tune executor memory (spark.executor.memory).
Originally published on Ayodhyyya. Last updated June 1, 2026.