Apache Pig Tutorial: Learn Data Flow from Scratch (2026)
Apache Pig was my gateway drug to Hadoop programming. Before Pig, writing MapReduce meant hundreds of lines of Java for even trivial transformations. Pig Latin — a dataflow scripting language — reduced the same logic to a dozen lines. Pig compiled scripts into MapReduce or Tez jobs, letting you express multi-step transformations without worrying about the execution engine.
This tutorial covers the practical side of Pig Latin: loading varied formats, transforming data with FOREACH and filters, joining datasets, and using UDFs.
Pig Latin Basics: Load and Store
Pig Latin is a procedural language that describes how data flows through transformations. Every script starts with LOAD, which reads data and assigns a schema. PigStorage is the default loader for delimited text; AvroStorage and ParquetLoader handle columnar formats. The STORE command writes output.
DUMP prints output for debugging. DESCRIBE shows the schema. EXPLAIN shows the execution plan.
sales = LOAD 'hdfs:///data/sales.csv' USING PigStorage(',')
AS (order_id:int, customer_id:chararray, amount:float, order_date:chararray);
DESCRIBE sales;
region_sales = GROUP sales BY region;
STORE region_sales INTO 'hdfs:///output/sales_summary' USING PigStorage(',');FOREACH, FILTER, and Data Transformation
FOREACH applies row-level transformations: projection, arithmetic, string manipulation, and nested operations. FILTER removes rows that don't match a condition. You can nest FOREACH inside GROUP operations for per-group processing.
TOBAG, TOTUPLE, and TOMAP construct complex data types. FLATTEN un-nests tuples and bags into multiple rows.
cleaned = FILTER sales BY amount > 0 AND amount IS NOT NULL;
enriched = FOREACH cleaned GENERATE
order_id,
ROUND(amount * 1.1) AS amount_with_tax;
region_grp = GROUP cleaned BY region;
top5 = FOREACH region_grp {
sorted = ORDER cleaned BY amount DESC;
top5 = LIMIT sorted 5;
GENERATE group AS region, top5;
}Joins: Equi, Replicated, and Skewed
Pig supports equi joins, replicated joins (broadcast small table to all mappers), and skewed joins (handle hot keys separately). Replicated joins avoid a shuffle when one table fits in memory. Skewed joins sample data to identify hot keys and prevent reducer bottlenecks.
I use replicated joins for dimension tables under 10 million rows. Skewed joins are essential for clickstream data with heavy tails.
customers = LOAD 'hdfs:///data/customers' AS (customer_id:int, name:chararray, segment:chararray);
-- Replicated join:
enriched = JOIN sales BY customer_id, customers BY customer_id USING 'replicated';
-- Skewed join:
skew_joined = JOIN clicks BY page_id, page_dim BY page_id USING 'skewed';GROUP, COGROUP, and Nested Queries
GROUP collates rows with the same key into a bag. COGROUP groups rows from multiple relations by key without joining, producing nested bags useful for difference and intersection queries. FLATTEN on a COGROUP output produces a FULL OUTER JOIN.
I use COGROUP for comparing datasets by key alignment.
cogrouped = COGROUP sales BY customer_id, customers BY customer_id;
-- Find customers with no purchases:
no_purchases = FILTER cogrouped BY IsEmpty(sales);
-- Full outer join:
outer = FOREACH cogrouped GENERATE FLATTEN(sales), FLATTEN(customers);User Defined Functions in Pig
Pig UDFs extend the built-in function set. REGISTER JARs, then reference functions by class name. EVAL functions process rows; FILTER functions return boolean; ALGEBRAIC functions optimize aggregation by implementing partial aggregation in the map phase.
The Algebraic interface is most important for performance — a SUM UDF implementing Algebraic runs in a combiner.
REGISTER 'hdfs:///udfs/pig-math.jar';
DEFINE geo_distance com.example.GeoDistance();
with_distance = FOREACH shipment GENERATE
order_id,
geo_distance(origin_lat, origin_lon, dest_lat, dest_lon) AS distance_km;Optimization: Parallelism and Combiner Usage
Pig default parallelism is determined by input file size. Set PARALLEL to control reducer count. The OPTIMIZER setting enables rules: MergeFilter, PushUpFilter, and ColumnMapKeyPrune are the most impactful.
Algebraic UDFs invoke the combiner automatically. Use LIMIT during development for fast iteration on a sample.
SET default_parallel 50;
SET opt.multiquery true;
SET pig.exec.mapPartAggr true;
sample = LIMIT sales 1000;
result = GROUP sample BY region;
ILLUSTRATE result;Frequently Asked Questions
What is the difference between Pig and Hive?
Pig uses a procedural dataflow language (Pig Latin) for step-by-step transformations. Hive uses SQL. Pig is better for ETL pipelines with complex multi-step logic.
Is Apache Pig still actively used?
Usage has declined as Spark replaced it in most new pipelines. Many legacy Hadoop deployments still run Pig jobs.
How does Pig handle schema evolution?
Pig uses schema-on-read; the schema is defined at LOAD time. NULLs are inserted when fields cannot be parsed.
Can Pig process Avro or Parquet files?
Yes. Pig has built-in support for AvroStorage and ParquetLoader that preserve the schema embedded in the file.
Originally published on Ayodhyyya. Last updated June 1, 2026.