Data Engineering Tutorial: Learn Data Pipeline from Scratch (2026)
Data engineering is the unsung foundation of data science. Before any dashboard, model, or insight exists, someone must build the pipeline that extracts, cleans, transforms, and loads the data. After architecting pipelines processing terabytes daily for e-commerce and fintech companies, I have learned that robustness and observability matter more than clever algorithms.
This tutorial covers the full lifecycle: ingestion from multiple sources, batch and streaming processing, data lake and warehouse storage, orchestration, and monitoring. You will build a complete pipeline using Spark, Kafka, Airflow, and dbt — the modern data stack.
Batch vs. Streaming Ingestion
Batch ingestion moves data on a schedule (hourly, daily) and is simpler, cheaper, and easier to audit. Streaming ingestion moves data continuously with sub-second latency and is essential for real-time use cases (fraud detection, monitoring). Most organizations start with batch and add streaming only when the latency requirement demands it.
The tool choice depends on volume. For small volumes (GB/day), simple scripts with Airflow scheduling suffice. For large volumes (TB/day), use Spark for batch and Kafka + Flink for streaming. The key principle: make ingestion idempotent — re-running the same job should produce the same result.
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('data_ingestion').getOrCreate()
df = spark.read \
.option('header', 'true') \
.option('inferSchema', 'true') \
.csv('/raw/orders/2026/07/*.csv')
df.write \
.mode('append') \
.parquet('/lake/orders/')Data Lakes and the Lakehouse
A data lake stores raw data in its native format on object storage like S3 or ADLS. A data warehouse stores curated, structured data optimized for analytics (Snowflake, BigQuery, Redshift). The lakehouse architecture merges both: ACID transactions and schema enforcement on top of cheap object storage, enabled by Apache Iceberg, Delta Lake, or Hudi.
Iceberg provides time travel, schema evolution, and partition evolution without rewriting data. This makes it the de facto standard for new data lake projects as of 2026.
CREATE TABLE lake.orders (
order_id BIGINT,
customer_id STRING,
amount DECIMAL(10,2),
order_ts TIMESTAMP
)
USING iceberg
PARTITIONED BY (days(order_ts))
LOCATION 's3://data-lake/orders/'
SELECT * FROM lake.orders FOR SYSTEM_TIME AS OF '2026-07-01 00:00:00'Transformations with dbt
dbt (data build tool) lets you write transformations as SQL SELECT statements, and it handles dependency resolution, incremental builds, testing, and documentation. Each model is a SQL file that dbt materializes as a table or view. Tests are written as YAML assertions: uniqueness, not-null, referential integrity, and custom SQL checks.
The mental model is software engineering for data: version-controlled SQL, CI/CD, code review, and modular design. dbt compiles your models into a DAG and executes them in dependency order.
SELECT
order_id,
customer_id,
CAST(amount AS DECIMAL(10,2)) AS amount,
TO_TIMESTAMP(order_date) AS order_ts
FROM {{ source('raw', 'orders') }}
WHERE order_id IS NOT NULL
-- schema.yml
version: 2
models:
- name: stg_orders
columns:
- name: order_id
tests: [unique, not_null]Orchestration with Airflow
Airflow schedules and monitors workflows as DAGs (Directed Acyclic Graphs). Each node is a task, and edges define dependencies. Tasks run on workers, and the scheduler determines which tasks are ready to run. The web UI shows DAG runs, task logs, and failure alerts.
Best practices: use DockerOperator or KubernetesPodOperator for task isolation, set retries with exponential backoff, and define SLAs that trigger alerts when a DAG runs longer than expected. Avoid putting business logic in DAG files — treat them as configuration.
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'data-team',
'retries': 2,
'retry_delay': timedelta(minutes=5),
}
with DAG(
'orders_pipeline',
start_date=datetime(2026, 1, 1),
schedule='@daily',
catchup=False,
default_args=default_args
) as dag:
ingest = PythonOperator(task_id='ingest_orders', python_callable=run_ingestion)
transform = PythonOperator(task_id='transform_orders', python_callable=run_dbt)
quality = PythonOperator(task_id='data_quality_checks', python_callable=run_checks)
ingest >> transform >> qualityData Quality and Observability
Garbage in, garbage out. Data quality checks catch schema changes, null spikes, drifted distributions, and broken referential integrity. Great Expectations is the leading open-source library: you define suites of expectations and run them as data lands. Failed expectations can alert, block downstream pipelines, or quarantine bad data.
Observability goes beyond quality to cover freshness, volume, and lineage. Monte Carlo and Sifflet provide managed observability; OpenLineage provides open-source lineage tracking.
import great_expectations as gx
context = gx.get_context()
datasource = context.sources.pandas_default
batch = datasource.add_dataframe_asset('orders').build_batch(dataframe=df)
expectations = [
gx.expectations.ExpectColumnValuesToBeBetween(
column='amount', min_value=0, max_value=100000
),
gx.expectations.ExpectColumnValuesToNotBeNull(column='order_id'),
]
results = context.run_validation(expectations, batch)The Modern Data Stack
The modern data stack (MDS) as of 2026 typically includes: object storage (S3), a table format (Iceberg), a compute engine (Spark or Trino), a transformation tool (dbt), an orchestrator (Airflow or Dagster), a catalog (Unity Catalog or Apache Polaris), and a BI layer (Metabase or Preset). For streaming, add Kafka + Flink.
The most important advice: start with the minimum viable stack. A single PostgreSQL instance with well-written SQL views can serve a team of 10 analysts for months. Add complexity only when you genuinely outgrow the current solution.
services:
postgres:
image: postgres:16
environment:
POSTGRES_DB: analytics
superset:
image: apache/superset:latest
ports:
- '8088:8088'
dbt:
image: ghcr.io/dbt-labs/dbt-postgres:1.8
volumes:
- ./dbt:/usr/app
working_dir: /usr/appFrequently Asked Questions
What is the difference between a data engineer and a data scientist?
A data engineer builds and maintains the infrastructure that collects, stores, and processes data. A data scientist analyzes that data to generate insights and models. Engineers ensure data is available, reliable, and performant; scientists ask interesting questions of that data.
Do I need to know distributed systems to be a data engineer?
Not at entry level, but it becomes essential as you scale. Understanding consistency models, partitioning, replication, and the CAP theorem helps you diagnose why your Spark job failed. Start with Python/SQL, then add distributed computing concepts.
Should I use Spark or DuckDB for data processing?
Spark is designed for terabytes to petabytes across many machines. DuckDB is an embedded OLAP database that runs on a single machine and often outperforms Spark on datasets under 100 GB. Use DuckDB for small-to-medium workloads, Spark when you outgrow a single machine.
How do I handle PII in data pipelines?
Tokenize or encrypt PII at ingestion time. Store the mapping in a separate, access-controlled vault. In the data lake, use column-level access controls and never log raw PII. For analytics, aggregate early — you rarely need individual-level data for dashboards.
Originally published on Ayodhyyya. Last updated June 1, 2026.