Azure Data Factory Tutorial: ETL and Data Integration from Scratch (2026)

Published: October 22, 2025 | Category: Microsoft | Reading time: ~15 min

When our data warehouse needed to ingest data from twenty different sources — SQL Server, Salesforce, FTP, REST APIs — I turned to Azure Data Factory. It transformed a fragile collection of Python scripts into a robust, monitored, and manageable ETL pipeline. ADF has been central to every enterprise data project I have delivered since.

Azure Data Factory is Microsoft's cloud-based ETL and data integration service. It creates, schedules, and orchestrates data pipelines that move and transform data across 100+ built-in connectors. With a visual designer, code-free transformations (Data Flows), and programmatic control via SDKs, it handles batch and real-time data integration at scale.

📖 Table of Contents

Pipelines, Activities, and Linked Services
Copy Data Activity and Connectors
Mapping Data Flows: Code-Free Transformations
Control Flow: Conditions, Loops, and Variables
Integration Runtimes and Connectivity
Monitoring, Alerts, and CI/CD

Pipelines, Activities, and Linked Services

A pipeline is a logical grouping of activities. Activities define work — copy data, transform, execute Azure functions. Linked services connect to data sources. Datasets represent data structures. Trigger-based execution schedules pipelines.

{ "name":"IngestSales","properties":{
  "activities":[{
    "name":"CopySQLToBlob","type":"Copy",
    "inputs":[{"referenceName":"SalesDataset"}],
    "outputs":[{"referenceName":"BlobSalesOutput"}],
    "typeProperties":{
      "source":{"type":"SqlSource","sqlReaderQuery":"SELECT * FROM Orders WHERE OrderDate >= @delta"},
      "sink":{"type":"BlobSink"}
    }
  }]
}}

Copy Data Activity and Connectors

Copy Data is the most common activity — it moves data between 100+ sources and sinks. Connectors span Azure (Blob, SQL, ADLS), databases (Oracle, MySQL, SAP), file formats (CSV, Parquet, Avro), and protocols (SFTP, HTTP, REST). Use staging for cross-region copies.

{ "name":"CopyActivity","type":"Copy","typeProperties":{
  "source":{"type":"AzureSqlSource","sqlReaderQuery":"select * from Orders"},
  "sink":{"type":"DelimitedTextSink","storeSettings":{"type":"AzureBlobFSWriteSettings"}},
  "enableStaging":true,
  "translator":{"type":"TabularTranslator","columnMappings":[{"source":"Id","sink":"OrderId"}]}
}}

Mapping Data Flows: Code-Free Transformations

Data Flows are visual transformation designers that run on Spark clusters. Transformations include joins, aggregations, pivots, lookups, window functions, and derived columns. No code required — drag, drop, and configure. Parameterize for reusability. Monitor Spark execution metrics.

// Data Flow transformations available in the visual designer:
// - Source -> Filter -> Join -> Aggregate -> Derived Column -> Sink
// Common patterns:
// - Slowly Changing Dimension (Type 1 and 2)
// - Star schema dimensional modeling
// - Data quality checks and deduplication

Control Flow: Conditions, Loops, and Variables

Control flow orchestrates activity execution. Execute Pipeline activity triggers child pipelines. If Condition branches execution. ForEach loops over collections. Set Variable stores state. Wait pauses execution. Until and Switch for advanced branching.

{ "name":"CheckAndExecute","type":"IfCondition","typeProperties":{
  "expression":"@greater(activity('GetCount').output.count,0)",
  "ifTrueActivities":[
    {"name":"ExecuteChild","type":"ExecutePipeline","typeProperties":{"pipeline":"TransformData"}}
  ]
}}

Integration Runtimes and Connectivity

Integration Runtime (IR) is the compute infrastructure for data movement and transformation. Azure IR for cloud data movement. Self-Hosted IR for on-premises/private network access. Azure-SSIS IR lifts SQL Server Integration Services (SSIS) packages to the cloud.

# Create self-hosted IR
az datafactory integration-runtime self-hosted create -g MyRG -f MyFactory --name MySelfHostedIR
# Download and install gateway from Microsoft Download Center
# Register with authentication key from the portal

Monitoring, Alerts, and CI/CD

Monitor pipelines in real-time via Azure Portal, Monitor API, or Log Analytics. Set up alerts on pipeline failures and latency. CI/CD via ADF Studio's Git integration (Azure Repos, GitHub). ARM template export for deployment across environments. Global parameters for environment-specific config.

az datafactory pipeline run query --factory-name MyFactory --resource-group MyRG --filters "{\"operand\":\"PipelineName\",\"operator\":\"Equals\",\"values\":[\"IngestSales\"]}"
# CI/CD with ARM templates:
az datafactory validate --factory-name MyFactory --resource-group MyRG --arm-template arm-template.json

Frequently Asked Questions

What is the difference between Azure Data Factory and SSIS?

ADF is a cloud-native PaaS ETL service with pay-per-execution pricing. SSIS is on-premises Windows-based. ADF can run SSIS packages via Azure-SSIS IR for migration scenarios.

Can ADF handle real-time streaming?

ADF primarily handles batch ETL. For real-time streaming, use Azure Stream Analytics, Event Hubs, or Apache Spark on HDInsight/Synapse.

How does ADF handle schema changes in source systems?

Use dataset schema drift options, mapping data flows with schema detection, and flexible column mappings. Set up alerts for failed type conversions.

What is the pricing model for Azure Data Factory?

Pay per activity execution, data movement units, and data flow cluster compute hours. Self-hosted IR costs for the VM running the gateway.

Originally published on Ayodhyyya. Last updated June 1, 2026.