Azure Data Factory and Integration Runtimes
Azure Data Factory is a cloud-based data integration service for creating and scheduling data-driven workflows. It orchestrates data movement and transforms data at scale, with pipelines that ingest from disparate stores and combine them into a single destination. It's an ETL service — extract, transform, load — and the part that generates the most design questions isn't the pipelines but the runtime that actually executes them.
The workflow it supports
- Connect and collect — ingest data from different sources into a centralised location.
- Transform and enrich — transform using a compute service such as Azure Databricks or Azure HDInsight Hadoop.
- Publish with CI/CD — use GitHub and Azure Pipelines to deliver the ETL process incrementally before publishing to the analytics engine.
- Monitor — watch the pipeline in the Azure portal for scheduled activities and failures.
Step three is the one that separates a maintainable data platform from a fragile one. Treating pipelines as source-controlled artefacts deployed through a release process, rather than as things edited live in a portal, is what makes the difference when a transformation needs changing under pressure.
The components
Pipelines and activities. A pipeline is a logical grouping of activities that together perform a task; an activity is a single processing step. Data Factory supports data movement, data transformation, and control activities — the last of which is what gives pipelines branching, looping, and conditional execution rather than a straight line.
Datasets are data structures within the data stores — they name and shape the data an activity reads or writes, without holding it.
Linked services define the connection information Data Factory needs to reach external resources. If a dataset says what, a linked service says where and with which credentials.
Data flows let data engineers develop transformation logic without writing code, and can be operationalised through the existing scheduling, control flow, and monitoring capabilities. That's the visual-transformation surface, for teams who'd rather not maintain Spark code.
Integration runtimes are the bridge between activities and linked services — the compute that actually performs the movement or dispatches the activity.
The three integration runtimes
This is the part worth knowing precisely, because it's what most design questions turn on:
| Runtime | Where it runs | What it's for |
|---|---|---|
| Azure | Managed by Azure | Data movement between cloud data stores, and dispatching transform activities to cloud compute |
| Self-hosted | On a machine you run | Reaching data sources inside a private network — on-premises, or in a virtual network |
| Azure-SSIS | Managed by Azure | Lifting and shifting existing SQL Server Integration Services packages |
The self-hosted runtime, specifically
Moving data from an on-premises source to Azure storage requires a self-hosted integration runtime installed on the on-premises machine. It acts as a secure bridge enabling data movement and activity dispatch between on-premises sources and Data Factory.
The placement rule is the detail that gets tested and, more importantly, the one that gets deployments wrong: the self-hosted IR must be installed on the source server itself, or on another machine in the same network with access to the data. Installing it somewhere that can reach Azure but not the data source achieves nothing — it's the data-side reachability that matters, not the Azure-side.
The reason this design exists is worth stating: the runtime makes outbound connections to Azure, so it works without inbound firewall rules or a VPN. That's what makes it viable in networks where opening inbound access would be a non-starter.
Where it fits against Synapse
Data Factory's capabilities also appear inside Azure Synapse Analytics as Synapse Pipelines, and the two are close enough that choosing between them comes down to a handful of differences:
| Azure Data Factory | Synapse Pipelines | |
|---|---|---|
| Cross-region integration runtime (data flows) | Supported | Not supported |
| Integration runtime sharing across factories | Supported | Not supported |
| Power Query activity | Supported | Not supported |
| Global parameters | Supported | Not supported |
| Deploying pipelines and flows with ARM templates | Supported | Not supported |
| Solution templates | Data Factory template gallery | Synapse Workspace Knowledge centre |
| Monitoring Spark jobs for data flow | Not supported | Supported via Synapse Spark pools |
The rows that usually decide it: if you need cross-region data flows, sharing across factories, global parameters, or ARM-template deployment of pipelines, Data Factory. If your transformation work is happening in Spark pools you want to monitor alongside the pipelines, Synapse. Microsoft Fabric also carries its own Data Factory experience, which is where Microsoft's analytics direction points — the Data Lake and Synapse post covers that shift.
Ingesting into a data lake
Data Factory is one of three routes into Azure Data Lake Storage, and which you use depends on the data's shape:
- Ad-hoc data — AzCopy, the Azure CLI, PowerShell, or Azure Storage Explorer.
- Relational data — Azure Data Factory, transferring from sources including Cosmos DB, SQL Database, and SQL Managed Instance.
- Streaming data — Azure Stream Analytics, or Event Hubs Capture and IoT Hub message routing to land events in the lake directly.
The lake itself, and what sits on top of it, is covered in the post on Data Lake Storage and Synapse Analytics.