Azure Data Lake Storage, Databricks, and Synapse

A data lake is a repository of data stored in its natural format, usually as blobs or files. On Azure that's Azure Data Lake Storage, and the two analytics platforms that sit on top of it — Databricks and Synapse — overlap enough that choosing between them takes some care. It's also the corner of Azure moving fastest right now, so it's worth being explicit about which parts are stable and which are in transition.

Azure Data Lake Storage

Data Lake Storage is a scalable, cost-effective data lake solution for big data analytics built into Azure, combining a file system with a storage platform. It builds on Azure Blob Storage, inheriting Azure Storage's performance, availability, security, and durability.

It stores any type of data in its native format — structured, semi-structured, and unstructured — at massive scale.
It's designed primarily for Hadoop and any framework using HDFS as its data access layer, which can access the data directly.
It supports high throughput for I/O-intensive analytics and data movement.
Its access control model supports both Azure RBAC and POSIX ACLs.
It's priced at Azure Blob Storage levels — the analytics capability doesn't carry a storage premium.

What actually distinguishes it from Blob Storage

Course material commonly carries a Data Lake versus Blob Storage comparison table claiming that Data Lake "must manually configure data replication" while Blob "provides geo-redundant storage by default", and that Blob doesn't support granular access. Those rows are wrong, and the reason is structural: Data Lake Storage Gen2 is Blob Storage with the hierarchical namespace enabled. It's not a separate service sitting alongside it. It uses the same redundancy options as any storage account, and POSIX ACLs are a Gen2 capability layered on Blob Storage rather than something Blob lacks.

The differences that are real:

	Blob Storage	Data Lake Storage Gen2
Namespace	Flat — folders are virtual, part of the object name	Hierarchical — true file system directories
Access path	REST APIs, client libraries	Adds HDFS-compatible access via the ABFS driver
Directory operations	Per-object across everything beneath	Metadata operations
Access control	Azure RBAC	Azure RBAC and POSIX ACLs

That third row is the whole argument for analytics workloads. Renaming or deleting a directory in a flat namespace touches every object underneath it; in a hierarchical namespace it's a single metadata change. For pipelines that reorganise large datasets between stages, the difference is enormous.

One caveat before enabling it: several blob features aren't supported on accounts with a hierarchical namespace, including point-in-time restore, blob versioning, and object replication. Check the feature support matrix before switching it on rather than after.

Using the lake

Three steps, and the ingestion routes differ by data shape:

Ingest data. Ad-hoc data via AzCopy, the Azure CLI, PowerShell, or Storage Explorer. Relational data via Azure Data Factory, from sources including Cosmos DB, SQL Database, and SQL Managed Instance. Streaming data via Azure Stream Analytics, or Event Hubs Capture and IoT Hub routing.
Access the stored data.
Configure access control over who can read what, using RBAC and ACLs.

Azure Databricks

Azure Databricks is a fully managed cloud-based big data and machine learning platform, giving data science and engineering teams one place for large-scale processing and ML. It's built entirely on Apache Spark, and its managed Spark platform makes large-scale Spark workloads straightforward to run. Data scientists get a built-in API for SQL, Java, Python, R, and Scala.

The architecture split worth knowing

Databricks separates into two planes, and this is the answer to "where does my data actually go":

Control plane — hosts jobs, notebooks with query results, and the cluster manager, plus the web application, hive metastore, security ACLs, and user sessions. Microsoft manages these in collaboration with Databricks.
Data plane — contains the runtime clusters hosted within the workspace. All data processing and storage happens within the client subscription. No data processing ever takes place in the Microsoft or Databricks-managed subscription.

That split is the security answer for anyone asking where their data is processed, and it's the detail worth remembering from any Databricks overview.

Databricks provides environments for different kinds of work: SQL for analysts running queries against a data lake, with visualisations and shareable dashboards; Data Science and Engineering, an interactive workspace for collaboration between data engineers, data scientists, and ML engineers; and Machine Learning, an end-to-end environment with managed services for experiment tracking, model training, feature development, and model serving.

A typical pipeline through it: data is ingested into Azure through Data Factory in batches, or streamed near real-time via Apache Kafka, Event Hubs, or IoT Hub; it lands in a data lake for long-term persistence in Blob Storage or Data Lake Storage; then Databricks reads from multiple sources and turns it into insight using Spark.

Azure Synapse Analytics

Synapse combines big data analytics, enterprise data warehousing, and data integration, running queries over serverless or provisioned resources at scale. It supports ingestion, exploration, transformation, and management, for both BI and machine learning.

It implements a massively parallel processing (MPP) architecture: a control node acts as the front end all applications interact with, and a pool of compute nodes provides the processing power, with data distributed evenly across them. Queries are submitted as Transact-SQL and Synapse runs them.

Five components:

SQL pool — serverless and dedicated resource models over a node-based architecture. Dedicated pools give predictable performance and cost; the always-available serverless SQL endpoint suits irregular or unplanned workloads.
Spark pool — a cluster running Apache Spark for data preparation, engineering, ETL, and machine learning, with logic written in Python, Scala, SQL, or C# via .NET for Apache Spark.
Pipelines — the capabilities of Azure Data Factory, covered in the post on Data Factory and integration runtimes.
Synapse Link — connects to Azure Cosmos DB for near real-time analytics over operational data, but is no longer supported for new projects (see below).
Synapse Studio — a web-based IDE for creating SQL and Spark pools, defining and running pipelines, and configuring links to external data sources.

Where this is heading

Two things are worth knowing if you're planning new work rather than sitting an exam. Microsoft Fabric is now the strategic direction for analytics on Azure, with a migration path from Synapse dedicated SQL pools into Fabric warehouses. And Synapse Link for Cosmos DB is no longer supported for new projects — Microsoft's own documentation says not to use it, and points to Cosmos DB mirroring for Microsoft Fabric, now generally available, instead.

Synapse as a whole hasn't been retired, and the exam material still describes it as above — so both remain worth knowing. But a greenfield analytics platform started today deserves a look at Fabric before committing to Synapse.

Choosing between them

Databricks where the work is Spark-native — data science, machine learning, complex transformation — and the team is comfortable writing code.
Synapse where the centre of gravity is SQL and data warehousing, and you want pipelines, warehousing, and Spark in one workspace.
Both is common and legitimate: Databricks for engineering and ML, Synapse or Fabric for the warehouse and BI layer, over one shared data lake.

The lake underneath doesn't change, which is the point of keeping it a lake.