Mounting ADLS in Databricks? Here's What You're Missing

The Problem

You’re building a data pipeline in Databricks. Before you write a single line of transformation logic, there’s one question you have to answer first: your source data sits in ADLS Gen2, so how does Databricks actually get access to it?

This sounds trivial, but it’s the first real design decision in any pipeline, and how you answer it decides whether your setup is secure and governable, or a liability waiting to be found in a security review.

There are three ways teams have solved this over time:

1. Hardcoding storage account keys in Spark config
2. Mounting the storage account (dbutils.fs.mount)
3. Storage Credentials + External Locations (Unity Catalog)

Let’s go through why the first two approaches don’t hold up, and why the third one is now the standard.

Way 1: Spark Config with Storage Account Key

The simplest way to give Databricks access to ADLS is to pass the storage account’s access key directly into the Spark session.

spark.conf.set(

    "fs.azure.account.key.<storage-account-name>.dfs.core.windows.net",

    "<storage-account-access-key>"

)

df = spark.read.format("csv").load("abfss://<container>@<storage-account-name>.dfs.core.windows.net/raw/sales/")

Why this isn’t the right way

• The key is plaintext. Anyone with access to the notebook, job config, or cluster environment variable can read it and use it outside Databricks entirely.
• The key doesn’t expire. If it leaks, whoever has it has full account-level access - read, write, delete - until someone manually rotates it.
• No per-user access control. Every user running this code gets the exact same access, regardless of what they actually need.
• Rotation breaks things. The moment the key is rotated for security hygiene, every notebook, job, and cluster referencing the old key fails, usually with something like:

Operation failed: "Server failed to authenticate the request." 403 (AuthenticationFailed)

This approach works for a five-minute demo. It does not belong in production.

Way 2: Mounting the Storage Account

Mounting was the next step up - instead of passing keys around in every notebook, you mount the storage account once, and everyone reads from a clean path like /mnt/sales-data/.

dbutils.fs.mount(

    source="wasbs://<container>@<storage-account-name>.blob.core.windows.net/",

    mount_point="/mnt/sales-data",

    extra_configs=configs

)

df = spark.read.format("csv").load("/mnt/sales-data/raw/sales/")

This was a genuine improvement; at least the secret is pulled from a secret scope instead of sitting in code. For a while, this was the recommended pattern.

Why teams are moving away from it

• Mounts are workspace-global, not user-scoped. Once someone mounts /mnt/sales-data, it’s visible and usable by every user in the workspace - there’s no way to say “only the finance team can read this.”
• The access behind the mount is still one identity. It’s a service principal instead of a raw key, but it’s still one broad credential shared by everyone who touches that mount. The blast radius problem from Way 1 hasn’t gone away - it’s just one layer deeper.
• No audit trail. You can’t tell, after the fact, which user read which file through the mount. For any regulated data (PII, financial records), that’s a compliance gap.

Operational fragility. Real errors teams run into:

com.databricks.backend.daemon.dbutils.FileInfo:

Directory already mounted: /mnt/sales-data
shaded.databricks.org.apache.hadoop.fs.azure.AzureException:

No credentials found for account <storage-account> in the configuration

• Mounts silently breaking after secret rotation, two people trying to mount/unmount the same path and colliding, stale mounts pointing at credentials that no longer exist.

It doesn’t fit Unity Catalog’s governance model at all. Unity Catalog governs access at the catalog/schema/table level with fine-grained permissions; a workspace-global mount point bypasses that entirely. Databricks has officially marked mounting as legacy for this reason.

Mounting solved a convenience problem. It never solved the governance problem.

The Shift: Storage Credentials & External Locations

This is where Unity Catalog changes the model. Instead of a shared secret or a shared mount, access to ADLS is defined once, centrally, and governed like any other object in Unity Catalog.

Storage Credential - a UC object that wraps an Azure Access Connector’s managed identity. There’s no key or secret involved at all, Azure handles the identity, and UC references it.

External Location - a UC object that binds a specific ADLS path (container/folder) to a Storage Credential. Once this exists, access to that path is controlled with standard GRANT / REVOKE statements, just like a table.

%sql
-- Grant a team read access to a specific external location

GRANT READ FILES ON EXTERNAL LOCATION sales_raw_location TO data-engineering-team;

-- Revoke it just as easily

REVOKE READ FILES ON EXTERNAL LOCATION sales_raw_location FROM data-engineering-team;

Why this actually solves the business problem

Go back to the original problem statement: the business needs data ingested from ADLS reliably, but it also needs to know who can access what, be able to revoke access instantly, and prove that during an audit. None of the earlier approaches could do all three at once.

Setting It Up

1. Storage Credential in Unity Catalog

A Storage Credential doesn’t stand on its own; it needs an Access Connector behind it (the managed identity we walked through earlier). We’re not re-walking the connector creation here; assume it already exists. Here’s ours (demo-acc-conn), with its Resource ID copied, ready to hand to Databricks:

In Catalog Explorer, creating the Storage Credential is just pointing UC at that connector. Choose credential type Azure Managed Identity, give it a name:

Scroll down and paste the connector’s Resource ID into Access Connector ID, this is the field that ties the two together:

Save it, and the credential is created, no key, no secret, just a reference to the managed identity:

Notes:

• Creating this requires the CREATE STORAGE CREDENTIAL privilege at the metastore level
• At this point, Databricks hasn’t verified anything yet; it’s just recorded that this credential should point to that identity. Whether that identity can actually reach the storage account gets tested next.

2. External Location

This is where the permission requirements actually surface. Fill in the path and pick the Storage Credential you just created:

Hit Create, and Databricks immediately tests the connection, and it fails:

Read, List, and Write all fail. The error is blunt: “The associated Storage Credential does not grant permission to perform all necessary operations.” The credential exists, but the managed identity behind it has no role on the storage account yet - Unity Catalog can’t do anything until Azure IAM says it’s allowed to.

So we go to the storage account → Access Control (IAM) → Add role assignment → assign Storage Blob Data Contributor to the connector’s managed identity:

Test the connection again, and it’s still not fully working. The error now spells out exactly what’s missing:

“Please ensure the IAM roles Storage Account Contributor, Storage Blob Data Contributor, EventGrid EventSubscription Contributor, Storage Queue Data Contributor are correctly assigned to your managed identity...”

This is the real-world gotcha: Storage Blob Data Contributor alone gets you basic data access, but Unity Catalog also needs Storage Account Contributor for management-plane operations (like provisioning file events). So we go back to IAM and assign that role too, the same way as above.

Test connection one more time - Read, List, Write, and Delete all pass:

(File Events Read can still fail here if you didn’t assign the EventGrid/Queue roles - that’s only needed if you plan to use file notifications/Auto Loader with file events, not for basic reads.)

And now the External Location can actually browse the container - the ADLS files are visible right inside Databricks:

Notes:

• Creating an External Location requires the CREATE EXTERNAL LOCATION privilege
• Access after this point is controlled with GRANT READ FILES / GRANT WRITE FILES on the external location, the IAM roles above are what let the connector reach storage; UC grants are what let specific users/teams use that external location

Closing

Mounting solved a convenience problem, one clean path instead of keys scattered across notebooks. Storage Credentials and External Locations solve the actual problem the business has: governed, auditable, revocable access to data. That’s what a production pipeline needs, and that’s why the shift is happening.