Unity Catalog introduces several new securable objects to grant privileges to data in cloud object storage.
- A storage credential is a securable object representing an Azure managed identity or Microsoft Entra ID service principal.
- Once a storage credential is created access to it can be granted to principals, users and groups.
- An external location is a securable object that combines a storage path with a storage credential that authorizes access to that path.
Storage credential
A storage credential is an authentication and authorization mechanism for accessing data stored on your cloud tenant.
Once a storage credential is created access to it can be granted to principals (users and groups).
Storage credentials are primarily used to create external locations, which scope access to a specific storage path.
Storage credential names are unqualified and must be unique within the metastore.
External Location
An object that combines a cloud storage path with a storage credential that authorizes access to the cloud storage path.
Step by step Demo
Let’s say I have a container on ADLS, called “mainri-asa-file-system”
1. Allow “access connector” for azure databricks to access
Azure Portal > storage Account > Access Control (IAM) > add role assignment
Add “storage Blob Data Contributor” role
Assign to the access connector for azure databricks
2. Create Storage credential
Azure Databricas > Catalog > add a storage credential
Fill in:
- Credential Type: Azure Managed Identity
- Storage credential name: mainri-asa-file-system-storage-credential
- Access connector ID: /subscriptions/9348xxx108d/resourceGroups/mainri/providers/Microsoft.Databricks/accessConnectors/unity-catalog-access-connector-Premium
To get Access connector ID:
the fork looks this way
3. Grant Permission
Azure Databricks > catalog > storage credentials > permissions > Grant
(or continue from above step 1. Create Storage credential)
Create external Locations
Azure Databricks > Catalog > Add an external location
Fill in :
- External location name: mainri-asa-file-system
- Storage credential
- URL
url pattern: abfss://<container_name>@<storage_account_Name>.dfs.core.windows.net/<path>
So I use this
abfss://mainri-asa-file-system@asamainriadls.dfs.core.windows.net
you might get error, likes this
Error: User does not have CREATE EXTERNAL LOCATION on Metastore ‘mainri-metastore-estus2’.
Reasons: Metastore ‘mainri-metastore-estus2’ was created by erjunchen_entraid@erjunchenmainri.onmicrosoft.com
but I login databricks used erjun.chen@mainri.ca
Solution:
Log out from erjun.chen@mainri.ca , then login use erjunchen_entraid@erjunchenmainri.onmicrosoft.com
error solved.
Please do not hesitate to contact me if you have any questions at William . chen @ mainri.ca
(remove all space from the email account 😊)
Appendix: