Secret Scopes:
A secret scope is a boundary within which secrets are stored. We can create secret scopes to securely manage access to sensitive data and credentials.
Each scope can contain multiple secrets (key-value pairs) like tokens or passwords.
Secrets:
Secrets are the actual values you want to store securely (e.g., API tokens, connection strings). You can store secrets in specific scopes.
Types of Secret Scopes in Databricks
- Databricks-Backed Secret Scopes: Stored directly in Databricks and managed entirely within the platform.
- Azure Key Vault-Backed Secret Scopes: Secrets are stored in Azure Key Vault and managed externally but accessed from within Databricks.
Common dbutils.secrets Commands:
- dbutils.secrets.listScopes(): Lists all the available secret scopes.
- dbutils.secrets.list(“scope-name”): Lists all the secrets within a given scope.
- dbutils.secrets.get(“scope-name”, “key”): Retrieves the secret for the given scope and key.
#Lists all the available secret scopes
dbutils.secrets.listScopes()
#Lists all the secrets within a given scope
dbutils.secrets.list("scope-name")
# Retrieves the secret for the given scope and key
my_secret = dbutils.secrets.get(scope="my_scope", key="my_secret_key")
example in notebooks and response are shown later.
Creating and Managing Secret Scopes
To create and manage secret scopes, you can either use the Databricks CLI or Databricks UI, depending on the permissions and environment you’re working with.
Create an Azure Key Vault-backed secret scope
1: Go to https://<databricks-instance>/#secrets/createScope.
Replace <databricks-instance> with the workspace URL of your Azure Databricks deployment. This URL is case sensitive (scope in createScope must be uppercase).
e.g. https://adb-44260********40553.13.azuredatabricks.net/#secrets/createScope
2: Enter the name of the secret scope.
Secret scope names are case insensitive
3. Manage Principal
Use the Manage Principal drop-down to specify whether All Users have MANAGE permission for this secret scope or only the Creator of the secret scope (that is to say, you).
4. DNS Name and Resource ID
DNS Name, for example, https://mainri-key-vault.vault.azure.net/
These properties are available from the Settings > Properties tab of an Azure Key Vault in your Azure portal.
5. Click the Create button.
The secret scope named mainri_secret_scopes has been added.
Manage secrets in this scope in Azure KeyVault with manage principal = creator
Create an Azure Key Vault-backed secret
Nothing special, the normal proceed to create secret in azure key vault, omitting.
If you need, please review my previous article “Create and Using Key Vault in Azure Ecosystem”.
Create Secret scopes using Databricks CLI
#Python
# create Secret scopes
databricks secrets create-scope --initial-manage-principal users
#bash
%bash
databricks secrets put --scope <existing-scope-name> --key <secret-key>
Use Secrets
dbutils.secrets.listScopes()
dbutils.secrets.list('mainri_secret_scopes')
dbutils.secrets.get(scope="mainri_secret_scopes", key="creator")
Please do not hesitate to contact me if you have any questions at William . chen @ mainri.ca
(remove all space from the email account 😊)
Appendix