Yesterday I delivered a presentation on "Data Cleansing using Databricks". Listed below are the questions that came up as well as the answers related to Databricks.
- Does state transfer from one cell to another in the Databricks notebook? Yes, state does transfer from one cell to another, as long as you're running in the same notebook session and on the same cluster.
- Can Databricks Jobs (aka "pipelines") be accessed through a notebook? Yes, you can fully interact with Jobs from a notebook using the Databricks REST API or the Databricks CLI.
Some of these interactions from a notebook include:
- Trigger a job run
- Check job run status
- Cancel a run
- Retrieve job metadata
However, you can't directly "open" the Jobs UI from a notebook, and you can't modify job definitions without using the API
Listed below is an example for accessing a Job from a notebook
import requests
import json
token = dbutils.secrets.get("my-scope", "my-token")
workspace = "https://<your-databricks-instance>"
job_id = 12345
resp = requests.post(
f"{workspace}/api/2.1/jobs/run-now",
headers={"Authorization": f"Bearer {token}"},
json={"job_id": job_id}
)
print(resp.json())
- Can one notebook call another notebook? Yes, you can use dbutils.notebook.run() to call other notebooks from Python code.
- Can I import/export SQL Server data to/from Databricks database? Yes. This can be done using JDBC, Azure Data Factory, or Synapse to name a few options
- How do I install and access the Databricks CLI?
From Windows Command Prompt:
> pip install databricks-cli
This will install databricks.exe in the same folder as the Python scripts
(i.e. C:\Users\JimSmith\AppData\Roaming\Python\Python313\Scripts).
If so, the following commands would be need to executed from that folder.
> databricks configure --token
You'll be prompted for:
Databricks workspace URL (Example: https://adb-123456789.12.azuredatabricks.net)
Personal Access Token. This can be generated in Databricks by clicking
User Settings (User icon in upper right corner) → Developer → Generate New Token
After that, the CLI is ready to use.
> databricks clusters list
For more information, see What is the Databricks CLI?
