Mar '26 Regional Tech Events

User Groups

• Mar 3: Ohio North Database Training
• Mar 11: Azure Cleveland
• Mar 19: GLUG.NET
• Mar 25: Cleveland C# User Group

 

Conferences

• Feb – Jun: Agent Camp
• Mar 16: Memphis Agent Camp

 

Databricks Q&A

Yesterday I delivered a presentation on "Data Cleansing using Databricks".  Listed below are the questions that came up as well as the answers related to Databricks.

 

1. Does state transfer from one cell to another in the Databricks notebook? Yes, state does transfer from one cell to another, as long as you're running in the same notebook session and on the same cluster.

 

2. Can Databricks Jobs (aka "pipelines") be accessed through a notebook? Yes, you can fully interact with Jobs from a notebook using the Databricks REST API or the Databricks CLI.

Some of these interactions from a notebook include:

• Trigger a job run
• Check job run status
• Cancel a run
• Retrieve job metadata

 

However, you can't directly "open" the Jobs UI from a notebook, and you can't modify job definitions without using the API

 

Listed below is an example for accessing a Job from a notebook

import requests

import json

 

token = dbutils.secrets.get("my-scope", "my-token")

workspace = "https://<your-databricks-instance>"

 

job_id = 12345

 

resp = requests.post(

    f"{workspace}/api/2.1/jobs/run-now",

    headers={"Authorization": f"Bearer {token}"},

    json={"job_id": job_id}

)

 

print(resp.json())

 

3. Can one notebook call another notebook? Yes, you can use dbutils.notebook.run() to call other notebooks from Python code.

 

4. Can I import/export SQL Server data to/from Databricks database? Yes. This can be done using JDBC, Azure Data Factory, or Synapse to name a few options

 

5. How do I install and access the Databricks CLI?

From Windows Command Prompt:

> pip install databricks-cli

 

This will install databricks.exe in the same folder as the Python scripts

(i.e. C:\Users\JimSmith\AppData\Roaming\Python\Python313\Scripts).

If so, the following commands would be need to executed from that folder.

 

> databricks configure --token

 

You'll be prompted for:

Databricks workspace URL (Example: https://adb-123456789.12.azuredatabricks.net)

Personal Access Token.  This can be generated in Databricks by clicking

User Settings (User icon in upper right corner) → Developer → Generate New Token

 

After that, the CLI is ready to use.

> databricks clusters list

 

For more information, see What is the Databricks CLI?