Claude Code Makes Databricks Easy: Jobs, Notebooks, SQL & Unity Catalog (All via the CLI)

Working with Databricks to complete an analysis, build a job, or troubleshoot production workflows can take hours, if not days. But once you enable Claude Code with access to the Databricks CLI, the time you spend on these activities shrinks dramatically. Tasks that used to take all afternoon turn into minutes.

After watching this video, you’ll be able to unlock Databricks in a completely new way, using Claude Code (or any generative AI tool that can operate in your command line) to create Databricks notebooks and jobs, run SQL, troubleshoot issues, automate repetitive steps, and interact with your Databricks environment faster and more productively than ever before.

In this video, I’ll walk you step-by-step through how to build this workflow yourself, including:
1) How to install and authenticate to Databricks with the Databricks CLI
2) How to use the Databricks CLI and Claude Code together to do things like investigate Unity Catalog and run SQL, create notebooks, create jobs, troubleshoot jobs, and more.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

My Databricks CLI Installation Guide: https://github.com/kyle-chalmers/data...
Databricks CLI CLAUDE.md file: https://github.com/kyle-chalmers/data...
My Databricks Workflow Overview: https://github.com/kyle-chalmers/data...
Official Databricks CLI Install Guide: https://developer.atlassian.com/cloud...
Databricks MCP Documentation: https://docs.databricks.com/aws/en/ge...
My Databricks Community Post About the Issue With the Databricks Claude Desktop Connector: https://community.databricks.com/t5/g...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

00:00 - The Value Story and What You Are Going to Learn
02:42 - Issue with Claude Desktop Databricks Connector and Depreciated Databricks MCP
04:16 - Installing the Databricks CLI
05:07 - Authentication with Databricks CLI - OAuth v. PAT
05:48 - OAuth Databricks CLI Authentication
07:32 - PAT Databricks CLI Authentication
09:18 - Test for Both Authentication Methods
10:10 - Databricks CLI Context in CLAUDE.md file
11:12 - Overview of Databricks Example Workflow
11:43 - Claude Code Databricks Prompt Initiation and Plan Review
14:04 - Review of Claude Code’s Notebook Output in CLI and Databricks UI
15:56 - Review of Claude Code’s Databricks Job Output and Troubleshooting
17:11 - Review and Closing Remarks

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

👋 I’m Kyle Chalmers and after 8+ years leading data teams, I’m here to make AI approachable for all data professionals and enthusiasts. Subscribe for weekly/bi-weekly insights, demos, and lessons on the future of data work.

Connect with me on LinkedIn:   / kylechalmers  

I plan on introducing more content in my Substack publication in the future as well: https://substack.com/@kylechalmers1

#Databricks #DatabricksCLI #ClaudeCode #ClaudeAI #AIWorkflow #DataEngineering #AnalyticsEngineering #AIForDataTeams #LLMAutomation #DataAnalytics #AICoding #NotebookAutomation #DatabricksTroubleshooting #JobOrchestration #DatabricksNotebooks