What Happens When Claude Meets Databricks?

The Problem

Meet the data engineering team at GlobalMart, a mid-size e-commerce company running its entire analytics platform on Databricks. They build and maintain a medallion architecture: raw CSV files land in a volume, get ingested into Bronze Delta tables, cleaned in Silver, and aggregated into Gold for the business team to query.

The pipeline works. But building it is slow. Every sprint looks like this:

The Databricks Field Engineering team recently introduced the AI Dev Kit, an open-source toolkit that connects Claude Code, Gemini, Cursor, etc., directly to your Databricks workspace. GlobalMart’s team decided to try it for their next pipeline sprint. This is what they learned.

Prerequisites

Before installing AI Dev Kit, make sure the following are in place:

• Python 3.10+ - python --version
• Databricks CLI v0.200+ - used once for initial workspace configuration
• Claude Code (latest) - the CLI that runs AI Dev Kit. Requires a Claude Pro subscription.
• Git - to clone the AI Dev Kit repository
• Databricks workspace - with Unity Catalog and Serverless compute enabled
• Databricks PAT token - Settings → Developer → Access Tokens → Generate New Token

Databricks CLI - one-time use only.

You only need it to run databricks configure --profile DEFAULT.

databricks configure --profile DEFAULT

This saves your workspace URL and token to ~/.databrickscfg. After that, the MCP server reads this file automatically, and you never need the CLI again.

Installing AI Dev Kit

AI Dev Kit is open source and maintained by Databricks Field Engineering. The full documentation and installation guide is available on GitHub:

https://github.com/databricks-solutions/ai-dev-kit

Step-by-step setup guide used in this post:

https://github.com/mentorskool-org/databricks-ai-dev-kit-workshops/blob/main/00_SETUP_GUIDE.md

Installation Steps

• Step 1 - Run the installer

On Windows, open PowerShell and run this single command:

powershell powershell -ExecutionPolicy Bypass -Command "irm https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.ps1 | iex"

This one command does everything automatically:

• Creates a Python virtual environment at ~/.ai-dev-kit/.venv/ and installs all dependencies
• Copies the 37 skill files into ~/.claude/skills/
• Creates ~/.claude/mcp.json and registers the Databricks MCP server in it

Screen recording of running the PowerShell install command. Show the full terminal output cloning, installing dependencies, copying skills, and writing mcp.json until the installer completes successfully.

After installation, two folders are created on your machine:

What is inside mcp.json? The setup script creates ~/.claude/mcp.json and writes the Databricks server registration into it. Here is what the file contains:

{
  "mcpServers": {
    "databricks": {
      "command": "~/.ai-dev-kit/.venv/python",
      "args": ["~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py"]
    }
  }
}

This file has one job: tell Claude how to start the MCP server. When you type claude Claude reads this file and executes:

~/.ai-dev-kit/.venv/python run_server.py

That starts the MCP server as a background process. The key "databricks" is the name of the server; it is how Claude identifies which server to route Databricks tool calls to. If you ever needed to add a second MCP server (for a different tool), you would add another key alongside "databricks" in this same file.

You never edit mcp.json manually. The setup script writes it for you. The only reason to look at it is to understand how Claude knows where the MCP server lives, or to troubleshoot if the MCP connection stops working.

How It Works - The 4 Components

AI Dev Kit has four components that work together every time you type a prompt in Claude Code:

• databricks-skills - 37 markdown files loaded into Claude’s memory at startup. They teach Claude which tools to use, when to use them, and Databricks best practices.
• databricks-mcp-server - A Python server running in the background on your machine. Receives tool call requests from Claude and routes them to tools-core.
• databricks-tools-core - The Python library that contains the actual functions. It uses the Databricks SDK and your saved token to call Databricks REST APIs.
• databricks-builder-app - A React + FastAPI app for building Databricks Apps. Not covered in this post.

What is MCP?

MCP stands for Model Context Protocol - an open standard created by Anthropic. It is the communication format that allows Claude to call external tools running on your machine.

Think of it as a pipe between two processes on your laptop:

Claude Code process  ←──── stdio pipe ────→  MCP Server process

  (your terminal)         JSON messages       (run_server.py)

When Claude wants to do something in Databricks, it sends a JSON message through this pipe. The MCP server receives it, calls the right Python function, and sends the result back.

How do you know MCP is working?

When you see manage_sql_warehouse (MCP) In Claude’s output, the full chain is connected. If you see a Bash command like databricks warehouses list Instead, MCP is not running.

List catalogs went through MCP. No copy-paste, no manual execution. Claude called Databricks directly from the terminal, authenticated with your saved token.

Coming in Part 2

Now that the stack is set up and the connection is verified, it’s time to build. In Part 2, GlobalMart’s team uses AI Dev Kit to build their full Bronze → Silver → Gold pipeline:

• Why Claude hallucinates without context - and how to fix it
• Building Bronze, Silver, and Gold layers with natural language prompts
• How to prevent Claude from accidentally dropping your tables
• Generating column metadata using AI