materialize-databricks-create skill
✓ VerifiedCreate Databricks materializations to stream Estuary collections into Unity Catalog tables using flowctl.
- GitHub stars
- 7
- Files
- 1
- Last commit
- Nov 4, 2026
- License
- Apache-2.0
- Allowed tools
- —
What does the materialize-databricks-create skill do?
materialize-databricks-create creates Databricks materializations to stream Estuary collections into Unity Catalog tables using flowctl.
Summary generated from the skill's SKILL.md.
About the materialize-databricks-create skill
Streaming Estuary collections into Databricks Unity Catalog tables is the primary function of this skill. It assists developers and data engineers in configuring Databricks as a destination for captured data by generating the necessary flowctl configurations. The process involves gathering specific requirements such as workspace URLs, SQL Warehouse paths, and authentication credentials like Personal Access Tokens or OAuth2 Machine-to-Machine settings.
An agent should load this skill when a user requests to send, materialize, or load data into Databricks. The skill produces a complete flow.yaml specification file and provides the exact flowctl commands required to publish and verify the materialization. It relies on external documentation via WebFetch to ensure prerequisites like Unity Catalog and active SQL Warehouses are met. Unlike manual configuration, this skill automates the retrieval of the latest connector image tags from the registry and provides specific troubleshooting steps for common errors like gRPC size limits or schema mismatches.
What an agent can do with it
- 1Generate materialization configurationBuild a flow.yaml file with the correct connector version and authentication settings.
- 2Troubleshoot sync errorsResolve issues related to column mapping, large document sizes, or stuck pending statuses.
- 3Automate connector versioningRetrieve the most recent image tag from the connector registry to avoid hardcoding versions.
Written from the skill's SKILL.md. Every use case is checked against a quote from it.
How to install the materialize-databricks-create skill
Each command copies the skill folder into the client's skills directory. Restart the client or start a new session to load it.Install the materialize-databricks-create skill in Plugin
/plugin marketplace add estuary/agent-skills /plugin install estuary-materializations@estuary
Install the materialize-databricks-create skill in Claude Code
~/.claude/skills/materialize-databricks-create/SKILL.md
$ npx degit estuary/agent-skills/skills/materialize-databricks-create ~/.claude/skills/materialize-databricks-createInstall the materialize-databricks-create skill in Claude Code (project)
.claude/skills/materialize-databricks-create/SKILL.md
$ npx degit estuary/agent-skills/skills/materialize-databricks-create .claude/skills/materialize-databricks-createInstall the materialize-databricks-create skill in Codex
~/.agents/skills/materialize-databricks-create/SKILL.md
$ npx degit estuary/agent-skills/skills/materialize-databricks-create ~/.agents/skills/materialize-databricks-createInstall the materialize-databricks-create skill in Gemini CLI
~/.gemini/skills/materialize-databricks-create/SKILL.md
$ npx degit estuary/agent-skills/skills/materialize-databricks-create ~/.gemini/skills/materialize-databricks-createInstall the materialize-databricks-create skill in Claude.ai
npx degit estuary/agent-skills/skills/materialize-databricks-create materialize-databricks-create && zip -r materialize-databricks-create.zip materialize-databricks-create # claude.ai → Customize → Skills → + → Create skill → Upload a skill → materialize-databricks-create.zip
Automated checks
Ran Nov 16, 2026Full report →Passed our automated checks. That is not a security audit: read the scripts before you let an agent run them. How checks work
- Source repository linked and reachableestuary/agent-skills/skills/materialize-databricks-create is public and reachablegithub.com/estuary/agent-skills/tree/HEAD/skills/materialize ↗Pass
- Open source license (OSI)Apache-2.0spdx.org/licenses/Apache-2.0.html ↗Pass
- Commit or release in the last yearLast commit 2026-11-04github.com/estuary/agent-skills/commits ↗Pass
- No download-and-run or credential theft in scriptsNo scripts in the skill folderPass
- No hidden instructions in SKILL.mdNo override phrases, safeguard switches, credential requests or hidden charactersgithub.com/estuary/agent-skills/blob/HEAD/skills/materialize ↗Pass
- Pre-approved tools are scopedNo pre-approved tools; the agent asks before using onegithub.com/estuary/agent-skills/blob/HEAD/skills/materialize ↗Pass
- SKILL.md follows the Agent Skills formatValid name and description; says when to use it; 214 linesgithub.com/estuary/agent-skills/blob/HEAD/skills/materialize ↗Pass
SKILL.md
View on GitHubCreate Databricks Materialization
Create a Databricks materialization using flowctl to stream data from Estuary collections into Databricks Unity Catalog tables.
Applies to: materialize-databricks
Step 0: Load Connector Documentation
Before proceeding, fetch the official connector docs for prerequisites, config reference, and setup instructions.
Load the docs page: https://docs.estuary.dev/reference/Connectors/materialization-connectors/databricks/
Use WebFetch to load this page. It covers:
- Prerequisites (Unity Catalog, SQL Warehouse, credentials)
- Full config property reference
- Authentication setup (PAT or OAuth2 M2M)
- Staging via Unity Catalog Volumes
Search Kapa for tribal knowledge (if the Estuary MCP is configured):
Search kapa ai knowledge sources for "materialize databricks common issues"
If Kapa MCP is not configured, the user can set it up: https://docs.estuary.dev/features/mcp-integration/
This skill provides the flowctl workflow and troubleshooting that docs don't cover.
Step 1: Gather Requirements
Before writing any YAML, ask the user:
- Workspace URL? — Databricks workspace hostname (e.g.,
dbc-abc123.cloud.databricks.com) - SQL Warehouse HTTP path? — Found in SQL Warehouse > Connection details
- Catalog and schema? — Unity Catalog name and target schema
- Authentication method? — Personal Access Token (PAT) or OAuth2 Machine-to-Machine (M2M)
- Non-default data plane? — Most users use the default. Ask if they need a non-default data plane.
- Source collections? — Which Estuary collections to materialize
- Hard deletes? — Off by default. Without it, deleted rows stay in Databricks marked with
_meta/op: 'd'. Enable to physically remove them. - Delta updates? — Off by default. Switches from one-row-per-key (standard merge) to append-only. Use for event logs or history tables.
- Sync schedule? — Controls how often batches are written to Databricks (default: 30 minutes,
0sfor real-time). Affects latency and compute cost.
Step 2: Find the Correct Connector Version
Always use the latest numbered version tag. Query the connector registry:
flowctl raw get --table connector_tags \
--query 'documentation_url=eq.https://go.estuary.dev/materialize-databricks' \
--query 'select=image_tag,documentation_url' \
--output yaml
Use the returned image_tag — never hardcode a version.
Step 3: Help User Complete Prerequisites
Walk the user through prerequisites from the docs loaded in Step 0:
- Unity Catalog enabled — Required for staging data via Volumes
- SQL Warehouse running — Must be active during publish and ongoing operation
- Schema exists — Create the target schema if it doesn't exist
- Credentials ready — PAT token or OAuth2 M2M client ID/secret
Staging: The connector automatically creates a Unity Catalog Volume (flow_staging) for staging — no user setup needed for storage.
Refer to the docs page for exact setup steps.
Step 4: Create the Spec File
Build flow.yaml using the config reference from the docs.
With Personal Access Token (PAT):
materializations:
<TENANT>/<PATH>/materialize-databricks:
endpoint:
connector:
image: ghcr.io/estuary/materialize-databricks:<VERSION>
config:
address: "<WORKSPACE>.cloud.databricks.com"
http_path: "/sql/1.0/warehouses/<WAREHOUSE-ID>"
catalog_name: "<CATALOG>"
schema_name: "<SCHEMA>"
credentials:
auth_type: "PAT"
personal_access_token: "<TOKEN>"
bindings:
- source: <TENANT>/<collection-path>
resource:
table: "<TABLE_NAME>"
With OAuth2 Machine-to-Machine (M2M):
materializations:
<TENANT>/<PATH>/materialize-databricks:
endpoint:
connector:
image: ghcr.io/estuary/materialize-databricks:<VERSION>
config:
address: "<WORKSPACE>.cloud.databricks.com"
http_path: "/sql/1.0/warehouses/<WAREHOUSE-ID>"
catalog_name: "<CATALOG>"
schema_name: "<SCHEMA>"
credentials:
auth_type: "OAuth2 M2M"
client_id: "<CLIENT-ID>"
client_secret: "<CLIENT-SECRET>"
bindings:
- source: <TENANT>/<collection-path>
resource:
table: "<TABLE_NAME>"
Important: The connector is single-threaded — one failing binding blocks all bindings in the materialization. Group critical tables together and isolate experimental ones.
Step 5: Publish
flowctl catalog publish --source flow.yaml --auto-approve
Step 6: Verify
# Check status
flowctl catalog status <TENANT>/<PATH>/materialize-databricks
# View logs
flowctl logs --task <TENANT>/<PATH>/materialize-databricks --since 5m | jq -c '{ts, message}'
Status progression:
PENDING— Normal for ~30 seconds during shard assignmentBACKFILLING— Initial data sync from collectionsOK— Running normally with real-time updates
Troubleshooting
"Field selection validation failed"
Cause: Usually a UI state issue (stale cache) or schema evolution conflict (required field no longer exists in collection)
Fix:
- Try a browser cache clear and reload
- Click the "Refresh" button in the Field Selection section
- Check if any required fields were pruned from the collection's inferred schema (complexity limit of 1000 fields)
- If a required field disappeared, either un-require it or re-add it to the collection schema
DELTA_UNSUPPORTED_DROP_COLUMN
Cause: Connector tried to drop a column but the table's column mapping mode doesn't support it
Fix: Enable column mapping mode on the affected table in Databricks:
ALTER TABLE <catalog>.<schema>.<table> SET TBLPROPERTIES ('delta.columnMapping.mode' = 'name');
"schema not found" or "catalog not found"
Cause: catalog_name or schema_name is misspelled, or credentials lack access
Fix: Verify the exact catalog and schema names in Databricks. Check that the service principal or PAT user has access to both.
64 MB gRPC document size limit
Cause: Very large documents exceeding the gRPC message size limit
Fix: Use field selection to exclude large fields, or create a derivation to truncate oversized documents before materializing.
"Merge query too large"
Cause: Too many bindings or too many fields in a single materialization
Fix: Split into multiple materializations with fewer bindings, or use field selection to reduce the number of fields per binding.
PAT re-entry needed after backfill
Cause: Known issue where PAT may need to be re-entered after triggering a backfill
Fix: Re-enter the same PAT in the connector configuration. This is a known UI quirk, not a credential expiration.
SQL Warehouse auto-stopped
Cause: SQL Warehouse stopped due to inactivity timeout
Fix: Ensure the SQL Warehouse auto-stop timeout is set long enough, or the connector will restart it automatically on the next sync cycle. Set auto-stop to minimum if cost permits.
Materialization stuck in PENDING
Wait 30-60 seconds — this is normal during shard assignment. If still stuck:
flowctl logs --task <TENANT>/<PATH>/materialize-databricks --since 5m | jq 'select(.level == "error")'
Related Skills
schema-field-selection— Control which fields are materializedconnector-disable-enable— Pause/restart existing materializationsconnector-delete-recreate— Nuclear option for stuck materializationsestuary-logs— Deep log analysisestuary-catalog-status— Status checking
Files in this skill
1materialize-databricks-create skill questions
How do I install the materialize-databricks-create skill in Claude Code?
Run npx degit estuary/agent-skills/skills/materialize-databricks-create ~/.claude/skills/materialize-databricks-create. Claude Code loads skills from ~/.claude/skills (all projects) or .claude/skills (one project) and uses a skill when its description matches the task.
Which clients can use the materialize-databricks-create skill?
Any client that reads the Agent Skills format: Claude Code, Codex, Gemini CLI and claude.ai (upload as a ZIP). Scripts in the skill run only where the client allows code execution.
What are the prerequisites for Databricks materialization?
You need Unity Catalog enabled, a running SQL Warehouse, an existing target schema, and valid credentials (PAT or OAuth2 M2M).
How can I handle deleted rows in Databricks?
By default, deleted rows stay in Databricks marked with a metadata flag. You can enable hard deletes to physically remove them.
What should I do if I encounter a DELTA_UNSUPPORTED_DROP_COLUMN error?
You should enable column mapping mode on the affected table in Databricks using the ALTER TABLE command.
How long does the PENDING status typically last?
It is normal for the status to remain PENDING for approximately 30 to 60 seconds during shard assignment.