[v2-rebuild] databricks-pack — community design review (6 skills, feedback wanted before code)

[jeremylongshore]

Context

The databricks-pack plugin in this marketplace is being rebuilt from the ground up. The v1 was thin documentation; v2 is a working plugin pack with live workspace access — six skills, one shared MCP server, real detection logic against real workloads.

Every design decision is on the table here before code lands. I have catalogued 34 production failure modes from secondary research (Reddit, GitHub issues, Databricks community forum, vendor docs) and ranked them into six skills. The research is solid for what gets formally reported. It is not solid for what gets cursed at silently — and that is the input I am asking this community to provide.

The honest stance: I can build any of this. I am not the one running production Databricks workloads. If you tell me which failure modes you would actually want this to catch, which detection thresholds will fire correctly at 2 AM versus on every healthy workspace, and which skill designs senior data engineers will actually reach for, I will build what you say — not what I would guess. Implementation is not the rate-limiting input; calibrated priority signal is.

The six skills

#	Skill	What it catches	Issue	Status
1	databricks-workspace-mcp	Typed control-plane access (clusters.events, instance pools, pipeline event logs, external locations, storage credentials)	#789	🟡 design open
2	databricks-cost-leak-hunter	Ranked real-dollar cost leaks from system.billing.usage + live workspace state (pilot — start here)	#790	🟡 design open
3	databricks-cluster-forensics	Cold starts, launch failures, Photon fallback, DBR-upgrade triage from live cluster events	#791	🟡 design open
4	databricks-streaming-guardian	Delta + Liquid Clustering + Structured Streaming + DLT operations (the largest skill, twelve failure modes)	#792	🟡 design open
5	databricks-uc-migration-pilot	Unity Catalog readiness, IAM/SCIM diagnostics, system-table access tracing (Sept 30 2026 deadline driver)	#793	🟡 design open
6	databricks-bundle-medic	Asset Bundles deploy diagnostics, CMK rotation, PrivateLink endpoint audit	#794	🟡 design open

Start here if you only have 10 minutes

Read #790 — databricks-cost-leak-hunter first. It is the pilot skill — the first thing in the rebuilt pack a user will actually run, designed to be FinOps-grokkable on the first read. The four cost categories it catches are the ones I would most want validated against your real workloads, and the design questions inside are the ones I am most uncertain about. If you only leave thumbs-up / thumbs-down on the question bullets there, you have already calibrated the most important skill in the pack.

How to engage

• Comment on any issue with thoughts, corrections, or alternative approaches.
• Thumbs-up / thumbs-down on individual design-question bullets is equally valuable — that is pure priority signal and takes 30 seconds.
• Voice memo via WhatsApp works for anyone in Luciano's community who would prefer to speak instead of type in English — I will transcribe back into the relevant issue with credit. Portuguese is fine.

What this is not

• Not a code review request. There is no code yet — by design.
• Not a sales pitch. The pack will ship under the same marketplace license as every other plugin here.
• Not a request for unpaid consulting hours. Any depth of engagement is fine; no engagement is also fine.

Why public

Every catalogued failure mode is already public (forum threads, GitHub issues, vendor docs). The implementation is the novelty, not the design — so the design can live in the open and the people who would benefit can shape it before code lands. Full transparency was deliberate; if any of you would have made a different call, please say so in this thread.

Reference material

All design records, pain research, and pressure tests live in plugins/saas-packs/databricks-pack/000-docs/ and are linked individually below. Index file: 000-INDEX.md.

Pain research (RL-RSRC)

The 34 catalogued failure modes, grouped by Databricks domain. These are the inputs that drove which skills got built and at what threshold.

Doc	What it covers
002-RL-RSRC	Domain 1 — Compute / Photon / DBR versioning / cost
003-RL-RSRC	Domain 2 — Delta Lake, Liquid Clustering, Structured Streaming, DLT
004-RL-RSRC	Domain 3 — Unity Catalog, Asset Bundles, identity, workspace ops, secrets, networking
005-RL-RSRC	Architectural patterns extracted from Anthropic-published Claude Code skills (2026)
006-RL-RSRC	v2-rebuild synthesis — failure modes → six skills mapping
010-RL-RSRC	Databricks MCP landscape (official, managed)
011-RL-RSRC	Databricks MCP community landscape (2026)
012-RL-RSRC	Claude Code / Cowork ↔ Databricks MCP integration architecture

Architecture decisions (AT-ADEC)

Locked design decisions with reasoning. These are the load-bearing records — if you disagree, this is where to push back.

Doc	What it locks
007-AT-ADEC	CTO decision — Databricks Pack v2 rebuild (REVISED)
013-AT-ADEC	Epic 1 MCP scope adjustment — auth flows, dual-MCP design, scope cut from 8 → 6 endpoints

Pressure tests (RA-REVW)

Adversarial validation of the decisions above.

Doc	What it stress-tests
008-RA-REVW	Pack-handling pressure test — version bump in place vs. fresh pack
009-RA-REVW	Pilot-timing pressure test — databricks-cost-leak-hunter as pilot, 3-5 day window

Direct reading is welcome but not required. The issue bodies summarize what is needed for response.

• Jeremy Longshore
  intentsolutions.io

---

Internal tracking (beads for agent context — not for external action):

• [v2-rebuild] databricks-workspace-mcp: typed control-plane access for the five v2 skills #789 workspace-mcp · claude-koxw
• [v2-rebuild] databricks-cost-leak-hunter: ranked real-dollar cost leaks from live workspace data #790 cost-leak-hunter · claude-02m1
• [v2-rebuild] databricks-cluster-forensics: cold-start, launch-failure, Photon, DBR-upgrade triage from live cluster events #791 cluster-forensics · claude-hsoc
• [v2-rebuild] databricks-streaming-guardian: Delta / Liquid Clustering / Structured Streaming / DLT operations skill #792 streaming-guardian · claude-vjaw
• [v2-rebuild] databricks-uc-migration-pilot: Unity Catalog readiness, IAM/SCIM diagnostics, system-table access tracing #793 uc-migration-pilot · claude-h53a
• [v2-rebuild] databricks-bundle-medic: Asset Bundles deploy diagnostics, CMK rotation, PrivateLink endpoint audit #794 bundle-medic · claude-jhnj

[Gingiris-1031]

Interesting architecture call packaging six skills under one shared MCP server. A few thoughts from having shipped a multi-skill pack myself:

On the shared workspace-mcp approach (#789)

Making it the dependency layer for the other five skills is clean in theory, but watch out for the "one broken auth = all skills dead" failure mode. In my experience building skill collections, the skills that get adopted fastest are ones that work independently — even if that means some redundant setup. Users who only care about cost leaks shouldn't need to configure cluster-events access.

On the "calibrated priority signal" ask

The most honest feedback I can give: cost-leak-hunter as the pilot is the right move. It's the only one where the output is immediately legible to non-infra people (finance/FinOps teams see dollar signs, not log lines). If you can make it produce a one-page "here's $X/month you're wasting" output in under 30 seconds, that's your adoption hook — everything else follows.

On skill-level architecture patterns

If it's useful, I maintain a collection of 11 production skills at github.com/Gingiris-1031/gingiris-skills — different domain (growth/marketing playbooks, not infra), but the structural patterns around SKILL.md frontmatter, inter-skill references, and "install one or install all" packaging might save you some trial-and-error on the scaffolding side.

One specific pattern that worked well: each skill declares its own description and triggers in frontmatter so discovery tools can route to it independently, even when installed as part of a bundle. Avoids the "monolith plugin that does 6 things but you can't find the one you need" problem.

[jeremylongshore]

Yipei, thanks for swinging by, and for the depth.

For the public thread context since the Gingiris alt doesn't say it: Yipei Wei, cofounder / ex-COO of Toeverything (AFFiNE, approaching 70K stars on toeverything/AFFiNE), Forbes 30 Under 30 Asia 2024. That's a serious resume to land in a community-design-review thread, and the comment matched the resume. Appreciated.

Want to push back on the shared-MCP framing though, because I think it conflates two patterns that look the same from the outside.

Bundled monolith means "one plugin does 6 things, can't find the one you want." Agreed: terrible UX, engineers reject it on sight. Shared MCP server means "one typed dependency that N skills consume." Different pattern. We're shipping the second, not the first.

On the install path: each v2 skill is its own SKILL.md with its own description + tags in frontmatter (point 4 in your comment, already the marketplace-tier required-frontmatter rubric here). A user who only wants cost-leak detection gets that one skill, no cluster-events handler pulled in. The pack-install path bundles all six for convenience, not because they're welded together. The MCP underneath is plumbing, not bundling. Invisible to the user, addressable per-tool by the skills that need it.

On "one broken auth = all skills dead": the risk assumes a single credential surface. The actual design is two MCPs, not one. Databricks' managed SQL MCP serves system.* reads (cost data, query history, streaming progress). The custom workspace MCP serves cluster events, pipelines, storage credentials, external locations. They authenticate independently. Losing access to one does not disable the other. cost-leak-hunter and cluster-forensics fail independently when their respective MCP is unavailable. Scope boundary is logged at 013-AT-ADEC-epic1-mcp-scope-adjustment.md.

On the cost-leak-hunter pilot pick and the "one-page $/month output in 30 seconds" as the adoption hook: that's the framing the pilot was designed around. The FinOps-grokkable deliverable IS the conversion moment, and every detection check underneath it serves that one output. Reading externally the same way it reads internally is the right signal.

If you want to play further 😉

1. A FinOps-grokkable cost-leak draft. Would you read one and tell me whether it reads to a CFO without rewriting? The AFFiNE README pattern already does this. Would the same framing carry for a CFO audience here?
2. How Toeverything handled the OSS to commercial line. Not asking for commercial detail. Just the framing of when paid lines got drawn and how the community took it. This pack ships MIT; the partner-SaaS variant question is open for Q3 2026.

DM @asphaltcowb0y on X or comment here, whichever works.

• Jeremy Longshore
  intentsolutions.io