[v2-rebuild] databricks-bundle-medic: Asset Bundles deploy diagnostics, CMK rotation, PrivateLink endpoint audit

[jeremylongshore]

What this skill does

databricks-bundle-medic is the deploy and infrastructure skill of the pack. It covers the Databricks Asset Bundles surface (which is immature tooling — the replacement for the deprecated dbx, with a moving bug list across CLI versions) plus the deploy-time infrastructure operations that bite production teams when they touch encryption keys, networking topology, or workspace-level configuration that compute interacts with at boot.

What it catches

• databricks bundle bind missing UC catalog and external location support — teams with existing UC resources cannot bring them under DAB management without editing terraform.tfstate by hand or destroying and recreating the resource. Open at Bundle bind does not support Unity Catalog resources (catalogs, external locations) databricks/cli#4842 as of CLI v0.295.x. (004-RL-RSRC D4)
• "unexpected EOF reading terraform.tfstate" on every redeploy after the first — bug in the CLI's workspace-files streaming read path. Workaround: DATABRICKS_BUNDLE_ENGINE=direct (officially preview, in practice adopted permanently). Open at bundle deploy fails with "unexpected EOF" reading terraform.tfstate on every deploy after initial databricks/cli#4986. (004-RL-RSRC D5)
• Schema GRANT ordering bug — DAB's resource graph does not encode the dependency that GRANTs supplying privileges needed at resource-creation time must apply before the resources that exercise those privileges. First deploy fails with "User does not have CREATE TABLE on Schema"; second deploy succeeds because the first applied the grants before failing. Open at Schema-Grants need to run before pipeline deployment databricks/cli#4573. (004-RL-RSRC D6)
• CMK rotation requires terminating every cluster, pool, and SQL warehouse in the workspace — a hard maintenance window that multi-team 24x7 workspaces cannot easily accommodate. (004-RL-RSRC D8)
• PrivateLink + VPC endpoints trap — enabling PrivateLink for the Databricks control plane is mistaken for "all traffic is private now," when S3/STS/Kinesis traffic still traverses the NAT and racks up cross-AZ + NAT processing charges until separate VPC endpoints are configured per service. (004-RL-RSRC D9)

Design questions I want pushback on

1. PreToolUse hook on databricks bundle deploy. Plan is: download the remote terraform.tfstate, validate it parses as JSON, warn if size has shrunk vs. last known good copy, cache a known-good copy locally as a recovery escape hatch. Right behavior, or too cautious?
2. PostToolUse auto-retry on D6. On the specific "User does not have CREATE TABLE on Schema" stderr pattern, auto-retry once. Plan is to fire ONLY on this exact known-transient failure, never on others. Right call, or risky?
3. databricks bundle bind workaround scope. D4 — should the skill ship a terraform.tfstate editor (powerful but dangerous), or only refuse to proceed and direct the user to wait for upstream fix (safe but unhelpful)?
4. CMK rotation runbook. D8 needs a workspace-drain sequence. Plan is to enumerate all clusters/pools/warehouses, ask for confirmation, terminate in dependency order, run rotation, restart. Should the skill do the termination, or only produce the sequence as a runbook the user runs manually?
5. PrivateLink audit detection. D9 is the easiest to detect (enumerate VPC endpoints, check for S3/STS/Kinesis coverage) but the report needs cost math. Are the AWS cost-explorer integrations worth the auth complexity, or is "here are the missing endpoints, expected NAT cost is X" enough?

What I am not asking about right now

• Whether to support dbx migration — out of scope, dbx is deprecated.
• Whether to handle Azure-specific networking — Azure ExpressRoute and PrivateLink Service work is acknowledged but deferred to v3.
• Whether to add Terraform state import features — Terraform-side problems live in the Terraform provider's issue tracker.

How to respond

Comment below with any thoughts, leave thumbs-up / thumbs-down on individual bullets in the design questions, or send a voice memo on WhatsApp and I will transcribe it into the issue with attribution. English is not required for voice memos — Portuguese is fine.

Source bead: claude-jhnj in the local beads workspace.

---

Reference material

Most relevant for this skill:

Doc	What it covers
004-RL-RSRC	Unity Catalog / Asset Bundles / identity / workspace ops / secrets / networking pain catalog

Full reference set + cross-skill context: see umbrella issue #795 § Reference material.

• Jeremy Longshore
  intentsolutions.io