Skip to content

direct: write deployment state atomically - #6329

Merged
denik merged 2 commits into
mainfrom
denik/dstate-atomic-save
Aug 20, 2026
Merged

direct: write deployment state atomically#6329
denik merged 2 commits into
mainfrom
denik/dstate-atomic-save

Conversation

@denik

@denik denik commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

unlockedSave truncated the state file in place, so a crash mid-write left a state file the CLI cannot parse next to an intact WAL. Open parses the state file before it stats the WAL and returns on error, so recovery never ran and the deployment state was unrecoverable.

Write to a temp file in the same directory and rename it over the state file instead. replayWAL already removes the WAL only after the save succeeds.

unlockedSave truncated the state file in place. A crash mid-write left a
state file the CLI cannot parse next to an intact WAL that Open never
reads, because Open parses the state file first and returns on error. The
deployment state was then unrecoverable.

Write to a temp file in the same directory and rename it over the state
file instead; replayWAL already removes the WAL only after the save
succeeds.

Co-authored-by: Isaac
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: f485b01

Run: 32354726517

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 1 4 274 1162 6:30
💚​ aws windows 1 4 276 1160 3:43
💚​ azure linux 1 4 273 1162 6:08
💚​ azure windows 1 4 275 1160 4:12
💚​ gcp linux 1 4 274 1162 7:26
💚​ gcp windows 1 4 276 1160 4:47
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:31 gcp windows TestAccept
3:44 aws linux TestAccept
3:43 azure linux TestAccept
3:42 gcp linux TestAccept
3:29 aws windows TestAccept
3:03 azure windows TestAccept

The test pinned how the state file is written (replaced, not truncated in
place) rather than what callers observe. What matters is that the save
lands; the mechanism is an implementation detail explained in a comment.

Co-authored-by: Isaac
@denik
denik added this pull request to the merge queue Aug 20, 2026
// only then removes the WAL, and Open parses the state file before it looks at
// the WAL. A torn write would therefore leave a state file that Open rejects
// next to an intact WAL it never reads.
func (db *DeploymentState) unlockedSave() error {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a unit test for the failure paths would be good to assert this actually works

@@ -0,0 +1 @@
Write the deployment state atomically so an interrupted save cannot leave a state file that the CLI refuses to read.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR link

Merged via the queue into main with commit c718024 Aug 20, 2026
28 checks passed
@denik
denik deleted the denik/dstate-atomic-save branch August 20, 2026 10:14
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: c718024

Run: 32358011816

Env ❌​FAIL 🟨​KNOWN 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
❌​ aws windows 12 2 1 2 1093 984 228:09
❌​ azure windows 12 2 1 2 987 1021 168:24
❌​ gcp windows 12 2 1 2 977 1025 192:47
17 interesting tests: 12 FAIL, 2 KNOWN, 2 SKIP, 1 RECOVERED
Test Name aws windows azure windows gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/apps/job_permissions ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/apps/job_permissions/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/invariant/continue_293 ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/invariant/continue_293/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=app.yml.tmpl ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/invariant/migrate ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/invariant/migrate/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=app.yml.tmpl ❌​F ❌​F ❌​F
🟨​ TestAccept/bundle/invariant/no_drift 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=app.yml.tmpl/READPLAN= ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=app.yml.tmpl/READPLAN=1 ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/apps/lifecycle-started-omitted ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/apps/lifecycle-started-toggle ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ❌​F
💚​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S
Top 50 slowest tests (at least 2 minutes):
duration env testname
31:54 aws windows TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
25:22 aws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
22:46 aws windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
15:18 aws windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
15:18 aws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
14:08 gcp windows TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
11:43 aws windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
9:42 aws windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
9:21 aws windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
8:48 aws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
8:45 azure windows TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:42 aws windows TestAccept/bundle/resources/clusters/resize-terminated-fallback/DATABRICKS_BUNDLE_ENGINE=direct
8:22 aws windows TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
8:17 aws windows TestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct
8:16 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
8:15 gcp windows TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct
8:10 aws windows TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
8:00 aws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
7:51 azure windows TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
7:47 aws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
7:44 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
7:44 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
7:42 aws windows TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
7:12 aws windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
7:11 aws windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
7:05 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct
6:16 azure windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
5:02 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:47 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:44 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct
3:55 gcp windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
3:53 gcp windows TestAccept/bundle/quiet-levels/DATABRICKS_BUNDLE_ENGINE=terraform
3:52 gcp windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct
3:46 gcp windows TestAccept/bundle/resources/registered_models/basic/DATABRICKS_BUNDLE_ENGINE=terraform
3:35 aws windows TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
3:35 aws windows TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:33 gcp windows TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct
3:31 gcp windows TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.12
3:28 aws windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes/READPLAN=
3:25 gcp windows TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9
3:25 azure windows TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=yes/READPLAN=
3:23 aws windows TestAccept/bundle/resources/registered_models/basic/DATABRICKS_BUNDLE_ENGINE=terraform
3:23 azure windows TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform
3:21 azure windows TestAccept/bundle/quiet-levels/DATABRICKS_BUNDLE_ENGINE=terraform
3:21 aws windows TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=yes/READPLAN=
3:16 gcp windows TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.13
3:12 aws windows TestAccept/bundle/resources/postgres_endpoints/update_autoscaling/DATABRICKS_BUNDLE_ENGINE=direct
3:10 gcp windows TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no/READPLAN=
3:07 gcp windows TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
3:06 gcp windows TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes/READPLAN=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants