Skip to content

fix(deployment): make reconciliation leases durable - #205

Merged
alongubkin merged 2 commits into
alon/alien-320-harden-aws-cloudformation-byoc-release-pathfrom
alon/alien-324-make-deployment-leases-cancellable-and-keep-short-controller
Jul 25, 2026
Merged

fix(deployment): make reconciliation leases durable#205
alongubkin merged 2 commits into
alon/alien-320-harden-aws-cloudformation-byoc-release-pathfrom
alon/alien-324-make-deployment-leases-cancellable-and-keep-short-controller

Conversation

@alongubkin

@alongubkin alongubkin commented Jul 25, 2026

Copy link
Copy Markdown
Member

Summary

  • persist delayed reconciliation wakeups instead of repeatedly reacquiring work that is not ready
  • keep eligible controller work under one bounded lease and run independent ready resources concurrently
  • renew ownership through a dedicated endpoint without rewriting deployment state
  • cancel promptly when ownership is lost and clear scheduled delays for explicit user actions

Why

A fresh multi-resource deployment exposed two framework failures: short controller waits were repeatedly yielded through the shared scheduler, and manager replicas could remain occupied retrying work after lease loss. This preserves durable checkpoints while avoiding unnecessary distributed scheduling between adjacent controller steps.

Validation

  • cargo check -p alien-infra -p alien-deployment -p alien-manager --all-features
  • 851 tests across alien-infra, alien-deployment, and alien-manager passed (1 skipped)
  • generated OpenAPI artifacts were produced through the repository generators

Dependency

Stacked on #179.

Linear: https://linear.app/alienplatform/issue/ALIEN-324/make-deployment-leases-cancellable-and-keep-short-controller

Persist delayed wakeups, keep eligible controller work under the active lease, renew ownership without rewriting deployment state, and stop execution promptly when ownership is lost.
@greptile-apps

greptile-apps Bot commented Jul 25, 2026

Copy link
Copy Markdown

Greptile Summary

The PR makes deployment reconciliation leases durable and cancellable.

  • Persists delayed reconciliation wakeups and atomically rechecks wakeup eligibility when acquiring work.
  • Adds a dedicated lease-renewal endpoint and transport support without rewriting deployment state.
  • Runs independent ready resources concurrently under a bounded lease and stops work promptly after ownership loss.
  • Clears scheduled delays for explicit deploy, retry, redeploy, and destroy actions.
  • Updates SQLite persistence, reconciliation callers, tests, and generated OpenAPI specifications.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the lock-acquisition update now atomically rechecks the persisted wakeup predicate before claiming a deployment.

Important Files Changed

Filename Overview
crates/alien-manager/src/stores/sqlite/deployment.rs Persists reconciliation wakeups and rechecks their eligibility in the atomic lock-acquisition update, resolving the previously reported race.
crates/alien-manager/src/loops/deployment.rs Coordinates bounded lease ownership, renewal, cancellation, and deployment-loop execution.
crates/alien-deployment/src/runner.rs Extends shared reconciliation-loop behavior for durable delays and prompt cancellation.
crates/alien-infra/src/core/executor.rs Executes independent ready resource controllers concurrently within the configured bound.
crates/alien-manager/src/routes/sync.rs Adds the authenticated lease-renewal API path and request handling.
crates/alien-manager/tests/sqlite_store_tests.rs Expands SQLite store coverage for durable scheduling and lease behavior.

Reviews (2): Last reviewed commit: "fix(manager): recheck delayed work durin..." | Re-trigger Greptile

Repeat wakeup eligibility in the atomic lock update so a concurrent checkpoint cannot schedule work after candidate selection and still be acquired early.
@alongubkin
alongubkin merged commit 02acc4f into alon/alien-320-harden-aws-cloudformation-byoc-release-path Jul 25, 2026
14 of 15 checks passed
@alongubkin
alongubkin deleted the alon/alien-324-make-deployment-leases-cancellable-and-keep-short-controller branch July 25, 2026 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant