Skip to content

implement cooperative heartbeat-based locking for rolling deployments - #203

Open
dembrane-sam-bot wants to merge 1 commit into
mainfrom
sam/update-heartbeat-locking
Open

implement cooperative heartbeat-based locking for rolling deployments#203
dembrane-sam-bot wants to merge 1 commit into
mainfrom
sam/update-heartbeat-locking

Conversation

@dembrane-sam-bot

Copy link
Copy Markdown
Contributor
  • What is this change?: Implements heartbeat-based file locking and cooperative standby for Cloud Run rolling deployments to prevent split-brain duplicates during startup/deployment.
  • What did Sam notice that led to this?: Rolling updates spin up a new container while the old container is still draining. The new container's _recover_from_reactions immediately re-queued in-flight tasks (left at :eyes: or :hourglass: by the draining container), creating duplicate sessions and GCS write collisions.
  • Tier?: Tier 3 (runtime)
  • Confidence?: 100% (passes all 565 runtime tests + 6 brand new heartbeat-lock tests + 45 structural eval tests).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant