fix(deploy): the lock_timeout prelude was invalid SQL and blocked every deploy - #371
Merged
Conversation
…ry deploy #368 shipped `SET lock_timeout = 5s;` — unquoted. Postgres answers "trailing junk after numeric literal", and under ON_ERROR_STOP=1 that failed the FIRST migration in the ladder. So a guard whose entire purpose was to stop a stalled session from blocking deploys blocked every deploy itself. Caught on its first real run; no DDL executed, because the prelude is the first statement in the session and the ladder stops there. Two things were wrong, and the second is the one worth keeping. 1. Quote the value. '5s' and '5000' are both accepted by Postgres; a bare 5s is a syntax error. Verified against the live server, not inferred. 2. A safety feature must not be able to brick the thing it protects. The prelude is now PROBED once, before the ladder runs. If the server rejects it — a typo'd MIGRATION_LOCK_TIMEOUT, a server that spells it differently — the runner says so loudly and applies migrations WITHOUT it. Degrading to the previous behaviour is survivable; an undeployable box is not. With the probe in place, the original unquoted bug would have produced a warning and a successful deploy instead of an outage in the deploy path. Why the tests missed it, and what changed so they cannot again: - `test_migrations_set_a_lock_timeout` asserted the string was PRESENT, not that it was valid SQL. Presence is not validity. `test_the_lock_timeout_value_is_ QUOTED` now parses the emitted statement and requires a quoted value. It scans only non-comment lines — the runner quotes the broken form in its own explanation of this bug, and a test that cannot tell an example from an instruction would fail on the documentation that prevents the next occurrence. - The shell harness's fake psql drained stdin without looking at it, so any SQL passed. It now mimics Postgres's actual rule and rejects a bare unit. Re-run with the original bug reintroduced, it reproduces "trailing junk" exactly — and the deploy still succeeds, via the new fallback. - `test_a_bad_prelude_degrades_instead_of_bricking_the_deploy` pins the fallback path itself, so a future edit cannot quietly make the guard fail closed again. Verify: uv run pytest tests/unit/test_stalled_session_cannot_freeze_the_db.py -q -> 12 passed uv run ruff check . --select F821,F601,F602,F502,F7,B006 -> All checks passed! bash -n on the runner -> OK Red-first confirmed: with the unquoted form reintroduced, test_the_lock_timeout_value_is_QUOTED fails and the fixed form passes. Behavioural, against a SQL-validating fake psql: valid prelude, clean apply -> exit 0, 1 attempt lock busy 2x then succeeds -> exit 0, 3 attempts lock busy past the retry budget -> exit 1, exactly 3 attempts MIGRATION_LOCK_TIMEOUT=nonsense -> exit 0, warns, applies without the timeout (the new fallback) No migration. No .env change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(deploy): the lock_timeout prelude was invalid SQL and blocked every deploy
#368 shipped
SET lock_timeout = 5s;— unquoted. Postgres answers "trailingjunk after numeric literal", and under ON_ERROR_STOP=1 that failed the FIRST
migration in the ladder. So a guard whose entire purpose was to stop a stalled
session from blocking deploys blocked every deploy itself. Caught on its first
real run; no DDL executed, because the prelude is the first statement in the
session and the ladder stops there.
Two things were wrong, and the second is the one worth keeping.
Quote the value. '5s' and '5000' are both accepted by Postgres; a bare 5s is
a syntax error. Verified against the live server, not inferred.
A safety feature must not be able to brick the thing it protects. The prelude
is now PROBED once, before the ladder runs. If the server rejects it — a
typo'd MIGRATION_LOCK_TIMEOUT, a server that spells it differently — the
runner says so loudly and applies migrations WITHOUT it. Degrading to the
previous behaviour is survivable; an undeployable box is not. With the probe
in place, the original unquoted bug would have produced a warning and a
successful deploy instead of an outage in the deploy path.
Why the tests missed it, and what changed so they cannot again:
test_migrations_set_a_lock_timeoutasserted the string was PRESENT, not thatit was valid SQL. Presence is not validity.
test_the_lock_timeout_value_is_ QUOTEDnow parses the emitted statement and requires a quoted value. It scansonly non-comment lines — the runner quotes the broken form in its own
explanation of this bug, and a test that cannot tell an example from an
instruction would fail on the documentation that prevents the next occurrence.
passed. It now mimics Postgres's actual rule and rejects a bare unit. Re-run
with the original bug reintroduced, it reproduces "trailing junk" exactly —
and the deploy still succeeds, via the new fallback.
test_a_bad_prelude_degrades_instead_of_bricking_the_deploypins the fallbackpath itself, so a future edit cannot quietly make the guard fail closed again.
Verify:
uv run pytest tests/unit/test_stalled_session_cannot_freeze_the_db.py -q
-> 12 passed
uv run ruff check . --select F821,F601,F602,F502,F7,B006 -> All checks passed!
bash -n on the runner -> OK
Red-first confirmed: with the unquoted form reintroduced,
test_the_lock_timeout_value_is_QUOTED fails and the fixed form passes.
Behavioural, against a SQL-validating fake psql:
valid prelude, clean apply -> exit 0, 1 attempt
lock busy 2x then succeeds -> exit 0, 3 attempts
lock busy past the retry budget -> exit 1, exactly 3 attempts
MIGRATION_LOCK_TIMEOUT=nonsense -> exit 0, warns, applies without the
timeout (the new fallback)
No migration. No .env change.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com