Skip to content

fix(deploy): the lock_timeout prelude was invalid SQL and blocked every deploy - #371

Merged
vjvarada merged 1 commit into
mainfrom
fix/lock-timeout-syntax
Aug 6, 2026
Merged

fix(deploy): the lock_timeout prelude was invalid SQL and blocked every deploy#371
vjvarada merged 1 commit into
mainfrom
fix/lock-timeout-syntax

Conversation

@vjvarada

@vjvarada vjvarada commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

fix(deploy): the lock_timeout prelude was invalid SQL and blocked every deploy

#368 shipped SET lock_timeout = 5s; — unquoted. Postgres answers "trailing
junk after numeric literal", and under ON_ERROR_STOP=1 that failed the FIRST
migration in the ladder. So a guard whose entire purpose was to stop a stalled
session from blocking deploys blocked every deploy itself. Caught on its first
real run; no DDL executed, because the prelude is the first statement in the
session and the ladder stops there.

Two things were wrong, and the second is the one worth keeping.

  1. Quote the value. '5s' and '5000' are both accepted by Postgres; a bare 5s is
    a syntax error. Verified against the live server, not inferred.

  2. A safety feature must not be able to brick the thing it protects. The prelude
    is now PROBED once, before the ladder runs. If the server rejects it — a
    typo'd MIGRATION_LOCK_TIMEOUT, a server that spells it differently — the
    runner says so loudly and applies migrations WITHOUT it. Degrading to the
    previous behaviour is survivable; an undeployable box is not. With the probe
    in place, the original unquoted bug would have produced a warning and a
    successful deploy instead of an outage in the deploy path.

Why the tests missed it, and what changed so they cannot again:

  • test_migrations_set_a_lock_timeout asserted the string was PRESENT, not that
    it was valid SQL. Presence is not validity. test_the_lock_timeout_value_is_ QUOTED now parses the emitted statement and requires a quoted value. It scans
    only non-comment lines — the runner quotes the broken form in its own
    explanation of this bug, and a test that cannot tell an example from an
    instruction would fail on the documentation that prevents the next occurrence.
  • The shell harness's fake psql drained stdin without looking at it, so any SQL
    passed. It now mimics Postgres's actual rule and rejects a bare unit. Re-run
    with the original bug reintroduced, it reproduces "trailing junk" exactly —
    and the deploy still succeeds, via the new fallback.
  • test_a_bad_prelude_degrades_instead_of_bricking_the_deploy pins the fallback
    path itself, so a future edit cannot quietly make the guard fail closed again.

Verify:
uv run pytest tests/unit/test_stalled_session_cannot_freeze_the_db.py -q
-> 12 passed
uv run ruff check . --select F821,F601,F602,F502,F7,B006 -> All checks passed!
bash -n on the runner -> OK

Red-first confirmed: with the unquoted form reintroduced,
test_the_lock_timeout_value_is_QUOTED fails and the fixed form passes.

Behavioural, against a SQL-validating fake psql:
valid prelude, clean apply -> exit 0, 1 attempt
lock busy 2x then succeeds -> exit 0, 3 attempts
lock busy past the retry budget -> exit 1, exactly 3 attempts
MIGRATION_LOCK_TIMEOUT=nonsense -> exit 0, warns, applies without the
timeout (the new fallback)

No migration. No .env change.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

…ry deploy

#368 shipped `SET lock_timeout = 5s;` — unquoted. Postgres answers "trailing
junk after numeric literal", and under ON_ERROR_STOP=1 that failed the FIRST
migration in the ladder. So a guard whose entire purpose was to stop a stalled
session from blocking deploys blocked every deploy itself. Caught on its first
real run; no DDL executed, because the prelude is the first statement in the
session and the ladder stops there.

Two things were wrong, and the second is the one worth keeping.

1. Quote the value. '5s' and '5000' are both accepted by Postgres; a bare 5s is
   a syntax error. Verified against the live server, not inferred.

2. A safety feature must not be able to brick the thing it protects. The prelude
   is now PROBED once, before the ladder runs. If the server rejects it — a
   typo'd MIGRATION_LOCK_TIMEOUT, a server that spells it differently — the
   runner says so loudly and applies migrations WITHOUT it. Degrading to the
   previous behaviour is survivable; an undeployable box is not. With the probe
   in place, the original unquoted bug would have produced a warning and a
   successful deploy instead of an outage in the deploy path.

Why the tests missed it, and what changed so they cannot again:

- `test_migrations_set_a_lock_timeout` asserted the string was PRESENT, not that
  it was valid SQL. Presence is not validity. `test_the_lock_timeout_value_is_
  QUOTED` now parses the emitted statement and requires a quoted value. It scans
  only non-comment lines — the runner quotes the broken form in its own
  explanation of this bug, and a test that cannot tell an example from an
  instruction would fail on the documentation that prevents the next occurrence.
- The shell harness's fake psql drained stdin without looking at it, so any SQL
  passed. It now mimics Postgres's actual rule and rejects a bare unit. Re-run
  with the original bug reintroduced, it reproduces "trailing junk" exactly —
  and the deploy still succeeds, via the new fallback.
- `test_a_bad_prelude_degrades_instead_of_bricking_the_deploy` pins the fallback
  path itself, so a future edit cannot quietly make the guard fail closed again.

Verify:
  uv run pytest tests/unit/test_stalled_session_cannot_freeze_the_db.py -q
      -> 12 passed
  uv run ruff check . --select F821,F601,F602,F502,F7,B006  -> All checks passed!
  bash -n on the runner                                     -> OK

  Red-first confirmed: with the unquoted form reintroduced,
  test_the_lock_timeout_value_is_QUOTED fails and the fixed form passes.

  Behavioural, against a SQL-validating fake psql:
    valid prelude, clean apply         -> exit 0, 1 attempt
    lock busy 2x then succeeds         -> exit 0, 3 attempts
    lock busy past the retry budget    -> exit 1, exactly 3 attempts
    MIGRATION_LOCK_TIMEOUT=nonsense    -> exit 0, warns, applies without the
                                          timeout (the new fallback)

No migration. No .env change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vjvarada
vjvarada merged commit 04278d4 into main Aug 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant