fix(db): one wedged handler could freeze the whole database — three bounds

[vjvarada]

The incident

Sending mail stopped working on 2026-08-06. It was reported as "I can't send emails — is it the multi-user work?" It was not an auth or scoping regression: the send path's ownership and feature gates were intact. It was a database freeze, and email was simply the first surface anyone noticed.

Verified live on the box with pg_blocking_pids:

pid 1208980  idle in transaction, 14h44m
             SELECT org_domains FROM email_assistant_settings WHERE account_id = $1
                 ↓ holds ACCESS SHARE
pid 1210326  ALTER TABLE email_assistant_settings ADD COLUMN IF NOT EXISTS ...   (waiting)
                 ↓ ACCESS EXCLUSIVE parks at the head of a FIFO queue
pid 1209232, 1209228, 1216158, ...  every later reader of that table, indefinitely

The chain:

1. An LLM completion hung — acompletion was awaited with no wall-clock bound.
2. The caller held a SQLAlchemy AsyncSession across that await. A session opens a transaction on first execute() and holds it until commit/rollback/close, so the hung call parked a Postgres transaction idle in transaction for 14h44m.
3. The migration runner asked for ACCESS EXCLUSIVE on that table and waited. Postgres's lock queue is FIFO, so every later reader queued behind the waiting ALTER — including SELECT signature FROM email_assistant_settings, which the send path makes on its way to the provider.
4. Each blocked reader pinned a pooled connection until the email engine's 10+20 pool drained, and unrelated endpoints started answering QueuePool limit of size 10 overflow 20 reached.

The BFF aborts the send POST at 30s, so the user-visible symptom was a failed send with zero POST /email/send lines in three days of gateway logs — the request never completed, so uvicorn never logged it. Absence of the access line was the symptom.

Terminating the stale backend cleared it: the blocked query went from indefinite to 1.5ms. A new session took the same lock six minutes later and the freeze reproduced identically. That is what makes this a code fix rather than an operational one.

The fix — three independent bounds

Any one of them breaks the chain. All three are here because there was no bound at all, and the cheapest way for this to come back is for someone to restore one link while the other two still look fine.

Link	Bound	Where
Unbounded await	asyncio.wait_for at 90s per attempt (LLM_REQUEST_TIMEOUT_SECS)	acb_llm/client.py
Unbounded held transaction	idle_in_transaction_session_timeout=600s via asyncpg server_settings	gateway/db.py + 7 engines, email_ingestion
Migration waits for a lock	SET lock_timeout=5s + 5 bounded retries	the migration runner

Three details worth reviewing rather than skimming:

• except TimeoutError sits ABOVE the generic handler. asyncio.wait_for's TimeoutError stringifies to '', so the existing any(token in str(exc).lower() ...) transient check reads it as non-transient and re-raises. Getting the order wrong would convert one slow provider response into a hard failure on attempt 1 — the fix becoming a new, quieter outage. A test pins it.
• 600s, not 300s. A rules run legitimately awaits a completion with a session open, and complete retries 3×90s + 6s backoff ≈ 276s. If the DB ceiling dropped under that, Postgres would start killing healthy transactions mid-retry, which reads as random unattributable failures under load. test_idle_ceiling_clears_the_llm_worst_case fails if someone raises one number without the other.
• The idle ceiling is scoped to the app's own connections, not set via ALTER ROLE. pg_dump and the migration runner connect as the same role and must not inherit an app-tuned deadline.

test_every_gateway_engine_carries_the_bounds is a structural guard: per-site opt-in is exactly how the email engine ended up as the one that drained.

Verification

uv run pytest tests/unit/test_stalled_session_cannot_freeze_the_db.py -q   -> 10 passed
uv run pytest tests/unit -q -k email                                       -> 973 passed, 1 skipped
uv run pytest tests/unit -q -k "llm or db or engine or connect or migration"
    -> 410 passed, 1 failed (test_workflows_engine::test_module_node_runs_generated_code,
       PRE-EXISTING — red on a clean stashed tree too)
uv run ruff check . --select F821,F601,F602,F502,F7,B006                   -> All checks passed!
uv run xenon --max-absolute F --max-modules F --max-average B apps packages -> exit 0

Not just green — confirmed red without the fix. With the asyncio.wait_for removed, both LLM tests fail (in bounded time; each carries an outer CI guard so a broken build fails rather than hangs). The retry loop was exercised against a fake psql:

case	result
clean apply	exit 0, 1 attempt
lock busy 2x then succeeds	exit 0, 3 attempts, retries logged
lock busy forever	exit 1 after exactly 3, prints the pg_stat_activity query that finds the holder
real SQL error	exit 1 after exactly 1 — no retry, real error surfaced

Where multi-user came in

As the amplifier, not the bug. A second mailbox (Ishaanpilar@fracktal.in) was connected 2026-08-05 08:56 UTC, doubling every per-account background loop against pools sized when there was one mailbox. The sweep for the new account ran seconds after the lock was taken. Suggestive, not proven — pg_stat_activity does not expose the bound account_id.

Not in this PR

• The email automation package holds DB sessions across provider and LLM calls throughout. That is now bounded, not removed. Restructuring those transaction boundaries is its own change.
• acb-pull.service was exiting 128 (git fetch -q origin release) on repeated ticks around 06:42–06:58 UTC. It has since recovered on its own — last successful pull 07:20:49Z, box at 8d83ca10. Worth a look at why it flapped, but it is not blocking this deploy.
• The unit file acb-pull.service lives only on the box, not in the repo, so the concurrent-tick hazard in it (a fixed /tmp/acb-pull-run.sh that a later tick can overwrite while an earlier one is still executing it) cannot be fixed here. Two overlapping apply_migrations.sh runs were observed during the incident, so the flock in vps_pull.sh is not covering everything it looks like it covers.
• The leak is still live on the deployed code. Re-checked after opening this PR: the same resolve_org_domains session was 20 minutes idle in transaction with a migration and two readers queued behind it. Cleared again by hand. Until this merges, the box stays one long LLM hang away from a repeat.

No migration. No .env change required — both new knobs have working defaults.

🤖 Generated with Claude Code