fix(email): stop holding a transaction across model and provider calls - #377
Merged
Conversation
The cause behind the cause. #368 BOUNDED the 2026-08-06 outage — a wall-clock ceiling on LLM calls, a 600s idle_in_transaction deadline, a lock_timeout on migrations. None of them stopped a session being parked in the first place. This removes the parking. `_maybe_classify_threads` (the Reply Zero backfill) is where the leaked session came from. The evidence pointed straight at it: every stalled backend's last statement was `SELECT org_domains FROM email_assistant_settings`, which is the final read of that function's setup block. What follows is a loop of up to _REPLY_DETERMINE_CAP = 40 `_mark_thread_replied` calls — each of which opens its OWN session and spends a full LLM determination in it, while the outer session sits `idle in transaction` holding ACCESS SHARE on everything it read. A migration's ALTER TABLE then queued behind that lock, and because Postgres's lock queue is FIFO every later reader of the table queued behind the waiting ALTER, including the `SELECT signature` the send path makes. Committing before the model call ends the transaction and releases the locks. The connection stays checked out; only the transaction closes. Worth stating plainly: the ceiling ALONE is not sufficient here, and after #368 this loop was arguably worse off. 40 capped calls idle far past the 600s deadline, at which point Postgres kills the session — and the `_upsert_thread_status` writes accumulated in the same loop, which did not commit until after it, would go with it. The backstop and this loop are only safe together. Two smaller instances of the same shape, both a network round-trip awaited with a read transaction open: - cleanup.py: `provider.authenticate()` after the account loader. - runner.py: `provider.authenticate()` after the credentials SELECT. Nothing is pending at either point, so the added commit only ends the transaction. Named, not fixed: the backfill's second loop awaits `classify_matches(db, ...)`, which does its own reads and so re-opens a transaction around its model call. Fixing that means changing classify_matches itself rather than its caller, and it is bounded by the acb_llm ceiling. Same for the sweep's per-message `apply_label`, whose provider call is bounded by httpx at 30s. The test asserts ORDER, not presence: a commit AFTER the model call would satisfy a "was commit called" check while leaving the lock held for the whole call. Red-first confirmed — remove the commit and it fails on the first model call. Verify: uv run pytest tests/unit/test_email_reply_zero.py -q -> 32 passed uv run pytest tests/unit -q -k email -> 979 passed, 1 skipped uv run ruff check . --select F821,F601,F602,F502,F7,B006 -> All checks passed! uv run xenon --max-absolute F --max-modules F --max-average B apps packages -> 0 No migration. No .env change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(email): stop holding a transaction across model and provider calls
The cause behind the cause. #368 BOUNDED the 2026-08-06 outage — a wall-clock
ceiling on LLM calls, a 600s idle_in_transaction deadline, a lock_timeout on
migrations. None of them stopped a session being parked in the first place.
This removes the parking.
_maybe_classify_threads(the Reply Zero backfill) is where the leaked sessioncame from. The evidence pointed straight at it: every stalled backend's last
statement was
SELECT org_domains FROM email_assistant_settings, which is thefinal read of that function's setup block. What follows is a loop of up to
_REPLY_DETERMINE_CAP = 40
_mark_thread_repliedcalls — each of which opens itsOWN session and spends a full LLM determination in it, while the outer session
sits
idle in transactionholding ACCESS SHARE on everything it read. Amigration's ALTER TABLE then queued behind that lock, and because Postgres's
lock queue is FIFO every later reader of the table queued behind the waiting
ALTER, including the
SELECT signaturethe send path makes.Committing before the model call ends the transaction and releases the locks.
The connection stays checked out; only the transaction closes.
Worth stating plainly: the ceiling ALONE is not sufficient here, and after #368
this loop was arguably worse off. 40 capped calls idle far past the 600s
deadline, at which point Postgres kills the session — and the
_upsert_thread_statuswrites accumulated in the same loop, which did notcommit until after it, would go with it. The backstop and this loop are only
safe together.
Two smaller instances of the same shape, both a network round-trip awaited with
a read transaction open:
provider.authenticate()after the account loader.provider.authenticate()after the credentials SELECT.Nothing is pending at either point, so the added commit only ends the
transaction.
Named, not fixed: the backfill's second loop awaits
classify_matches(db, ...),which does its own reads and so re-opens a transaction around its model call.
Fixing that means changing classify_matches itself rather than its caller, and
it is bounded by the acb_llm ceiling. Same for the sweep's per-message
apply_label, whose provider call is bounded by httpx at 30s.The test asserts ORDER, not presence: a commit AFTER the model call would
satisfy a "was commit called" check while leaving the lock held for the whole
call. Red-first confirmed — remove the commit and it fails on the first model
call.
Verify:
uv run pytest tests/unit/test_email_reply_zero.py -q -> 32 passed
uv run pytest tests/unit -q -k email -> 979 passed, 1 skipped
uv run ruff check . --select F821,F601,F602,F502,F7,B006 -> All checks passed!
uv run xenon --max-absolute F --max-modules F --max-average B apps packages -> 0
No migration. No .env change.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com