Skip to content

fix(email): stop holding a transaction across model and provider calls - #377

Merged
vjvarada merged 1 commit into
mainfrom
fix/txn-across-llm-calls
Aug 6, 2026
Merged

fix(email): stop holding a transaction across model and provider calls#377
vjvarada merged 1 commit into
mainfrom
fix/txn-across-llm-calls

Conversation

@vjvarada

@vjvarada vjvarada commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

fix(email): stop holding a transaction across model and provider calls

The cause behind the cause. #368 BOUNDED the 2026-08-06 outage — a wall-clock
ceiling on LLM calls, a 600s idle_in_transaction deadline, a lock_timeout on
migrations. None of them stopped a session being parked in the first place.
This removes the parking.

_maybe_classify_threads (the Reply Zero backfill) is where the leaked session
came from. The evidence pointed straight at it: every stalled backend's last
statement was SELECT org_domains FROM email_assistant_settings, which is the
final read of that function's setup block. What follows is a loop of up to
_REPLY_DETERMINE_CAP = 40 _mark_thread_replied calls — each of which opens its
OWN session and spends a full LLM determination in it, while the outer session
sits idle in transaction holding ACCESS SHARE on everything it read. A
migration's ALTER TABLE then queued behind that lock, and because Postgres's
lock queue is FIFO every later reader of the table queued behind the waiting
ALTER, including the SELECT signature the send path makes.

Committing before the model call ends the transaction and releases the locks.
The connection stays checked out; only the transaction closes.

Worth stating plainly: the ceiling ALONE is not sufficient here, and after #368
this loop was arguably worse off. 40 capped calls idle far past the 600s
deadline, at which point Postgres kills the session — and the
_upsert_thread_status writes accumulated in the same loop, which did not
commit until after it, would go with it. The backstop and this loop are only
safe together.

Two smaller instances of the same shape, both a network round-trip awaited with
a read transaction open:

  • cleanup.py: provider.authenticate() after the account loader.
  • runner.py: provider.authenticate() after the credentials SELECT.
    Nothing is pending at either point, so the added commit only ends the
    transaction.

Named, not fixed: the backfill's second loop awaits classify_matches(db, ...),
which does its own reads and so re-opens a transaction around its model call.
Fixing that means changing classify_matches itself rather than its caller, and
it is bounded by the acb_llm ceiling. Same for the sweep's per-message
apply_label, whose provider call is bounded by httpx at 30s.

The test asserts ORDER, not presence: a commit AFTER the model call would
satisfy a "was commit called" check while leaving the lock held for the whole
call. Red-first confirmed — remove the commit and it fails on the first model
call.

Verify:
uv run pytest tests/unit/test_email_reply_zero.py -q -> 32 passed
uv run pytest tests/unit -q -k email -> 979 passed, 1 skipped
uv run ruff check . --select F821,F601,F602,F502,F7,B006 -> All checks passed!
uv run xenon --max-absolute F --max-modules F --max-average B apps packages -> 0

No migration. No .env change.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

The cause behind the cause. #368 BOUNDED the 2026-08-06 outage — a wall-clock
ceiling on LLM calls, a 600s idle_in_transaction deadline, a lock_timeout on
migrations. None of them stopped a session being parked in the first place.
This removes the parking.

`_maybe_classify_threads` (the Reply Zero backfill) is where the leaked session
came from. The evidence pointed straight at it: every stalled backend's last
statement was `SELECT org_domains FROM email_assistant_settings`, which is the
final read of that function's setup block. What follows is a loop of up to
_REPLY_DETERMINE_CAP = 40 `_mark_thread_replied` calls — each of which opens its
OWN session and spends a full LLM determination in it, while the outer session
sits `idle in transaction` holding ACCESS SHARE on everything it read. A
migration's ALTER TABLE then queued behind that lock, and because Postgres's
lock queue is FIFO every later reader of the table queued behind the waiting
ALTER, including the `SELECT signature` the send path makes.

Committing before the model call ends the transaction and releases the locks.
The connection stays checked out; only the transaction closes.

Worth stating plainly: the ceiling ALONE is not sufficient here, and after #368
this loop was arguably worse off. 40 capped calls idle far past the 600s
deadline, at which point Postgres kills the session — and the
`_upsert_thread_status` writes accumulated in the same loop, which did not
commit until after it, would go with it. The backstop and this loop are only
safe together.

Two smaller instances of the same shape, both a network round-trip awaited with
a read transaction open:
- cleanup.py: `provider.authenticate()` after the account loader.
- runner.py:  `provider.authenticate()` after the credentials SELECT.
Nothing is pending at either point, so the added commit only ends the
transaction.

Named, not fixed: the backfill's second loop awaits `classify_matches(db, ...)`,
which does its own reads and so re-opens a transaction around its model call.
Fixing that means changing classify_matches itself rather than its caller, and
it is bounded by the acb_llm ceiling. Same for the sweep's per-message
`apply_label`, whose provider call is bounded by httpx at 30s.

The test asserts ORDER, not presence: a commit AFTER the model call would
satisfy a "was commit called" check while leaving the lock held for the whole
call. Red-first confirmed — remove the commit and it fails on the first model
call.

Verify:
  uv run pytest tests/unit/test_email_reply_zero.py -q     -> 32 passed
  uv run pytest tests/unit -q -k email                     -> 979 passed, 1 skipped
  uv run ruff check . --select F821,F601,F602,F502,F7,B006 -> All checks passed!
  uv run xenon --max-absolute F --max-modules F --max-average B apps packages -> 0

No migration. No .env change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vjvarada
vjvarada merged commit 73ab190 into main Aug 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant