Skip to content

fix: notify Jira after queue retries exhaust#155

Open
iamdhrv wants to merge 2 commits into
forge-sdlc:mainfrom
iamdhrv:fix/140-terminal-jira-notification
Open

fix: notify Jira after queue retries exhaust#155
iamdhrv wants to merge 2 commits into
forge-sdlc:mainfrom
iamdhrv:fix/140-terminal-jira-notification

Conversation

@iamdhrv

@iamdhrv iamdhrv commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an optional terminal-failure callback to QueueConsumer and invoke it only after a message reaches the dead-letter queue
  • claim terminal handling atomically in Redis by event ID so multiple retry pollers cannot emit duplicate notifications
  • have the orchestrator post a secret-redacted Jira error comment with the ticket, event/correlation ID, and recovery instructions
  • check existing Jira comments for the event marker to preserve idempotency across ambiguous API failures or replays
  • contain callback and idempotency-store failures so dead-letter handling and stream acknowledgement continue

Closes #140

Test plan

  • uv run pytest tests/unit/queue/test_consumer_retry.py tests/unit/orchestrator/test_worker.py -q (55 passed)
  • uv run pytest tests/unit/ -q (1462 passed, 2 skipped)
  • uv run ruff check src/forge/queue/consumer.py src/forge/orchestrator/worker.py tests/unit/queue/test_consumer_retry.py tests/unit/orchestrator/test_worker.py
  • uv run ruff format --check src/forge/queue/consumer.py src/forge/orchestrator/worker.py tests/unit/queue/test_consumer_retry.py tests/unit/orchestrator/test_worker.py

Type checking

uv run mypy src/forge/queue/consumer.py src/forge/orchestrator/worker.py still reports the existing strict-mode errors in these modules (15 total); the new callback and Jira handler add no new reported mypy errors.

queued_for_retry = await self._retry_queue.enqueue_for_retry(
entry.message, str(e)
)
if not queued_for_retry:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job keeping notification failures from interrupting DLQ handling. Could we also acknowledge the original stream entry when queued_for_retry is false? This branch currently notifies and then continues, so terminal failures reached through the retry poller remain indefinitely in the Redis pending-entry list. A best-effort xack, regardless of whether the callback succeeds, plus a test for this path would keep the PEL clean.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 3f3036b. The retry-poller terminal path now performs a best-effort xack after the callback attempt, including when the callback fails. Added focused tests for both callback success and failure; the full unit suite and PR CI pass.

redis_client = await self._get_redis()
await redis_client.xack(stream, CONSUMER_GROUP, message_id)

async def _notify_terminal_failure(self, message: QueueMessage, error: str) -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the cross-worker notification claim here. One concern: this claims only the callback; the retry entry itself is still read non-atomically by every worker via get_due_messages(), so two pollers can process the same event concurrently. If one succeeds while another exhausts the counter, we could move a successfully processed event to the DLQ and post a misleading terminal-failure comment; concurrent failures can also advance the retry counter more quickly than intended. Could we atomically claim/lease the retry entry before dispatching it (for example, with a Redis transaction or Lua operation), and add a multi-worker test?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 3f3036b. Due retry entries are now atomically claimed with a Redis Lua operation that advances their score to a 15-minute visibility lease before dispatch. A crashed worker’s entry becomes claimable again after the lease. Added both a concurrent-poller unit test and a real-Redis integration test; the full unit suite and PR CI pass.

@eshulman2 eshulman2 added bug Something isn't working area:workflow Workflow graphs, orchestration, state, routing, and gates area:integrations Jira and other external service integrations tech-debt Internal maintainability or architectural debt ready-for-review labels Jul 16, 2026
@eshulman2
eshulman2 self-requested a review July 16, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:integrations Jira and other external service integrations area:workflow Workflow graphs, orchestration, state, routing, and gates bug Something isn't working ready-for-review tech-debt Internal maintainability or architectural debt

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tech debt: notify Jira when workflow execution exhausts queue retries

2 participants