Skip to content

Fix service bus client lifecycle#4930

Open
JC-wk wants to merge 10 commits into
microsoft:mainfrom
JC-wk:fix-service-bus-client-lifecycle
Open

Fix service bus client lifecycle#4930
JC-wk wants to merge 10 commits into
microsoft:mainfrom
JC-wk:fix-service-bus-client-lifecycle

Conversation

@JC-wk

@JC-wk JC-wk commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

What is being addressed

inconsistent ServiceBusClient lifecycle management in deployment_status_updater.py, airlock_request_status_update.py, and runner.py.
key message listeners repeatedly instantiate ServiceBusClient inside infinite loops but discard the instances without calling close() or wrapping them in context managers. This can leak connection sockets and AMQP channels under transient reconnect loops.

How is this addressed

  • Prevents connection socket leaks: ServiceBusClient instances are now properly closed using async context managers
    Prevents AMQP channel leaks: Ensures channels are cleaned up even under transient reconnect scenarios
    Consistent pattern: All three files now follow the same best practice of using
  • Update CHANGELOG.md
  • Increment version

James Chapman added 2 commits June 8, 2026 15:53
…status_updater.py, airlock_request_status_update.py, and runner.py to prevent connection socket and AMQP channel leaks
…and deployment_status_updater.py for improved heartbeat logging and error handling; update test_runner.py to mock ServiceBusClient correctly; enhance runner.py with consistent exception handling and retry logic.
@JC-wk JC-wk requested a review from a team as a code owner June 8, 2026 16:20
Copilot AI review requested due to automatic review settings June 8, 2026 16:20
@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown

Unit Test Results

678 tests   678 ✅  7s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 44bb2b8.

♻️ This comment has been updated with latest results.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes inconsistent Azure Service Bus client lifecycle handling across the API service-bus listeners and the resource processor runner to reduce the risk of leaking sockets/AMQP channels during long-running polling and reconnect loops.

Changes:

  • Wrap ServiceBusClient usage in async context managers so clients are deterministically closed.
  • Add backoff (asyncio.sleep(10)) on connection/unknown exceptions in the runner and service-bus listeners.
  • Update runner unit tests to account for the async context manager usage; add a changelog entry.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
api_app/service_bus/deployment_status_updater.py Uses async with ServiceBusClient(...) to ensure the client is closed across reconnect cycles.
api_app/service_bus/airlock_request_status_update.py Uses async with ServiceBusClient(...) to ensure the client is closed; adjusts retry sleeps.
resource_processor/vmss_porter/runner.py Uses async with ServiceBusClient(...) for proper client teardown; adds retry sleeps on exceptions.
resource_processor/tests_rp/test_runner.py Updates mocks to handle the ServiceBusClient async context manager behavior.
CHANGELOG.md Adds an unreleased BUG FIXES entry for the lifecycle fix.

Comment thread api_app/service_bus/airlock_request_status_update.py Outdated
Comment thread CHANGELOG.md Outdated
Comment thread CHANGELOG.md
@rudolphjacksonm

Copy link
Copy Markdown
Collaborator

/test-extended

@github-actions

Copy link
Copy Markdown

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/28157965154 (with refid 93594175)

(in response to this comment from @rudolphjacksonm)

@maxmartin-cgi

Copy link
Copy Markdown
Collaborator

/test-extended

@github-actions

Copy link
Copy Markdown

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/28224500075 (with refid 93594175)

(in response to this comment from @maxmartin-cgi)

@maxmartin-cgi

Copy link
Copy Markdown
Collaborator

/test-extended

@github-actions

Copy link
Copy Markdown

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/28243761740 (with refid 93594175)

(in response to this comment from @maxmartin-cgi)

@maxmartin-cgi

Copy link
Copy Markdown
Collaborator

/test-extended

@github-actions

Copy link
Copy Markdown

🤖 pr-bot 🤖

⚠️ Cannot run tests as PR is not mergeable. Ensure that the PR is open and doesn't have any conflicts.

(in response to this comment from @maxmartin-cgi)

@marrobi marrobi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Opus 4.8


I found one correctness issue and one behavior risk:

  • In api_app/service_bus/airlock_request_status_update.py, ServiceBusClient is now created once outside the inner polling loop and reused forever. That means if the connection drops or the client becomes unhealthy after a transient failure, the code can keep retrying with the same stale client instead of recreating it. The previous pattern re-established the client each iteration; this change may reduce leaks, but it also makes recovery less robust. Consider moving the async with ServiceBusClient(...) back inside the retry loop, or explicitly recreating the client after connection-related failures.

  • In deployment_status_updater.py, the logic still keeps a single ServiceBusClient alive across the inner loop as well. If the intent is just to avoid leaks, that’s fine, but the current structure can hold the client open indefinitely while also suppressing reconnect churn. Make sure this is intentional and that there’s a clean exit/recreate path when the receiver or client encounters a fatal state.

  • resource_processor/vmss_porter/runner.py now wraps ServiceBusClient in async with, but the exception handlers only sleep and continue. If receive_message() ever exits due to a non-fatal error, the outer async with will close the client and then immediately reopen it on the next runner() invocation, which is fine. Just verify the tests cover the __aenter__/__aexit__ path for the new context-manager usage.

  • The changelog entry mentions all three files, but the behavioral change in airlock_request_status_update.py is materially different from the others: it changes reconnection semantics, not just lifecycle management. That should be called out more explicitly in the PR description or handled with a narrower refactor.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Comment thread api_app/service_bus/deployment_status_updater.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants