Fix service bus client lifecycle#4930
Conversation
…status_updater.py, airlock_request_status_update.py, and runner.py to prevent connection socket and AMQP channel leaks
…and deployment_status_updater.py for improved heartbeat logging and error handling; update test_runner.py to mock ServiceBusClient correctly; enhance runner.py with consistent exception handling and retry logic.
Unit Test Results678 tests 678 ✅ 7s ⏱️ Results for commit 44bb2b8. ♻️ This comment has been updated with latest results. |
There was a problem hiding this comment.
Pull request overview
This PR fixes inconsistent Azure Service Bus client lifecycle handling across the API service-bus listeners and the resource processor runner to reduce the risk of leaking sockets/AMQP channels during long-running polling and reconnect loops.
Changes:
- Wrap
ServiceBusClientusage in async context managers so clients are deterministically closed. - Add backoff (
asyncio.sleep(10)) on connection/unknown exceptions in the runner and service-bus listeners. - Update runner unit tests to account for the async context manager usage; add a changelog entry.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
api_app/service_bus/deployment_status_updater.py |
Uses async with ServiceBusClient(...) to ensure the client is closed across reconnect cycles. |
api_app/service_bus/airlock_request_status_update.py |
Uses async with ServiceBusClient(...) to ensure the client is closed; adjusts retry sleeps. |
resource_processor/vmss_porter/runner.py |
Uses async with ServiceBusClient(...) for proper client teardown; adds retry sleeps on exceptions. |
resource_processor/tests_rp/test_runner.py |
Updates mocks to handle the ServiceBusClient async context manager behavior. |
CHANGELOG.md |
Adds an unreleased BUG FIXES entry for the lifecycle fix. |
…0.25.17 and 0.13.4; refine debug logging in airlock_request_status_update.py
|
/test-extended |
|
🤖 pr-bot 🤖 🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/28157965154 (with refid (in response to this comment from @rudolphjacksonm) |
|
/test-extended |
|
🤖 pr-bot 🤖 🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/28224500075 (with refid (in response to this comment from @maxmartin-cgi) |
|
/test-extended |
|
🤖 pr-bot 🤖 🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/28243761740 (with refid (in response to this comment from @maxmartin-cgi) |
|
/test-extended |
|
🤖 pr-bot 🤖 (in response to this comment from @maxmartin-cgi) |
marrobi
left a comment
There was a problem hiding this comment.
From Opus 4.8
I found one correctness issue and one behavior risk:
-
In
api_app/service_bus/airlock_request_status_update.py,ServiceBusClientis now created once outside the inner polling loop and reused forever. That means if the connection drops or the client becomes unhealthy after a transient failure, the code can keep retrying with the same stale client instead of recreating it. The previous pattern re-established the client each iteration; this change may reduce leaks, but it also makes recovery less robust. Consider moving theasync with ServiceBusClient(...)back inside the retry loop, or explicitly recreating the client after connection-related failures. -
In
deployment_status_updater.py, the logic still keeps a singleServiceBusClientalive across the inner loop as well. If the intent is just to avoid leaks, that’s fine, but the current structure can hold the client open indefinitely while also suppressing reconnect churn. Make sure this is intentional and that there’s a clean exit/recreate path when the receiver or client encounters a fatal state. -
resource_processor/vmss_porter/runner.pynow wrapsServiceBusClientinasync with, but the exception handlers only sleep and continue. Ifreceive_message()ever exits due to a non-fatal error, the outerasync withwill close the client and then immediately reopen it on the nextrunner()invocation, which is fine. Just verify the tests cover the__aenter__/__aexit__path for the new context-manager usage. -
The changelog entry mentions all three files, but the behavioral change in
airlock_request_status_update.pyis materially different from the others: it changes reconnection semantics, not just lifecycle management. That should be called out more explicitly in the PR description or handled with a narrower refactor.
What is being addressed
inconsistent ServiceBusClient lifecycle management in deployment_status_updater.py, airlock_request_status_update.py, and runner.py.
key message listeners repeatedly instantiate ServiceBusClient inside infinite loops but discard the instances without calling close() or wrapping them in context managers. This can leak connection sockets and AMQP channels under transient reconnect loops.
How is this addressed
Prevents AMQP channel leaks: Ensures channels are cleaned up even under transient reconnect scenarios
Consistent pattern: All three files now follow the same best practice of using