Added APIC OOB connectivity checks for CSCwu91693 - #418
Conversation
asraf-khan
left a comment
asraf-khan
left a comment
There was a problem hiding this comment.
Make it as single function check for 2 different validation with help of helper inside same function.
Done. Merged both validations into a single check function |
|
|
||
| ### APIC OOB Connectivity | ||
|
|
||
| Due to [CSCwu91693][77], when an APIC cluster upgrade is triggered, the orchestrating APIC fans out an HTTPS POST to every peer APIC over the OOB management network. If OOB connectivity to any peer APIC is broken at upgrade time, only the reachable APICs receive the trigger and start upgrading. The unreachable APICs are silently skipped, leaving the cluster partially upgraded — a state that cannot be recovered remotely. |
There was a problem hiding this comment.
Apic bootx use https POST for upgrade starting from 602. Due to [CSCwu91693][77], If OOB connectivity to any peer APIC is broken during upgrade, only the reachable APICs receive the trigger and start upgrading. The unreachable APICs are silently skipped, leaving the cluster partially upgraded.
|
|
||
| This check performs two verifications: | ||
|
|
||
| 1. **Default port (443)**: Used by APIC bootx starting from 6.0(2). Applicable when target version is 6.0(2) or above. |
There was a problem hiding this comment.
- Default port (443): APIC used bootx starting from 6.0(2). default https port used is 443.
remote all AI marked special symbol.
| This check performs two verifications: | ||
|
|
||
| 1. **Default port (443)**: Used by APIC bootx starting from 6.0(2). Applicable when target version is 6.0(2) or above. | ||
| 2. **Custom HTTPS port (from `commHttps`)**: Used by the upgrade fanout starting from 6.2(1). Applicable when both current and target versions are 6.2(1) or above. If the configured port is the same as the default (443), this step is skipped as it is already covered above. |
There was a problem hiding this comment.
- Custom HTTPS port (from
commHttps): starting from 6.2(1) Customer ports are supported. Applicable when both current and target versions are 6.2(1) or above.
| if subprocess.call( | ||
| 'curl --max-time 5 -k -s -o /dev/null https://{}:{} 2>/dev/null'.format(ip, port), | ||
| shell=True | ||
| ) in [7, 28]: |
There was a problem hiding this comment.
anything as !0 should be tracked as failure.
| has_error = True | ||
|
|
||
| # Custom HTTPS port check: upgrade fanout uses commHttps port from 6.2(1) | ||
| if not (cversion.older_than("6.2(1a)") or tversion.older_than("6.2(1a)")): |
There was a problem hiding this comment.
if (cversion.newer_than("6.2(1a)"). target version can be ignored.
There was a problem hiding this comment.
Updated to if not cversion.older_than "6.2(1a)".
| @@ -0,0 +1,32 @@ | |||
| [ | |||
There was a problem hiding this comment.
missing testcase for both ipv6. please add that as well and test in lab.
3 participants
Summary:
This PR adds a new validation check: APIC OOB Connectivity.
The check detects broken OOB management connectivity between APIC controllers, which can cause a partial or failed cluster upgrade.
What Changed:
Check Behavior:
APIC OOB Connectivity:
Test Results:
Pytest Logs: Full Run:
CSCwu91693_PytestFullRun_logs.txt
Pytest Logs: apic_oob_connectivity_check:
CSCwu91693_PytestLogs.txt
Script Full Run:
CSCwu91693_FullRun_logs.txt
Note: stale_dbgacEpgSummaryTask_check shows an ERROR (name 'timedelta' is not defined) in the full run, this is a pre-existing issue in v4.2.0-dev, not introduced by this PR. It will be resolved by Muthu's separate PR.
Script Run Logs (PASS / NA / FAIL):
CSCwu91693_PASS:NA:FAIL_logs.txt