Skip to content

checks: report APIC database collection errors - #410

Open
monrog2 wants to merge 2 commits into
v4.2.0-devfrom
issue-327-dbsize-collection-errors
Open

checks: report APIC database collection errors#410
monrog2 wants to merge 2 commits into
v4.2.0-devfrom
issue-327-dbsize-collection-errors

Conversation

@monrog2

@monrog2 monrog2 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • prevent apic_database_size_check from reporting PASS when legacy mitmocounters files are missing, empty, unavailable, or malformed
  • retry each legacy counter read up to three times with a one-second delay to tolerate transient /debug population
  • capture and report the final command error instead of suppressing stderr
  • return structured ERROR details while preserving any oversized object-count findings already collected
  • move filtering and numeric top-four sorting into Python so shell pipelines cannot mask collection failures
  • add regression coverage for persistent and transient failures, partial collection, malformed rows, sorting, and the 1,500,000 threshold

Why

On affected pre-6.1(3a) releases, a missing mitmocounters/mo file caused cat to fail while the final sort process returned success. The check then received no rows and incorrectly reported PASS. Live 6.0(8e) testing also showed that the dynamic /debug files can be transiently unavailable even though they are subsequently readable by the same techsupport user.

Validation

  • focused tests: 17 passed
  • full local suite: 923 passed
  • internal Python 2.7 job: passed
  • internal Python 3.8 job: passed
  • live integration pipeline fix: Handle missing tver in SUP HW Rev check #168 produced 16 debug bundles
  • APIC 6.0(8e) bundle: all four legacy counter reads succeeded and the database-size check passed

The overall integration job remains red due to unrelated existing fabric/check failures.

Fixes #327

Report missing, empty, unavailable, or malformed mitmocounters data as an explicit error instead of passing incomplete collection. Preserve any oversized-class findings and add regression coverage for sorting and threshold behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@monrog2

monrog2 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Live validation findings on APIC 6.0(8e):

  • The updated check correctly stopped the previous false PASS and reported all four legacy DME counter reads as collection errors.
  • The generated debug bundle showed each direct cat /debug/<apic>/.../mitmocounters/mo returning exit status 1 within the same run.
  • Follow-up checks confirmed the paths are readable cross-APIC as both admin and the same techsupport account used by integration, ruling out a persistent path or permission problem.
  • The debug files are dynamically populated and were readable on retry, indicating transient /debug availability.

I am updating this PR to retry each legacy counter read a bounded number of times, capture the final command error instead of discarding stderr, and retain ERROR behavior when all attempts fail.

Retry each legacy mitmocounters read up to three times and capture the final command error when collection remains unavailable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@monrog2

monrog2 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Integration validation update:

  • Pipeline fix: Handle missing tver in SUP HW Rev check #168 ran commit 029d4d4 with debug-bundle collection enabled.
  • Python 2.7 and Python 3.8 jobs passed.
  • The integration job archived 16 .tgz bundles.
  • On the same APIC 6.0(8e fabric that previously returned exit status 1 for all four reads, the new bundle shows all four cat .../mitmocounters/mo 2>&1 commands succeeding on their first attempt.
  • apic_database_size_check.json reports ruleStatus: passed; no retry warnings were required in this run.

This validates the successful collection path in the live environment. Unit coverage separately verifies fail-once/succeed-on-retry and persistent-failure behavior. The overall integration job remains red for unrelated existing fabric/check failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant