Added check for certificate expiration issue

[asraf-khan]

Detection logic:

1. Find the fault using below query
   'faultInst.json?query-target-filter=or(eq(faultInst.code,"F4501"),eq(faultInst.code,"F4502"),eq(faultInst.code,"F4503"),eq(faultInst.code,"F3081"),eq(faultInst.code,"F3082"),eq(faultInst.code,"F4617"),eq(faultInst.code,"F4752"),eq(faultInst.code,"F4753"))'

2. if one fault F4501, F3081, F4617, F4752 found and state is “raised” or “soaking”
   Result = Manual & recommended_action ="Renew the certificate(s) before it expires to avoid service disruption."

3. if one fault F4502, F4503, F3082, F4753 and state is “raised” or “soaking”
   Result = FAIL_O & recommended_action="Renew the certificate(s) immediately to restore functionality."

4. if faults found as combination (e.g. expiring F3081 & expired F4502 ) means
   Result = FAIL_O & recommended_action="Renew expired certificate(s) immediately. For certificate(s) approaching expiry, renew before they expire to avoid service disruption."

Faults            Short Description                    Introduced         First build
F3081, F3082      SAML encryption cert expiring/expired     2018-02-22        3.1(2f)
F4501, F4502      KeyRing cert expiring/expired             2023-09-13        6.0(4c)
F4503, F4617      TP cert expired/expiring                  2024-04-22        6.1(1e)
F4752,F4753 Factory cert expiring/expired 6.1(5e)

For Factory certificate status if cversion < 6.1(5e), check via cli command "acidiag verifyapic"

[asraf-khan]

Attached log for reference.

APIC Log :

APIC Run log - SSL Cert expiry issue.docx

Pytest Log:

Pytest log - SSL Cert expiry issue.docx

[asraf-khan]

Attaching latest APIC & Pytest logs

APIC Logs:

APIC Run log - SSL Cert expiry issue.docx

Pytest Logs:

Pytest log - SSL Cert expiry issue.docx

[thjonson]

This is great — thanks for adding fault-based cert expiration detection. One
gap I noticed: this approach only surfaces an issue once ACI has already
raised the fault (F4501/F4502/etc.), which means the earliest signal is
whatever threshold ACI's fault logic uses internally, and it doesn't cover
node SSL certs the way pkiFabricNodeSSLCertificate does.

As a complementary check (not a replacement), I've been running a script
that queries pkiFabricNodeSSLCertificate directly (joined with
fabricNode for name/model/serial/role) and computes days-to-expiry per
node, independent of whether a fault has fired yet:

• EXPIRED — already past validityNotAfter
• CRITICAL — expires within ~90 days
• WARN — expires within ~365 days (well ahead of any fault threshold)
• Flags certs not chained to "Cisco Manufacturing CA"
• Flags notAfter in year 2099, which matches FN-72339 (renewed-SUDI
  signature) — not something the fault codes above catch, since it isn't a
  fault condition, just a suspicious cert attribute worth a manual review.

Since fabric node SSL cert expiry/re-validation failures are a distinct
failure mode from the SAML/KeyRing/TP/Factory certs this PR covers (nodes
stay Inactive on reboot if their SSL cert fails re-validation), would it
make sense to add this as an additional check in this same PR, or should it
be a separate follow-on check? Happy to share the reference script/logic if
useful.

[asraf-khan]

@thjonson Thanks for sharing your suggesstions.

In this PR itself added validation for leaf/spine certificate expiration using MO's "pkiFabricNodeSSLCertificate" irrespective of faults and alert the customer if it's expired and expiring (expires within ~30 days) which matched the days with expired & expiring fault codes and we maintain same with faults expiring & MO's based expiring.

can you please provide details for below item ? how to verify ? what is the impact of this customer will face ? what is the recommended_action we have to provide to customer if they face this ?

Flags notAfter in year 2099, which matches FN-72339 (renewed-SUDI
signature) — not something the fault codes above catch, since it isn't a
fault condition, just a suspicious cert attribute worth a manual review.