Skip to content

OCE report: add Authenticator app telemetry and turn the skill into a router, Fixes AB#3731627 - #461

Open
Shahzaib (shahzaibj) wants to merge 1 commit into
shjameel-microsoft-oce-noise-gatefrom
shjameel-microsoft-oce-authapp-v2
Open

OCE report: add Authenticator app telemetry and turn the skill into a router, Fixes AB#3731627#461
Shahzaib (shahzaibj) wants to merge 1 commit into
shjameel-microsoft-oce-noise-gatefrom
shjameel-microsoft-oce-authapp-v2

Conversation

@shahzaibj

@shahzaibj Shahzaib (shahzaibj) commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Why

On-call engineers had Broker telemetry in a weekly report and Authenticator app health in a Kusto dashboard nobody opened during a rotation. The ask was explicit: one slash command, both reports — not two commands, and not one merged document.

What this does

SKILL.md becomes a thin router. It resolves the reporting window once, picks a mode, and dispatches. All Broker analysis moves verbatim into assets/playbooks/broker.md; the new assets/playbooks/authapp.md is its Authenticator counterpart.

Mode Produces
both (default) both reports + oce-index-<curEnd>.html
broker oncall-wow-report-<curEnd>.html
authapp authapp-wow-report-<curEnd>.html

Two reports, not one

The two apps have different owners, different triage ladders and different escalation paths. A merged report forces every reader through the half they don't own. The both mode instead emits two standalone reports plus a one-page index digest linking them.

The playbooks are never read into one context

In both mode they run as parallel sub-agents. This isn't only about wall-clock — their Kusto conventions are mutually incompatible:

  • Broker: HLL device counting; sum(countDevices) is actively wrong.
  • Authenticator: sum(SucceededDCount) is the correct idiom.

Interleaving them in one context risks writing one app's numbers under the other app's rules. The router says so explicitly, and the shared hard-rules section calls out that app-specific rules are never interchangeable.

Authenticator coverage

Scenario funnels (Passkey / Entra MFA / Entra PSI / MSA NGC+SA), error-reason decomposition, abandonment, Broker API responsiveness, version share, and an optional App Center crash layer (--skip-crashes, since it needs a secret).

Plumbing

bootstrap-report.ps1, run-kql.ps1, validate-report.ps1 and find-suspect-prs.ps1 all gain -App broker|authapp. validate-report.ps1 also gains the Authenticator check profile — which fills the check 12 slot deliberately reserved in the previous PR, so this validator diff is a pure insertion with no renumbering. find-suspect-prs.ps1 gains -Repos so it can scan the authenticator repo instead of broker/common.

New build-index.ps1 reads the headline KPI tiles out of both finished reports and emits the digest. It is a digest, not an analysis — a cross-app finding gets written into both reports, and the index just links them.

Scope

This PR is purely additive Authenticator support plus the router refactor. It contains no Broker behaviour changes — those are in the parent PR, and the window-alignment fix is in the child. That separation is the whole point of the split.

SKILL.md shrinks substantially because its Broker content moves to assets/playbooks/broker.md rather than being deleted.

Verification

Full E2E run in default both mode: both reports generated, both validators pass, index built. Reports land in %USERPROFILE%\android-oce-reports\ — outside the workspace, so they can't be committed by accident.

Stack

This is 2 of 3:

  1. OCE report: add novelty noise gate and redesign the Broker report around it, Fixes AB#3733390 #460 — novelty noise gate + Broker report redesign → master
  2. this PR — Authenticator app report + router → PR 1's branch
  3. OCE report: align the noise gate to the report's rolling window, Fixes AB#3731628 #462 — align the noise gate to the report's rolling window → this branch

Fixes AB#3731627

… router

One slash command now produces both weekly reports. Previously an on-call
engineer had Broker telemetry only; Authenticator app health lived in a Kusto
dashboard nobody opened during a rotation.

SKILL.md becomes a thin router: it resolves the reporting window ONCE, picks a
mode, and dispatches. All Broker analysis moves verbatim into
assets/playbooks/broker.md; the new assets/playbooks/authapp.md is its
Authenticator counterpart. The two playbooks are deliberately never read into
one context -- in `both` mode they run as parallel sub-agents, because their
Kusto conventions are mutually incompatible (Broker must never sum(countDevices)
and uses HLL; Authenticator's sum(SucceededDCount) is the correct idiom there).
Interleaving them risks writing one app's numbers with the other's rules.

Modes: `both` (default) | `broker` | `authapp`. Outputs land in
%USERPROFILE%\android-oce-reports\ as two separate reports plus, in both mode,
an oce-index-<curEnd>.html digest built by the new build-index.ps1. Two reports
rather than one merged document: the apps have different owners, different
triage ladders and different escalation paths, and a merged report forces every
reader through the half they do not own.

Authenticator coverage: scenario funnels (Passkey / Entra MFA / Entra PSI /
MSA NGC+SA), error-reason decomposition, abandonment, Broker API responsiveness,
version share, and an optional App Center crash layer (--skip-crashes).

bootstrap-report.ps1, run-kql.ps1, validate-report.ps1 and find-suspect-prs.ps1
gain -App broker|authapp (validate-report.ps1 also gains the Authenticator check
profile, filling the reserved check 12 slot); find-suspect-prs.ps1 gains -Repos
so it can scan the authenticator repo instead of broker/common.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@shahzaibj
Shahzaib (shahzaibj) requested a review from a team as a code owner August 20, 2026 01:27
@github-actions

Copy link
Copy Markdown

✅ Work item link check complete. Description contains link AB#3731627 to an Azure Boards work item.

@github-actions

Copy link
Copy Markdown

❌ Invalid work item number: AB#3731627
. Work item number must be a valid integer.

Click here to learn more.

@github-actions github-actions Bot changed the title OCE report: add Authenticator app telemetry and turn the skill into a router OCE report: add Authenticator app telemetry and turn the skill into a router, Fixes AB#3731627 Aug 20, 2026
@github-actions

Copy link
Copy Markdown

✅ Work item link check complete. Description contains link AB#3731627 to an Azure Boards work item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant