Add Authenticator app report to the OCE weekly telemetry skill, Fixes AB#3731627 - #455
Open
Shahzaib (shahzaibj) wants to merge 2 commits into
Open
Add Authenticator app report to the OCE weekly telemetry skill, Fixes AB#3731627#455Shahzaib (shahzaibj) wants to merge 2 commits into
Shahzaib (shahzaibj) wants to merge 2 commits into
Conversation
|
❌ Work item link check failed. Description does not contain AB#{ID}. Click here to Learn more. |
Extends oncall-weekly-telemetry-report from a Broker-only skill into a router that produces two reports plus a combined index from one slash command, and reworks how findings are surfaced so on-call engineers can tell a new regression from an ongoing known issue at a glance. Authenticator support - SKILL.md is now a thin router: resolve window and mode, then delegate. Modes: both (default), broker, authapp. - App analysis moves into assets/playbooks/broker.md and authapp.md. The two apps' Kusto conventions are incompatible (Broker forbids sum(countDevices); AuthApp requires sum(*DCount)), so the playbooks run as parallel sub-agents and are never read into one context. - Adds an 8-query AuthApp pack, a Kusto cheatsheet, an AuthApp report template, an index template, and build-index.ps1. - bootstrap-report.ps1, run-kql.ps1, validate-report.ps1 and find-suspect-prs.ps1 take -App to select cluster, template and profile. Noise reduction The previous report ranked the attention section by volume, so a flat but huge error code outranked a real step change, and 60-day trends were a browsable catalog. A real run had 13 attention rows with zero charts next to them, while 100 sparklines sat in tables below - 27 of 29 trend rows duplicated content already shown elsewhere. - classify-novelty.js labels each series against its own 7-week baseline (NEW / ACCELERATING / ONGOING / VOLATILE / RECOVERY / IMPROVING / STABLE) from the existing bucket-trends sidecar, so no new queries. attention = NEW + ACCELERATING; everything else collapses into a fold. - Attention section is grouped New this week / Getting worse / Ongoing (folded), budgeted to 8 visible rows including at most 2 wins, and every visible row carries its own sparkline. - 60-day section becomes a detector: only promoted slow burns render outside the fold. - VOLATILE and RECOVERY rows drop the WoW chip for a vs-60d-median chip. A code once reported +398% WoW while sitting 94% below its own median. - Quiet weeks publish short rather than backfilling filler. Validation - Adds checks 13-18: material lead-row delta, row-body specificity, no ratio artifacts in chips, a sparkline on every visible attention row, the visible-row budget, and an app-aware cap on unfolded 60-day charts (6 Broker / 16 AuthApp - AuthApp's scenario table is the scoreboard). - Verified by generating both reports with sub-agents given only the playbooks and no hints about the redesign. Both passed every hard check: attention rows 13 -> 7 (Broker) and 4 (AuthApp), all charted, unfolded 60-day charts 38 -> 0. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
First full run of the router as an engineer invokes it: mode both, one resolved window, two parallel sub-agents, combined index. Both reports passed every hard check. The agents were given no hints about the report shape, and were asked to report anything they had to guess at - that question surfaced six real documentation defects. Broker - Coverage vs budget read as a contradiction: Step 5 mandates an attribution card for every regressed code and type, while Section 2 caps visible rows at 8. They govern different things. Both sides now say so and cross-reference: the budget caps visibility, the coverage rule caps omission, and surplus cards go in a collapsed fold rather than being dropped. 12 mandated cards with 7 visible rows is correct. - The two WoW bases were described but their division of labour was not. Adds a table: every number a reader sees comes from the rolling window; the calendar-week classifier contributes selection and narrative, never a figure. Authenticator - fetch-appcenter-crashes.js was documented as "--days 14". The script lives in the sibling release-monitoring-report skill and needs a subcommand plus --owner/--app/--version, so it could not be run as written. Replaced with the real invocation, the token resolution order, and a note that skipping the crash layer is expected. - The novelty step said to classify "13 scenario success-rate series" while the same paragraph excludes push notifications. Only 9 outcome funnels are classifiable; 13 = 9 outcome funnels + 4 PN families. - The scoreboard requires success/failure/unknown columns for all 13, but PN has no such model and Denied is a healthy outcome. PN rows now use completion/error with a literal n/a for unknown, never blank. - run-kql.ps1's signature was never stated on the AuthApp side. It takes -Query with the query TEXT and -Out, not -File/-OutFile. Verified: before-fixture still fails checks 15/16/18; both new reports pass; index links resolve; both reports agree on the window. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Shahzaib (shahzaibj)
force-pushed
the
shjameel-microsoft-oce-authapp-report
branch
from
August 18, 2026 07:20
362f0cb to
58d3485
Compare
Shahzaib (shahzaibj)
changed the title
|
✅ Work item link check complete. Description contains link AB#3731627 to an Azure Boards work item. |
|
❌ Invalid work item number: AB#3731627 Click here to learn more. |
github-actions
Bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends the
oncall-weekly-telemetry-reportskill so one slash-command invocation produces theweekly telemetry report for both the Android Broker and the Authenticator app.
Previously the skill covered Broker only. On-call engineers had no equivalent view of Authenticator
scenario health, and the obvious alternative — a second skill — would have meant two commands and
two chances to run them against different windows.
How
Two reports, one command, one window.
both(default)oncall-wow-report-<date>.html+authapp-wow-report-<date>.html+oce-index-<date>.htmlbrokerauthappThe reports stay separate files rather than one merged document — the two apps have different
audiences, different diagnostic ladders, and (critically) incompatible Kusto conventions. A
combined index page links them and surfaces both sets of headline KPIs.
SKILL.mdbecomes a router. It now only resolves the reporting window, picks the mode, andstitches results together. All app-specific analysis moved into two playbooks:
assets/playbooks/broker.md— error codes/types, spike + code attribution, latency, version adoptionassets/playbooks/authapp.md— scenario funnels (Passkey / Entra MFA / Entra PSI / MSA NGC+SA),error reasons, abandonment, Broker API responsiveness, crash rate
In
bothmode the playbooks run as parallel sub-agents, so two reports cost roughly thewall-clock of one. They are deliberately never read into the same context — the Broker's
"never
sum(countDevices), always HLL" rule is actively wrong on the Authenticator side, wheresum(SucceededDCount)is correct. Keeping them isolated prevents cross-contamination.Noise gating. New
classify-novelty.jslabels every key NEW / ACCELERATING / ONGOING /VOLATILE / RECOVERY / IMPROVING / STABLE against its own 7-week baseline. Only
NEW+ACCELERATING(plus at most 2 wins) render with charts; everything else collapses into a fold.This replaces the previous volume-ranked attention list, where a flat-but-huge error code
outranked a genuine step change and buried it.
New assets
assets/playbooks/{broker,authapp}.mdassets/scripts/classify-novelty.js,assets/scripts/build-index.ps1assets/templates/{authapp-report-template,index-template}.htmlassets/queries/authapp/**+assets/docs/authapp-kusto-cheatsheet.mdbootstrap-report.ps1,run-kql.ps1,validate-report.ps1,find-suspect-prs.ps1all gainedan
-App broker|authappswitchTesting
Run end-to-end via the slash command against live Kusto for the 2026-08-01 window. Both reports
generated and passed their own
validate-report.ps1 -App <app>profiles; the combined index builtfrom the two finished reports. AuthApp queries were validated against the live cluster
(
idsharedeus2.eastus2.kusto.windows.net) before being committed.Notes
$env:USERPROFILE\android-oce-reports\— outside the workspace, sothey can never be committed accidentally. Nothing generated is included here.
.github/skills/oncall-weekly-telemetry-report/. No product code touched.(it graded Sunday-aligned calendar weeks while the report displays a rolling 7-day window).
That fix is split out for reviewability and targets this branch:
#459 — it will auto-retarget to
masteronce this merges.Fixes AB#3731627