diff --git a/.github/skills/oncall-weekly-telemetry-report/SKILL.md b/.github/skills/oncall-weekly-telemetry-report/SKILL.md index 3c661051..37d2654a 100644 --- a/.github/skills/oncall-weekly-telemetry-report/SKILL.md +++ b/.github/skills/oncall-weekly-telemetry-report/SKILL.md @@ -1,42 +1,53 @@ --- name: oncall-weekly-telemetry-report -description: Generate the weekly Android Broker on-call (OCE) WoW + 60-day trend telemetry report as a polished self-contained HTML file. Use this skill for the weekly OCE rotation when asked to "produce the OCE report", "weekly on-call report", "WoW telemetry report", "weekly broker health report", or "generate this week's on-call summary". Pulls from `android_spans` materialized views, attributes regressions/improvements to PRs in `broker/` and `common/`, and writes to `$env:USERPROFILE\android-oce-reports\oncall-wow-report-YYYY-MM-DD.html` (outside the workspace so reports are never committed). +description: Generate the weekly Android on-call (OCE) WoW + 60-day trend telemetry reports as polished self-contained HTML. Produces TWO reports by default — the Android Broker report (from `android_spans` materialized views) and the Authenticator app report (from the Authenticator scenario materialized views) — plus a combined index page. Use this skill for the weekly OCE rotation when asked to "produce the OCE report", "weekly on-call report", "WoW telemetry report", "weekly broker health report", "authenticator weekly report", "authapp telemetry report", or "generate this week's on-call summary". Supports modes `both` (default), `broker`, and `authapp`. Writes to `$env:USERPROFILE\android-oce-reports\` (outside the workspace so reports are never committed). --- -# OCE Weekly Report +# OCE Weekly Report — router -Produce the weekly Android Broker on-call (OCE) telemetry report as a self-contained HTML file at `$env:USERPROFILE\android-oce-reports\oncall-wow-report-YYYY-MM-DD.html` (where `YYYY-MM-DD` is the **end-date of the rolling 7-day window** — see "Inputs to confirm" §1). Writes to the user's home folder, **outside the workspace**, so reports never accidentally get committed. +This skill produces the weekly Android on-call telemetry reports. It is a **router**: it resolves +the reporting window, decides which app playbook(s) to run, and stitches the results together into +a combined index. **All app-specific analysis lives in the playbooks** — read the one(s) you need +after resolving the mode. -The output mirrors the structure of the canonical template at [`assets/templates/report-template.html`](assets/templates/report-template.html). The Step 1 bootstrap script copies the template into `~/android-oce-reports/oncall-wow-report-.html`, **stamps the resolved rolling-7-day window into the title / meta line / Generated banner**, and you edit it in place from there. Do **not** redesign the layout each run. +| Playbook | Covers | Read when mode is | +|---|---|---| +| [`assets/playbooks/broker.md`](assets/playbooks/broker.md) | Android Broker — error codes/types, spike + code attribution, latency, broker version adoption | `broker`, `both` | +| [`assets/playbooks/authapp.md`](assets/playbooks/authapp.md) | Authenticator app — scenario funnels (Passkey / Entra MFA / Entra PSI / MSA NGC+SA), error reasons, abandonment, Broker API responsiveness, crash rate | `authapp`, `both` | + +> **Do not read both playbooks into one context when running `both`.** Run them as two parallel +> sub-agents (see § Orchestration). Each playbook is large by design; interleaving them degrades +> both reports and risks cross-contaminating the two apps' incompatible Kusto conventions. -**Before writing any KQL, read [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md).** It captures the canonical view names, helper functions, the HLL device-count gotcha, week-alignment rules, and ready-to-paste query templates — distilled from the production Android Broker Dashboard. +--- -Reusable helpers in [`assets/`](assets/): +## Mode selection -| File | Purpose | +**Default is `both`.** Infer the mode from the request; only ask if genuinely ambiguous. + +| Signal in the request | Mode | |---|---| -| [`report-template.html`](assets/templates/report-template.html) | Canonical layout — a real prior-week report kept verbatim. **Edit in place** (replace dates / values / verdicts / PR links); do not restyle. See [`template-readme.md`](assets/templates/template-readme.md) for what to change vs leave alone. | -| [`template-readme.md`](assets/templates/template-readme.md) | Author guide for `report-template.html` — what to change per week, color palette, CSS class quick-reference | -| [`kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md) | Schemas, helper funcs, gotchas, ready-to-paste KQL templates, AADSTS reference | -| [`code-attribution-template.md`](assets/docs/code-attribution-template.md) | Per-card checklist for the deep code-attribution block (Originator / Top throw site / Wrapper / Caller hot-spots / Underlying cause / Top error_messages / Likely PRs / Next step) | -| [`queries/`](assets/queries/) | Canonical KQL templates, one file per query — see [`queries/README.md`](assets/queries/README.md). Highlights: [`attr-union-by-dim.kql`](assets/queries/attr-union-by-dim.kql) (NEW — all 7 dims in one round-trip), [`error-message-and-location.kql`](assets/queries/error-message-and-location.kql) (now accepts BOTH `` and `` in one call) | -| [`templates/`](assets/templates/) | Copy-paste HTML snippets (`spike-card.html`, `traffic-attr-card.html`, `sparkline-footer.html`) | -| [`bucket-trends.js`](assets/scripts/bucket-trends.js) | Bucket all error codes into 60-day regression / spike / improvement / flat. Run with `--metric=devs` AND `--metric=reqs`. Pass `--end=YYYY-MM-DD` (the Sunday that OPENS the current in-progress week, exclusive — i.e. `startofweek(today)`, printed by bootstrap as "Trend delta cutoff") to exclude the partial week from the delta math, plus **`--include-partial-end`** to still chart it as the final bar. **`--summary` suppresses the verbose header; `--json=` emits a structured sidecar for programmatic consumption.** | -| [`agg.js`](assets/scripts/agg.js) | Per-error per-dim top-N rollup with WoW deltas. Workhorse for filling spike-attribution dim blocks. | -| [`summarize-attribution.js`](assets/scripts/summarize-attribution.js) | Roll up 7-dim attribution slices for spike-attribution cards. Supports BOTH `--union ` (preferred for 2-week WoW; pairs with `attr-union-by-dim.kql`) AND legacy `--label= file.json` per-dim mode. **Auto-detects the array-form schema produced by `assets/scripts/run-kql.ps1` — no schema-transformer step needed.** | -| [`find-suspect-prs.ps1`](assets/scripts/find-suspect-prs.ps1) | Parallel `git log -S` + `--grep` across broker/ + common/ for a class/method symbol, with PR numbers + URLs. Run *only after* the Originator pre-check has identified a specific throw-site class — the unscoped 4-week PR window is small enough (<30 PRs) to scan with plain `git log` first. | -| [`validate-report.ps1`](assets/scripts/validate-report.ps1) | Pre-publish validator. Catches stale tokens, devs/reqs leaks, mojibake (U+FFFD), unbalanced `
` depth in Section 2 (the nested-callout bug), KPI/trend sparkline coverage, code-attribution depth, layout-guard CSS presence, and suspicious low-peak fabricated `data-trend` arrays. Run as part of Step 7. | -| [`scripts/run-kql.ps1`](assets/scripts/run-kql.ps1) | **Direct-REST Kusto helper — drop-in fallback for the Azure Kusto MCP server when the MCP times out** (frequent on per-error-code queries). Acquires a token via `az`, POSTs to `/v2/rest/query`, writes a JSON file the JS helpers can consume directly. | -| [`scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1) | Bootstrap a new report from the canonical template. Auto-computes the rolling 7-day window (curEnd = today UTC; override with `-EndDate YYYY-MM-DD`), creates `_data//`, prunes `_data` folders older than 60 days, **stamps the resolved window into the `` / `<div class="meta">` block / Generated banner**, and detects "unfilled template stub" vs "real prior report" collisions using a template-only sentinel token (see § collision detection). | -| [`scripts/visual-smoke.ps1`](assets/scripts/visual-smoke.ps1) | Optional Playwright-based layout smoke test. Renders the report at 1400 px viewport, captures a full-page screenshot under `~/android-oce-reports/_visual/`, and runs DOM-based overflow + adjacent-card-gap detection. Catches the rendered-layout bugs (text bleed, cards touching) that pure HTML/CSS validation can't see. | +| *"the OCE report"*, *"weekly on-call report"*, *"this week's telemetry"*, no app named | `both` | +| *"broker report"*, *"broker health"*, *"error codes"*, *"spike attribution"* | `broker` | +| *"authenticator report"*, *"authapp"*, *"passkey/MFA/PSI/MSA scenarios"*, *"registration success rate"* | `authapp` | + +Optional flags the user may add: `--skip-crashes` (Authenticator only — skips the App Center +crash layer, which needs a secret), `--end YYYY-MM-DD` (see § Reporting window). --- -## Inputs to confirm with the user +## Reporting window (shared by both apps) -> **⚠️ Do NOT ask the user for a reporting date by default.** The skill uses a **rolling 7-day window** ending at start-of-day UTC on the invocation day so the most recent complete days are always captured. `assets/scripts/bootstrap-report.ps1` computes and stamps the window automatically. The previous "confirm the Sunday bucket with the user" flow silently produced stale windows (e.g. a Thursday run emitted a 4-day partial window and dropped the last complete week) — that failure mode is what this section explicitly guards against. +> **⚠️ Do NOT ask the user for a reporting date by default.** Both reports use the **same rolling +> 7-day window** ending at start-of-day UTC on the invocation day, so the most recent complete days +> are always captured and the two reports are directly comparable. +> [`assets/scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1) computes and stamps +> the window automatically. The previous "confirm the Sunday bucket with the user" flow silently +> produced stale windows (a Thursday run emitted a 4-day partial window and dropped the last +> complete week) — that failure mode is what this section explicitly guards against. -1. **Reporting window** — do NOT ask. Run `bootstrap-report.ps1` with no arguments; it resolves the window silently and prints: +1. **Reporting window** — do NOT ask. Run `bootstrap-report.ps1`; it resolves the window silently + and prints: ``` Resolved reporting window (UTC): # example values for a run on 2026-07-15 Last 7 days: 2026-07-08 -> 2026-07-15 (exclusive upper bound) @@ -44,512 +55,176 @@ Reusable helpers in [`assets/`](assets/): 60-day trend: 2026-05-16 -> 2026-07-15 (literal 60d ending today; chart includes current partial week) Trend delta cutoff: weeks < 2026-07-12 (startofweek(curEnd); pass as bucket-trends.js --end) ``` - These dates are stamped into the report's `<title>`, `<div class="meta">`, and Generated banner during bootstrap — you do not hand-edit them (see `assets/templates/template-readme.md` § Date fields). + These dates are stamped into each report's `<title>`, `<div class="meta">`, and Generated banner + during bootstrap — you do not hand-edit them. - **Override only when the user explicitly requests a non-default window.** Signals: "the week of X", "as of last Friday", "the report from three weeks ago", "in-progress data", "just today's numbers". Then use: + **In `both` mode, resolve the window ONCE and pass the same `-EndDate` to both bootstrap calls.** + Letting each playbook resolve its own window independently is how the two reports drift onto + different days when a run straddles UTC midnight. + + **Override only when the user explicitly requests a non-default window.** Signals: "the week of + X", "as of last Friday", "the report from three weeks ago", "in-progress data", "just today's + numbers". Then use: ```pwsh - .\bootstrap-report.ps1 -EndDate 2026-07-02 # e.g. reproduce the report as of Jul 2 + .\bootstrap-report.ps1 -App broker -EndDate 2026-07-02 # e.g. reproduce the report as of Jul 2 ``` - `-EndDate` is the exclusive upper bound of the current window (`curEnd`); the script derives `curStart = curEnd - 7d` and `prevStart = curEnd - 14d` deterministically. `-EndDate` must be today or earlier — future dates are refused. - - **Supported override = `-EndDate` only (shifts the window *end*).** The 7-day primary span, the 7-day baseline, and the 60-day trend form a fixed frame that moves *rigidly* with `-EndDate`. There is **no** custom start-date and **no** arbitrary-span flag — a request like "last 30 days" or "from Jun 1 to Jun 20" is **not** expressible via a single parameter. To produce a longer or custom span you must edit the KQL window placeholders (`<CUR_START>` / `<CUR_END>` / `<PREV_START>`; the baseline end is always `<CUR_START>`, so there is no separate `<PREV_END>` token) by hand. If the user asks for a non-7-day span, say so up front rather than silently emitting a 7-day report. - -2. **Comparison baseline** — auto-computed as the immediately-prior 7 days (`[curEnd - 14d, curEnd - 7d)`). No user input. - -3. **60-day trend window** — auto-computed as the **literal last 60 days ending today** (`[curEnd - 60d, curEnd)`), so both bounds move with `-EndDate`. The section is still Sun-Sat weekly-bucketed (Kusto `startofweek()` is Sunday-aligned) because the trend needs stable weekly denominators, but the final bar is the **current in-progress (partial) week** — the chart ends today. Regression/improvement **delta classification is still computed on complete weeks only** (`bucket-trends.js --end=startofweek(curEnd) --include-partial-end`); a partial week as "last" would read as a fake −99% improvement, so it is charted but excluded from the delta math. `bootstrap-report.ps1` prints both the 60d data window and the "Trend delta cutoff" (= `startofweek(curEnd)`). - -4. **Output filename** — `$env:USERPROFILE\android-oce-reports\oncall-wow-report-YYYY-MM-DD.html` where `YYYY-MM-DD` is the resolved `curEnd`. Example: run on 2026-07-09 (default) → `oncall-wow-report-2026-07-09.html`. User-scoped, outside the workspace. The filename date always matches the meta-line "Last 7 days" end-date; `validate-report.ps1` check #11 asserts this. - -**Kusto note (60-day trend section only):** `startofweek()` is Sunday-aligned, so `startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. When authoring weekly-bucketed queries (60-day trend, `wow-table-sparkline-series.kql`), verify by printing the distinct `startofweek(EventInfo_Time)` values from your first query. Off-by-one-week is the #1 silent error in weekly-bucket queries. - ---- - -## Required sections (in order) - -1. **Top-line health KPIs** — total requests, total devices, silent-auth reliability %, interactive reliability %, p95 latency on the hot spans. WoW delta on each. Inline SVG sparklines. -2. **Things that need attention this week** — callouts: - - **Denominator caveat** — explain any large total-spans device-count shift caused by span-emission changes (e.g. `goAsync()` refactors). Always state which denominator the report uses (auth-only: `SilentAuthStats` ∪ `InteractiveAuthStats`). - - **🔴 WoW regressions (last 7 days)** — *one* callout listing every code/type that moved sharply WoW, **sorted by current-week device count descending**. Built from the union of (a) the standard WoW table and (b) [`assets/queries/wow-movers.kql`](assets/queries/wow-movers.kql) so small-but-recent spikes appear in the same list as the high-volume ones. Each row uses the `.item` flat-row pattern (see `assets/templates/template-readme.md` § "Section 2 callouts"): name + inline metric chips + tags pushed right + one-line body + optional foot with `Attribution card →` link. **Section 2 rows are at-a-glance only** — do not duplicate the dim slicing / PR analysis / detailed verdict here; that belongs in the Section 4 spike-attribution card. Each row carries tags: `NEW` (first appeared this week or last), `60d↑` (also rising on 60d), and an originator chip (`broker` / `eSTS` / `Android` / `env`). Reader's eye prioritizes naturally by row order and tag combination — broker-tagged rows at the top demand the most attention. - - **Slow-burn 60-day regressions** — codes/types climbing on the 60d window that are flat WoW. Anything that *also* moved WoW belongs in the red callout above (with `60d↑`), not here. Link to the 60-Day Trend section. - - **Real wins this week**, with PR links. - - **Traffic shape** — flat / surge / collapse summary. -3. **📈 60-Day Trend Analysis** — built from the `ErrorStatsMetrics` materialized view over the **literal last 60 days ending today** (final bar = current in-progress week). **Run the bucketing pipeline FOUR times — the cross-product of `{error_code, error_type} × {devices, requests}`** — and union the regression sets. An entry (code OR type) is flagged if it regresses on either metric. Deltas are computed on complete weeks only; the partial current week is charted but excluded from classification. - - - **% of devices** affected (`devicesHit / authActiveDevices`) — catches errors hitting more users. - - **% of requests** affected (`errRequests / authTotalRequests`) — catches per-device retry storms (fewer users, more traffic per user). The previous report would have missed `kdfv2_key_derivation_error` (262 → 5,374 requests on ~57 devices) without this dim. - - Categories: True 60d regression / Ephemeral 60d spike (peak-then-recover) / True 60d improvement / Flat. Every rising entry — whether `error_code` or `error_type` — gets the same Spike Attribution + Code Attribution treatment (Step 4 / Step 5). - - Always apply `MergeUiRequiredExceptions(error_type)` before bucketing on type; otherwise the 6+ string variants of `UiRequiredException` will each be tracked separately and skew the buckets. -4. **🔎 Spike Attribution** — one card per WoW regression AND per 60-day regression, **for both `error_code` and `error_type` regressions**. Each card slices on **all 7 dimensions** (broker version, span, active broker pkg, calling app, account type AAD/MSA, shared-device mode, client SKU). Each card ends with a **deep Code Attribution block** (see Step 4 for the required fields) and a Traffic Attribution verdict. -5. **🚚 Traffic Attribution** — top-level section listing every error whose spike is fully or partly explained by traffic volume from a specific calling app, rather than a code regression. If none qualify this week, render the section with an explicit "None this week" note. -6. **Error codes — WoW with stable denominator** — full table with `Δ requests %` and `Δ devices %` columns and the 60d sparkline. -7. **Error types — WoW with stable denominator** — full table, **same columns and rigor as the error-codes table** (`Δ requests %`, `Δ devices %`, 60d sparkline, status pill). Any regressing type also gets a spike-attribution card in Section 4. For composite types (e.g. `ClientException` is the umbrella for many sub-codes), include a **decomposition card** that breaks the WoW Δ down into the top 3 contributing sub-codes — so a `ClientException` −5 pp drop is explicitly attributed to e.g. `−8.5 pp timed_out_execution` + `−3.4 pp unknown_authority` + `−0.15 pp illegal_argument_exception`. -8. **📊 Traffic analysis** — total requests/devices (WoW + 60d), top calling apps, top spans, **requests-per-device ratio** per error and overall (a rising ratio = retry storm; a falling ratio = caching gain), sampling-rate change indicator. -9. **Latency** — p50/p95/p99 by hot span. -10. **Broker version adoption** — week-over-week version share. -11. **Appendix** — query list and methodology. + `-EndDate` is the exclusive upper bound of the current window (`curEnd`); the script derives + `curStart = curEnd - 7d` and `prevStart = curEnd - 14d` deterministically. `-EndDate` must be + today or earlier — future dates are refused. + + **Supported override = `-EndDate` only (shifts the window *end*).** The 7-day primary span, the + 7-day baseline, and the 60-day trend form a fixed frame that moves *rigidly* with `-EndDate`. + There is **no** custom start-date and **no** arbitrary-span flag — a request like "last 30 days" + or "from Jun 1 to Jun 20" is **not** expressible via a single parameter. To produce a longer or + custom span you must edit the KQL window placeholders (`<CUR_START>` / `<CUR_END>` / + `<PREV_START>`; the baseline end is always `<CUR_START>`, so there is no separate `<PREV_END>` + token) by hand. If the user asks for a non-7-day span, say so up front rather than silently + emitting a 7-day report. + +2. **Comparison baseline** — auto-computed as the immediately-prior 7 days + (`[curEnd - 14d, curEnd - 7d)`). No user input. + +3. **60-day trend window** — auto-computed as the **literal last 60 days ending today** + (`[curEnd - 60d, curEnd)`), so both bounds move with `-EndDate`. Trend sections are Sun-Sat + weekly-bucketed (Kusto `startofweek()` is Sunday-aligned) because the trend needs stable weekly + denominators, but the final bar is the **current in-progress (partial) week** — the chart ends + today. Regression/improvement **delta classification is still computed on complete weeks only** + (`bucket-trends.js --end=startofweek(curEnd) --include-partial-end`); a partial week as "last" + would read as a fake −99% improvement, so it is charted but excluded from the delta math. + +**Kusto note (weekly-bucketed queries only):** `startofweek()` is Sunday-aligned, so +`startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. When authoring weekly-bucketed queries, verify +by printing the distinct week values from your first query. Off-by-one-week is the #1 silent error +in weekly-bucket queries. --- -## Step-by-step workflow - -### Step 1 — Bootstrap the new report file from the template - -This skill ships with a canonical template at [`assets/templates/report-template.html`](assets/templates/report-template.html) (a real prior report kept as the reference layout). **Use [`assets/scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1)** to handle all the boilerplate (rolling-window computation, `_data/<end-date>/` directory, header stamping, retention-pruning, collision detection): - -```pwsh -.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\bootstrap-report.ps1 -# Optional: explicit end-date (curEnd, exclusive upper bound) + force overwrite -# .\bootstrap-report.ps1 -EndDate 2026-07-02 -Force -``` - -What it does: -* Resolves the rolling 7-day window from the system clock in UTC (`curEnd = today`, `curStart = curEnd - 7d`, `prevStart = curEnd - 14d`) — or from `-EndDate` if passed. -* Creates `~/android-oce-reports/oncall-wow-report-<curEnd>.html` from the canonical template. -* Creates `~/android-oce-reports/_data/<curEnd>/` for raw KQL JSON payloads. -* **Stamps the resolved window into the report's `<title>`, `<div class="meta">` block, and Generated banner** — you never hand-edit header dates. The resolved window is echoed in the report header for transparency. -* Prunes `_data/<old-end-date>/` folders older than 60 days so the cache doesn't accumulate. -* **Collision detection (fail-safe):** an existing same-day report is silently re-bootstrapped only when it is *positively* identified as an unpopulated stub — it still carries the `OCE-UNPOPULATED-STUB` sentinel that bootstrap injects **and** its first KPI still equals the template's value. Anything else (sentinel removed, or KPIs edited) is treated as real work: **HARD HALT, exit 2**, requiring `-Force` to overwrite. `validate-report.ps1` refuses to pass a report that still carries the sentinel, so a published report can never be misclassified as a stub. - -Edit the bootstrapped file in place — the template ships as a real prior-week report (not a tokenized skeleton). **Walk top-to-bottom and replace every prior-week date / KPI value / table row / verdict / PR citation with current-week data.** The CSS, sparkline JS, section ordering, and attribution-card markup are canonical — do not redesign them. See [`assets/templates/template-readme.md`](assets/templates/template-readme.md) for the full guide on what to change vs leave alone, the sparkline color palette, the CSS class reference, and the two v8 layout traps. - -> **⚠️ UTF-8 trap — DO NOT use PowerShell `@'...'@` heredocs to compose HTML content containing emojis, em-dashes, arrows, or middle dots.** PowerShell silently strips multi-byte UTF-8 characters when piping heredocs to `Set-Content` / `Out-File`. Use Node.js (`fs.writeFileSync`), `[IO.File]::WriteAllText($path, $text, [System.Text.UTF8Encoding]::new($false))`, or explicit Unicode-pair literals (`[char]0xD83D + [char]0xDCCA` for 📊) instead. This trap cost ~30 min in v8 and required a full emoji-restoration pass — every callout icon, every section header emoji, every arrow link had to be re-injected. The validator's `U+FFFD` check catches the worst case (mojibake replacement char) but cannot detect characters that were silently stripped to nothing. - -Mark any unfinished card or table cell with the literal sentinel `EXAMPLE CONTENT BELOW` inside an HTML comment — the final-pass validator (Step 7) greps for it. - -If the template ever needs structural improvements (new section, new card style, etc.), update `assets/templates/report-template.html` in the skill folder and commit it so future weeks inherit the change. - -### Step 2 — Pull WoW reliability data - -Use the Kusto MCP tool against: -- **Cluster:** `https://idsharedeus2.kusto.windows.net` -- **Database:** `ad-accounts-android-otel` - -**Always prefer the canonical `materialized_view('XxxMetrics' or 'XxxUpdated')` variants** — these are what the production dashboard uses, are pre-aggregated and HLL-bucketed, and avoid the 240 s MCP timeout that plain `android_spans` queries hit. Full schema, gotchas, and query templates: [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md). - -> **Fallback when the Kusto MCP times out:** use [`assets/scripts/run-kql.ps1`](assets/scripts/run-kql.ps1). It acquires a token via `az account get-access-token`, POSTs directly to `/v2/rest/query`, and writes the result as a JSON file the JS helpers (`bucket-trends.js`, `summarize-attribution.js`) can consume directly. The skill's MCP-vs-REST switch is roughly: try the MCP once; if it returns `McpError -32001 (timeout)`, switch to the REST helper for the rest of the run. Run multiple queries in parallel via PowerShell `Start-Job`: -> -> ```pwsh -> $queries = @{ 'reliability.json' = $reliabilityKql; '60d-codes.json' = $codesKql; ... } -> $jobs = @() -> foreach ($f in $queries.Keys) { -> $q = $queries[$f] -> $jobs += Start-Job -ScriptBlock { -> param($Q, $O) & "$using:skillRoot\assets\scripts\run-kql.ps1" -Query $Q -Out $O -> } -ArgumentList $q, $f -> } -> $jobs | Wait-Job | Receive-Job; $jobs | Remove-Job -> ``` - -| Need | View | -|------|------| -| Per-error-code / per-error-type / per-span counts | `materialized_view('ErrorStatsMetrics')` | -| Total broker requests / devices | `materialized_view('BrokerAdoptionStatsUpdated')` | -| Silent auth reliability | `SilentAuthStatsAllRequestsMetrics` + `SilentAuthStatsRequestsWithoutExpectedErrorMetrics` | -| Interactive auth reliability | `InteractiveAuthStatsAllRequestsMetrics` + `InteractiveAuthStatsRequestsWithoutExpectedErrorMetrics` | -| Latency (p50/p95/p99) | `materialized_view('PerfStatsUpdated')` — use `percentile_tdigest(tdigest_merge(responseTimeTDigest), N, typeof(long))` | -| Broker version share | `BrokerAdoptionStatsUpdated` | -| Calling app share | `AppStatsUpdated` | -| SKU share | `SkuStatsUpdated` | -| Spike-by-flight slicing | `Operations_ByFlight`, `ErrorCodeBySpan_ByFlight`, `ErrorType_ByFlight` | - -Time filter: always use `EventInfo_Time` on materialized views. Use `PipelineInfo_IngestionTime` only on raw `android_spans`. - -**Three rules that will silently corrupt your data if violated** (full detail in the cheatsheet): - -1. **Distinct devices are HLL-encoded.** Use `dcount_hll(hll_merge(countDevicesHll))`, never `sum(countDevices)`. Summing double-counts every device that appears in more than one row. -2. **Apply the dashboard helper functions** so this report agrees with the dashboard: `MergeAccountType(account_type)`, `MergeIsSharedDevice(is_shared_device)`, `MergeUiRequiredExceptions(error_type)`. -3. **Auth-only denominator for reliability %s:** sum `countRequests` from `SilentAuthStatsAllRequestsMetrics` ∪ `InteractiveAuthStatsAllRequestsMetrics` — not total broker spans. Total span counts are sensitive to `goAsync()` / receiver refactors and will give false WoW reliability swings. - -### Step 3 — Pull 60-day trend - -Don't pre-filter to a hand-picked top-N list — small-but-rising errors (e.g. `null_pointer_error` at ~67K devices) will fall off and never show up in the trend section. Instead pull every error code **and every error type** with a meaningful baseline across the window, then bucket each. - -#### 3a. Per-error-code trend - -Use [`assets/queries/60d-trend-codes.kql`](assets/queries/60d-trend-codes.kql) (template; replace `<TREND_START>` and `<TREND_END>` tokens. **`<TREND_START>` = `curEnd − 60d`** and **`<TREND_END>` = `curEnd` (today), exclusive** — the literal last 60 days. `bootstrap-report.ps1` prints the resolved values): - -```kql -materialized_view('ErrorStatsMetrics') -| where EventInfo_Time >= datetime(<TREND_START>) and EventInfo_Time < datetime(<TREND_END>) -| where isnotempty(error_code) and error_code != 'success' -| summarize errs = sum(countOverall), - devs = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), error_code -| order by error_code asc, week asc -``` - -**Do NOT filter the partial in-progress week here.** The chart wants it as the final bar (the window ends today). The partial week is excluded from the regression/improvement **delta math** by `bucket-trends.js` via `--end=<TREND_CLASS_END> --include-partial-end` (see 3c), not at the source — a partial week driving the delta would read as a fake −99% improvement, which is exactly why classification and display are split in the JS. - -#### 3b. Per-error-type trend (same rigor) - -```kql -materialized_view('ErrorStatsMetrics') -| extend unified_error_type = MergeUiRequiredExceptions(error_type) -| where EventInfo_Time >= datetime(<TREND_START>) and EventInfo_Time < datetime(<TREND_END>) -| where isnotempty(unified_error_type) -| summarize errs = sum(countOverall), - devs = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), unified_error_type -| order by unified_error_type asc, week asc -``` - -`MergeUiRequiredExceptions` is mandatory — without it the 6+ string variants of `UiRequiredException` (raw, fully-qualified, com.microsoft.identity.common.exception.*) each show as separate rows and skew the buckets. - -#### 3c. Run the bucketer 4 times (cross-product of `{code, type} × {devices, requests}`) - -`bucket-trends.js` defaults to grouping by `error_code`. For the type runs you MUST pass `--key=unified_error_type` so it picks up the right column from the type-trend JSON. - -```pwsh -# Error codes — by devices, then by requests. -# TREND_START = curEnd - 60d (literal 60d start) -# TREND_CLASS_END = startofweek(today) ("Trend delta cutoff" printed by bootstrap) -# --include-partial-end charts the current partial week while excluding it from deltas. -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end --metric=reqs - -# Error types — by devices, then by requests (note --key) -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end --key=unified_error_type -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end --key=unified_error_type --metric=reqs -``` - -`--end` is `<TREND_CLASS_END>` = `startofweek(today)` (exclusive) — the Sunday that opens the current in-progress week. Weeks at or after it (the partial current week) are excluded from delta classification; `--include-partial-end` keeps that week in the emitted `series` so the chart ends today. The script also auto-detects partial end-buckets and warns if `--end` is omitted, but passing it explicitly is safer. - -Take the **union** of all four regression sets. Both `error_code` and `error_type` regressions get a spike-attribution card in Step 5. - -It will print regression / spike / improvement / flat buckets, sorted by peak. The thresholds (in case you need to tune): - -- **True 60d regression:** `delta > +15%` and trajectory is monotonic-ish (no single-week spike dominating). -- **Ephemeral 60d spike:** peak week is ≥3× the mean of the surrounding weeks (peak-then-recover shape). -- **True 60d improvement:** `delta < −15%`. -- **Flat:** otherwise. -- Codes/types with peak weekly devices `< 10K` (or peak weekly requests `< 100K` when `--metric=reqs`) are filtered out (`--peak-floor=N` to override). - -**Why both axes matter:** -- *codes × requests:* in v5, `kdfv2_key_derivation_error` spiked +1,951% on requests across only ~57 devices — a per-device retry storm device-only bucketing would have missed. -- *types × either:* `error_type` is the umbrella (e.g. `ClientException`, `ServiceException`, `UiRequiredException`) — a moving type that doesn't map cleanly to one moving code is a strong signal of a *new* sub-code being introduced or an existing one being reclassified (the v5 `ClientException` −10% drop was driven by `timed_out_execution` reclassification under PR #141, which would have been invisible from the codes table alone). - -**Always present side-by-side WoW tables for BOTH error_code AND error_type** with `Δ requests %` and `Δ devices %` columns; flag any row where either crosses threshold. - -#### 3d. WoW movers query — MANDATORY pass to catch small-base movers - -The 60d bucketer's `--peak-floor=10000` exists for good reason (otherwise the 60d regression list would be 200+ tiny noise codes), but it **silently drops every code whose absolute weekly volume stays under 10K** — even if that code is brand-new or just spiked 5× WoW. Real examples this skill has missed in the past: - -- `Failed to parse JWT` — went `7 → 32 → 54 → 46 → 55 → 892 → 3,461` over 7 weeks (2-week-old NEW spike, real broker code in `IDToken.parseJWT:38`). Never crossed the 10K floor. -- `Code:-11` — sat at ~1,030 devs/week for 7 weeks then jumped to 2,433 (+165% WoW). Sub-floor. -- `SSLHandshakeException` — devices flat at 260 but requests +186% WoW (per-device retry storm). The bucketer's reqs-axis floor (100K) just barely captures it but the device floor doesn't. - -To catch these, **always** run [`assets/queries/wow-movers.kql`](assets/queries/wow-movers.kql) **as a separate pass after the 60d bucketing**: +## Outputs -```kql -// inputs: <CUR_END> = curEnd (exclusive), <CUR_START> = curEnd - 7d, -// <PREV_START> = curEnd - 14d. Printed by bootstrap-report.ps1. -// floor: cDev>=500 OR cReq>=5000 move: |Δd|>=25% OR |Δr|>=50% OR new-this-window -``` - -Run it **twice — once for `error_code`, once for `error_type`**. **Merge its output rows into the same 🔴 WoW regressions callout as the standard WoW table** (sorted by current-week device count descending). Tag rows that came in via this pass with `NEW` if they were absent or near-zero in the prior week. Do *not* render this as a separate "emerging" callout — the size split is implementation detail; readers prioritize naturally by absolute device count + originator chip. - -For each WoW mover (regardless of size), you still owe the full Code Attribution treatment (Step 4). The dim-slicing pass (Step 5) is allowed to be deferred for sub-1K-device spikes if the throw-site + dominant message already pin the originator unambiguously — but say so explicitly in the card ("dims not yet sliced — file the bug first; pull dims if it persists"). - -### Step 4 — Code attribution (deep PR correlation) - -> ⚠️ **HARD RULE — Originator pre-check.** Before claiming `Originator: Broker` on any card, you MUST run [`assets/queries/error-message-and-location.kql`](assets/queries/error-message-and-location.kql) for that error code (or type) and read **(a) the throw-site stack and (b) the top 3 `error_message` strings**. Most broker error codes flow through `common/ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult, clientExceptionFromException}` — which intentionally bridge eSTS responses into broker exceptions. **If the throw site is in any of those three methods AND the error_message starts with `AADSTS`, the originator is eSTS, not broker.** See the AADSTS reference table in [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md). Cards that skip this step must be marked low-confidence, not high. -> -> **Window:** use the FULL 7-day rolling window (`<CUR_START>` → `<CUR_END>`) on `PipelineInfo_IngestionTime`, NOT a narrower 3–5 day slice — low-volume types (e.g. `SSLHandshakeException`, `IntuneAppProtectionPolicyRequiredException`) routinely return zero rows in a sub-window slice. If a code/type still returns nothing, fall back to the prior 14 days (`<PREV_START>` → `<CUR_END>`) before declaring "no data". - -For every regression card, the Code Attribution block **must** populate the following fields. Shallow PR-citation only is not acceptable. Use [`assets/docs/code-attribution-template.md`](assets/docs/code-attribution-template.md) as the per-card checklist. - -| Field | What goes in it | How to find it | -|---|---|---| -| **Originator** | Where the error physically originates: broker code / common / Android system (WebView / Conscrypt / Keystore) / 3rd-party lib (Nimbus JWT, okhttp) / eSTS server / environmental (enterprise TLS interception). Use the colour-coded `origin-tag` spans (`origin-broker`, `origin-android`, `origin-thirdparty`, `origin-env`). | Grep the error string across `broker/`, `common/`, `msal/`. If no match, it's not our code — search the Android SDK or call out as eSTS-returned. | -| **Top throw site** | Fully-qualified file:line where the exception is constructed, plus the % of cases that throw from this single site. | Pull `error_location` / stack-prefix from `android_spans` for the spiking error code (one targeted query, narrow time window). Cite the dominant site. | -| **Wrapper** | Broker/common code that catches the originator's exception and re-throws it as the user-visible error code. Often `IDToken.parseJWT()`, `ServiceException(...)`, `ExceptionAdapter.exceptionFromAuthorizationResult()`. | Walk up the stack from the throw site — check for `try { ... } catch (X e) { throw new Y(...); }` patterns in broker/common. | -| **Caller hot-spots** | Top 1–3 callers of the wrapper, with device counts. Helps identify the specific code path the regression flows through. | `android_spans` slice by `error_location` (or `error.stack_trace` first frame inside our code). | -| **Underlying cause** | The proximate cause one level deeper (e.g. "99% `CertificateException` from `TrustManagerImpl.verifyChain`", "84% `no_such_algorithm` from `ProviderFactory.getMessageDigest`"). | `android_spans` slice by `error.cause` or `error_message` first 80 chars. | -| **Top error_messages** | Top 3–5 distinct `error_message` strings with counts. Often reveals the 3rd-party library or environmental signal (e.g. `net::ERR_SSL_PROTOCOL_ERROR`, Zscaler-issued cert names). | `summarize count() by tostring(error_message)` on raw `android_spans` filtered to the spike. | -| **Likely PRs** | 1–3 PRs with confidence rating (high / medium / low / none), full GitHub URL, commit SHA, author, AB#, and a 1-sentence **why-it's-the-suspect** justification (not just the title). Use the `pr-card` markup. | See PR-grep below. **Cite confidence honestly** — "none" is a valid verdict for environmental errors. | -| **Next step** | Concrete action with a named owner: who runs the next slice, who files the bug, what flight to flip, what correlation IDs to pull. | Pulled from PR authors / CODEOWNERS for the affected file. | - -#### PR-grep workflow - -**Read the full PR window first, then reason — don't `--grep` blind.** The 4-week window across `broker/` and `common/` typically returns <30 PRs total, small enough to read end-to-end. Targeted `--grep` matches will miss PRs whose titles don't mention the error string (most of them). **The recommended order is:** - -1. **Run plain `git log` on both repos** for the 4-week window. Read the resulting list end-to-end before any greps. -2. **Cross-reference titles + dates** against the Originator pre-check throw-site class. -3. **Only when you have a specific symbol** to chase (e.g. the throw-site class identified in step 2), reach for `find-suspect-prs.ps1` to do the symbol-targeted parallel pickaxe + grep. - -The historical mistake (pre-v8) was to jump straight to `find-suspect-prs.ps1` without reading the window first, which silently dropped PRs whose titles didn't mention the symbol. - -```pwsh -# Step 1: read the full 4-week window -cd c:\Users\shjameel\Repos\android-complete\broker -git --no-pager log --since='<windowStart>' --until='<windowEnd>' --pretty=format:'%h | %ai | %an | %s' --no-merges - -cd ..\common -git --no-pager log --since='<windowStart>' --until='<windowEnd>' --pretty=format:'%h | %ai | %an | %s' --no-merges -``` - -For each candidate PR, **read the diff** to confirm it touches the throw site / wrapper class identified in the Originator pre-check. Don't cite a PR just because the title mentions a related concept. - -```pwsh -# Step 3 (optional): symbol-targeted focused follow-up. Use ONLY after step 1 gave -# you a specific class/method name to chase from the Originator pre-check. -# Searches both repos in parallel via `git log -S` (pickaxe on diff) AND `--grep` (subject). -# Returns a unified table: repo | date | author | sha | PR# | URL | subject. -.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\find-suspect-prs.ps1 ` - -Symbol 'ExceptionAdapter' -Since 2026-04-01 -Until 2026-05-09 -``` - -#### Repo URL patterns for citations - -| Repo | URL pattern | -|------|-------------| -| `common/` | `https://github.com/AzureAD/microsoft-authentication-library-common-for-android/pull/<num>` | -| `broker/` | `https://msft.ghe.com/security/ad-accounts-for-android/pull/<num>` | -| `msal/` | `https://github.com/AzureAD/microsoft-authentication-library-for-android/pull/<num>` | -| `adal/` | `https://github.com/AzureAD/azure-activedirectory-library-for-android/pull/<num>` | - -#### Non-broker errors - -For errors with no broker code in the stack (Android system errors like `Code:-10`/`Code:-11`, OEM-specific keystore failures, eSTS-returned codes, environmental TLS interception), explicitly cite **"⚪ None — not in scope"** with confidence `none`, and explain *why* in the why-it's-the-suspect line. Do not invent broker PRs to fill the slot. Tag these errors as `environmental` or `non-broker` so they're tracked but don't page. - -### Step 5 — Spike attribution dimensions - -**Coverage rule: every `error_code` AND every `error_type` that lands in either the WoW regression list OR the 60-day regression list MUST get a spike-attribution card.** No silent skips. - -**`ErrorStatsMetrics` already carries `account_type` and `is_shared_device`** (use the `MergeAccountType` / `MergeIsSharedDevice` helpers to normalize) — so you do **not** need a fallback to raw `android_spans` for these dims. Earlier versions of this skill claimed otherwise; that was wrong. The only dim that requires `android_spans` is `DeviceInfo_OsVersion` (OEM/version slicing). - -Slice on **all 7 dimensions** for each spike. **Preferred for 2-week WoW attribution: one union query that covers all 7 dims for all regressions in a single round-trip** — see [`assets/queries/attr-union-by-dim.kql`](assets/queries/attr-union-by-dim.kql). Typical payload for 8 codes × 2 weeks × 7 dims is ~800 KB, well under the MCP limit. Pipe the result into `summarize-attribution.js --union <file.json>` (which prints per-dim top-N share + Δ devices + Δ requests for every code). Fall back to the per-dim form ([`attr-codes-by-dim.kql`](assets/queries/attr-codes-by-dim.kql)) only when (a) you need a wider time window, or (b) the union response exceeds payload size. - -For `error_type` cards, swap `error_code in (codes)` for `unified_error_type in (types)` and aggregate by the `MergeUiRequiredExceptions(error_type)` extension — otherwise everything else is identical. - -> **Low-volume fallback (extends Step 4's pre-check fallback to the 7-dim union):** when a code/type returns sparse dim rows in the 7-day rolling window — typical for sub-1k-device entries like `TimeoutCancellationException`, `JsonSyntaxException`, `kdfv2_key_derivation_error` — widen the union query to **14 days** (use `<PREV_START>` as the lower bound so the window becomes `[curEnd − 14d, curEnd)`) before declaring "broad — needs targeted slice". The added week of context usually surfaces enough rows to compute concentration percentages. If a code STILL has no concentration after 14 days, mark every dim cell as "not sliced — sub-window volume; file the bug first, slice on persistence" — do NOT fabricate "Broad" verdicts. - -| # | Dimension | Source | Cross-check | -|---|-----------|--------|-------------| -| 1 | Broker version | `ErrorStatsMetrics` group by `broker_version` | Cross-reference `BrokerAdoptionStatsUpdated` to see if the version's request share *also* moved that week — if yes, the spike is rollout-driven, not code-driven | -| 2 | Span name | `ErrorStatsMetrics` group by `span_name` | A single span hosting >60% of the error → strong code-path signal | -| 3 | Active broker package | `ErrorStatsMetrics` group by `active_broker_package_name` | E.g. CompanyPortal vs Authenticator vs LTW | -| 4 | Calling package | `ErrorStatsMetrics` group by `calling_package_name` | If 1–2 callers dominate, this is likely a traffic-attribution case (see Step 6) | -| 5 | Account type (AAD vs MSA) | `ErrorStatsMetrics`, `extend t = MergeAccountType(account_type)` group by `t` | If the split deviates significantly from fleet (~85% AAD / 15% MSA), call it out | -| 6 | Shared device mode | `ErrorStatsMetrics`, `extend s = MergeIsSharedDevice(is_shared_device)` group by `s` | Shared-device fleets have very different error profiles | -| 7 | OS version | [`assets/queries/os-version-slice.kql`](assets/queries/os-version-slice.kql) — raw `android_spans`, group by `DeviceInfo_OsVersion` | **On-demand only** — slice OS-version when EITHER (a) the wrapper class is in `ExceptionAdapter.clientExceptionFromException` (catch-all wrapping a system exception, where the OEM/version often is the cause), OR (b) the error code is one of `Code:-6`, `Code:-10`, `Code:-11`, `unknown_crypto_error`, `io_error`, `null_pointer_error`. Otherwise mark the dim row as "not sliced this week — no OEM concentration suspected" and move on. Slicing OS-version on every card wastes a raw-spans query without changing the verdict. | - -#### Type cards have one extra required dimension: sub-code decomposition +All outputs land in `$env:USERPROFILE\android-oce-reports\` — **outside the workspace**, so reports +can never be committed accidentally. -Because `error_type` is an umbrella over many `error_code` values, every `error_type` regression card MUST also include an **8th dimension: sub-code breakdown** showing the top 3–5 `error_code`s rolled up under that type, with their device counts and Δ vs prior week. This lets the reader see whether the type-level move is driven by one sub-code or many — and routes the deep Code Attribution work to the right sub-code. - -```kql -let curEnd = datetime(<CUR_END>); -let curStart = datetime(<CUR_START>); -let prevStart = datetime(<PREV_START>); -let target_types = dynamic(['ClientException', 'ServiceException']); -materialized_view('ErrorStatsMetrics') -| extend unified_error_type = MergeUiRequiredExceptions(error_type) -| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd -| where unified_error_type in (target_types) -| extend week = iff(EventInfo_Time >= curStart, curStart, prevStart) -| summarize devs = dcount_hll(hll_merge(countDevicesHll)), - errs = sum(countOverall) - by week, unified_error_type, error_code -| order by unified_error_type asc, week asc, devs desc -``` - -Cite the dominant sub-codes inline in the type card's verdict (e.g. *"`ClientException` −10.2% drop is dominated by −8.5 pp `timed_out_execution` + −3.4 pp `unknown_authority`"*) and link to those sub-codes' own attribution cards. The deep Code Attribution block (Step 4) for the type card itself focuses on the **wrapper / catch-and-rethrow** path that defines the type (e.g. `BaseException.java`, `ServiceException.java` constructors), not on each sub-code. - -Feed the union JSON output into the summarizer (one round-trip): - -```pwsh -# Union mode (preferred). attr-union.json comes from attr-union-by-dim.kql. -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\summarize-attribution.js ` - --union attr-union.json --top=5 -# For type cards, add --key=unified_error_type -``` - -Legacy per-dim mode (one JSON per dimension) is still supported for the rare wider-time-window case: - -```pwsh -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\summarize-attribution.js ` - --label=span span.json ` - --label=calling_app app.json ` - --label=active_broker ab.json ` - --label=broker_version ver.json ` - --label=acct_type acct.json ` - --label=shared_dev shared.json ` - --label=client_sku sku.json -``` - -Ready-to-paste KQL for both forms: union → [`assets/queries/attr-union-by-dim.kql`](assets/queries/attr-union-by-dim.kql); per-dim → [`assets/docs/kusto-cheatsheet.md` § 8c](assets/docs/kusto-cheatsheet.md). - -**Concentration thresholds** (paint the dim bar red): -- > 80% in a single value → strong attribution (one root cause) -- 60–80% → medium attribution -- < 60% → broad / cross-cutting → say so explicitly, don't fabricate a single cause - -### Step 6 — Traffic analysis + traffic attribution - -Do this section in three parts. Traffic changes (up *or* down) need the same level of root-cause reasoning as error spikes — a uniform "−9% requests across all top apps with flat devices" is **not** a satisfactory verdict on its own; explain *why*. - -**6a. Top-line traffic shape.** Compare WoW *and* 60d for both totals and per-segment: - -```kql -materialized_view('BrokerAdoptionStatsUpdated') -| where EventInfo_Time > ago(70d) -| summarize totalReq = sum(countRequests), - totalDev = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time) -| order by week asc -``` - -For each of the following, report direction + magnitude: -- Total requests (WoW %, 60d %) -- Total devices (WoW %, 60d %) -- Requests-per-device ratio (a drop often means a benign caching improvement; a spike often means a retry storm) -- Top 10 calling apps (`AppStatsUpdated`) — which apps drove the change? -- Top spans by request volume — did one span explode or collapse? -- Sampling-rate change indicator: if total spans moved >20% but auth-only device count moved <5%, suspect a sampling/instrumentation change. +| Mode | Files produced | +|---|---| +| `broker` | `oncall-wow-report-<curEnd>.html` | +| `authapp` | `authapp-wow-report-<curEnd>.html` | +| `both` | both of the above **plus** `oce-index-<curEnd>.html` | -**6b. Reasoning for material traffic shifts (>10% on any segment).** For every span/app/active-broker that moved meaningfully WoW *or* 60d, run this slicing-and-correlation pass: +`<curEnd>` is `YYYY-MM-DD`, the end-date of the rolling 7-day window. Raw KQL payloads are cached +under `_data/<app>-<curEnd>/` so each report is reproducible; folders older than 60 days are pruned +automatically by `bootstrap-report.ps1`. -| # | Question | How to check | -|---|---|---| -| 1 | **Is the move concentrated in one span?** | Slice top-10 spans by `Δreq` absolute and `Δreq %`. A >50% move on a single span almost always points to a code change (span added / removed / sampled / `goAsync()`-ed). | -| 2 | **Is the move concentrated in one calling app?** | Slice `AppStatsUpdated` WoW. A single app moving >20% in requests with flat devices = client-side caching/retry change in that app — escalate to that app's owners, not broker. | -| 3 | **Is the move concentrated in one active broker pkg?** | Slice `BrokerAdoptionStatsUpdated` by `active_broker_package_name`. AppManager (LTW) vs Authenticator vs Intune CP often diverge during a rollout. | -| 4 | **Is the move concentrated in one broker version?** | Cross-check against rollout share. If a span dropped −80% on `16.0.1` but is flat on `15.1.0`, the cause is in the 16.0.1 diff. | -| 5 | **Did anything else co-move?** | A span dropping while `OnUpgradeReceiver`-style downstream spans also drop (`SecretKeyWrapping`, `WrappedKeyAlgorithmIdentifier` in v5) confirms a single upstream change. | +### The combined index (`both` mode only) -For every meaningful shift, **search for a causal PR** in the repos likely to affect telemetry shape: +After both reports validate, build the index: ```pwsh -# Broker (span add/remove, goAsync, scope changes, sampling/exporter config) -cd c:\Users\shjameel\Repos\android-complete\broker -git log --since='<last8wks>' --oneline -i ` - --grep='span|goAsync|receiver|telemetr|otel|trace|metric|sampl|exporter' - -# Common (instrumentation surfaces) -cd ..\common -git log --since='<last8wks>' --oneline -i ` - --grep='span|telemetr|otel|trace|sampl|instrument' +.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\build-index.ps1 -EndDate <curEnd> ``` -**Causal PR categories that meaningfully shift traffic counts** (flag any of these): - -- **Span removed / renamed / scope-narrowed** → drops the span's count to zero or partial -- **`goAsync()` / `BroadcastReceiver` refactor** → broadcast may complete before async work flushes the span (this is the v5 PR #88 / `OnUpgradeReceiver` story — call it out as a precedent) -- **Sampling-rate change** in broker `Otel*` / `Telemetry*` exporter config or `common/` instrumentation → uniformly scales counts up or down across many spans -- **New span added** in a hot path → request counts for that span jump from ~0 to material -- **Caller-side SDK change** (MSAL/MSAL_CPP/OneAuth release) that batches or caches requests → uniform per-app request drop with flat devices -- **Flight rollout** (ECS) that gates a code path on/off → bursty changes in a specific span on specific dates - -Cite the suspect PR(s) with the same confidence ratings used in Code Attribution (high / medium / low / none) and the same `pr-card` markup. If you can't pin one down, say so explicitly — *"uniform 5–22% per-app request drop with flat devices, no telemetry-platform PR identified, suspect caller-side SDK change in MSAL release X.Y"* is acceptable; "traffic is flat" without checking is not. - -**6c. Per-error traffic attribution (is the *error* spike traffic-driven?).** For every error code flagged in Step 5 as a regression, additionally check whether the spike is *traffic-driven* rather than *failure-rate-driven*: - -```kql -let curEnd = datetime(<CUR_END>); -let curStart = datetime(<CUR_START>); -let prevStart = datetime(<PREV_START>); -let target_code = "<error_code>"; -materialized_view('ErrorStatsMetrics') -| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd and error_code == target_code -| extend week = iff(EventInfo_Time >= curStart, curStart, prevStart) -| summarize errs = sum(countOverall), - devs = dcount_hll(hll_merge(countDevicesHll)) - by week, calling_package_name -| order by week asc, devs desc -``` +It reads the headline KPI tiles out of both finished reports and emits a single one-page digest +with links to each. It is a **digest, not an analysis** — do not write new findings into it that +aren't already in one of the two reports. If a cross-app finding matters (e.g. an Authenticator +Broker-API responsiveness spike that maps to a Broker error code), write it in **both** reports and +let the index link them. -If the spike is concentrated in a single calling app whose **overall** request volume also rose that week (cross-check `AppStatsUpdated`), and the **per-request failure rate is essentially flat**, classify the spike as a **traffic-attribution case** rather than a code regression: +--- -> Example: "`no_account_found` +60% devices this week is fully explained by Outlook's request volume rising 65% — the per-Outlook-request failure rate is unchanged. No broker code change is implicated." +## Orchestration (`both` mode) -Add a top-level **🚚 Traffic Attribution** section that lists every error matched to a traffic-driven origin, mirroring the Code Attribution section. **Each card must include**: the dominant calling app(s) with their WoW request-volume delta, the per-app per-request failure rate (now vs prior — show it's flat), and the recommended owner to route to (typically the calling app's team, not broker). If no errors qualify in a given week, render the section with an explicit "None this week" note rather than omitting it. +Run the two playbooks **in parallel as background sub-agents**, not sequentially. Serially, `both` +roughly doubles an already-long run; in parallel it costs about as much wall-clock as one report. -### Step 7 — Validate & write +1. Resolve the window once (`bootstrap-report.ps1 -App broker` prints it; capture `curEnd`). +2. Bootstrap **both** report files up front with the same `-EndDate`, so neither agent can drift: + ```pwsh + $S = '.\.github\skills\oncall-weekly-telemetry-report\assets\scripts' + & "$S\bootstrap-report.ps1" -App broker -EndDate <curEnd> + & "$S\bootstrap-report.ps1" -App authapp -EndDate <curEnd> + ``` +3. Launch two `general-purpose` background agents. Give each the **full** context it needs — the + playbook path, the resolved window dates, its bootstrapped report path, and its `_data` folder. + Require each agent to read **both** its playbook **and** § Shared hard rules in this file before + writing any HTML. Instruct each to do the work itself (not to advise), and to run its own + validator before reporting back. +4. Wait for both. Do not start the index until both validators pass. +5. Run `build-index.ps1`. +6. Report the three file paths to the user in chat. Do **not** paste report contents into chat. + +If one agent fails, still publish the other report and say plainly in chat which app failed and +why — a half-delivered rotation report beats a blocked one. -Run the bundled validator FIRST — it covers all the silent-failure cases this skill has tripped on in the past: +--- -```pwsh -.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\validate-report.ps1 -# defaults to most-recent oncall-wow-report-*.html under ~/android-oce-reports/ -# pass -Path explicitly to validate a specific file -``` +## Shared assets -The validator hard-fails on: -1. Stale `{{...}}` tokens or `EXAMPLE CONTENT BELOW` / `EXAMPLE_*` sentinels. -2. `devs` / `reqs` in user-facing text (KQL inside `<pre><code>` is exempted). -3. `U+FFFD` replacement characters (catches mojibake from emoji edits). -4. Unbalanced `<div>` depth in the Section 2 attention block (catches the inception-style nested-callout bug from past runs). -5. A second callout opening before the previous one closes (nested-callout sanity check). -6. **Chartless KPI grid** — if more than half the `.kpi` tiles lack a `data-spark` element (catches the v7 regression where the body was rebuilt without sparklines). Also warns when total chart count (sparks + trends + inline svgs) is < 15. -7. **Code-attribution depth** — each `.attr-card`'s "Code attribution" block must contain an `Originator` row (proxy for the full 8-field structure: Originator / Top throw site / Wrapper / Caller hot-spots / Underlying cause / Top error_messages / Likely PRs / Next step). Catches the v7-third-pass regression where cards shipped with a `pr-list`-only stub. -8. **Attribution-card layout guards (v8)** — the CSS must define `.attr-card { margin-bottom: 16px }` AND `.dim-row` overflow rules (`text-overflow: ellipsis` + `min-width: 0`). Catches the "cards touching" and "text bleeding out of dim boxes" regressions from a stale `<head>` block. -9. **Fabricated-sparkline heuristic (v8)** — warns when a `data-trend` array's peak value is < 100 (almost certainly hand-rolled rather than sourced from real data). See [`assets/queries/wow-table-sparkline-series.kql`](assets/queries/wow-table-sparkline-series.kql) for the canonical KQL that pulls real 8-week series for every code in the WoW tables. Its `<SPARK_START>` / `<SPARK_END>` tokens are the last **8 complete** Sun-Sat weeks (`<SPARK_END>` = `startofweek(today)`, exclusive) — deliberately distinct from the trend-chart's `<TREND_START>` / `<TREND_END>` (literal last 60 days ending today). Per-row sparklines stay on complete weeks so a partial final point doesn't create a misleading dip in every WoW row. - -Then: -- **Run the visual smoke test (recommended)** — catches rendered-layout bugs that pure HTML/CSS validation can't see: - - ```pwsh - .\.github\skills\oncall-weekly-telemetry-report\assets\scripts\visual-smoke.ps1 - # Opens the report at 1400px in headless Chromium via Playwright, captures a - # full-page screenshot to ~/android-oce-reports/_visual/, and runs DOM-based - # checks for: - # - element overflow inside .dim / .attr-card (catches "text bleeding out") - # - adjacent .attr-card pairs with gap < 8px (catches "cards touching") - # First run auto-installs Playwright + Chromium into %LOCALAPPDATA%\oce-skill-playwright - ``` -- Run `get_errors` on the HTML file (no errors expected — pure HTML/CSS). -- Verify no stale phrases from prior weeks remain (`Select-String` for retracted hypotheses, prior week's PR numbers). -- Verify every PR link in the new file is reachable (the file paths just before the link should match what `git log` returned). +| File | Purpose | +|---|---| +| [`scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1) | Bootstrap a report from its app's canonical template. `-App broker\|authapp`. Resolves + stamps the rolling window, creates `_data/<app>-<curEnd>/`, prunes old data, detects stub-vs-real collisions. | +| [`scripts/run-kql.ps1`](assets/scripts/run-kql.ps1) | Direct-REST Kusto helper — drop-in fallback when the Kusto MCP times out. `-App broker\|authapp` selects the cluster + database; `-Cluster`/`-Database` still override explicitly. | +| [`scripts/validate-report.ps1`](assets/scripts/validate-report.ps1) | Pre-publish validator. `-App broker\|authapp` selects the check profile. Shared checks (stale tokens, mojibake, div balance, sparkline coverage, header/filename date agreement) run for both. | +| [`scripts/build-index.ps1`](assets/scripts/build-index.ps1) | Build the combined `oce-index-<curEnd>.html` digest from two finished reports. | +| [`scripts/bucket-trends.js`](assets/scripts/bucket-trends.js) | Bucket any `{key, week, metric}` series into 60-day regression / spike / improvement / flat. App-agnostic — `--key=` selects the grouping column. | +| [`scripts/classify-novelty.js`](assets/scripts/classify-novelty.js) | Reads a `bucket-trends.js --json=` sidecar and labels each key **NEW / ACCELERATING / ONGOING / VOLATILE / RECOVERY / IMPROVING / STABLE** against its own 7-week baseline, plus family clustering. App-agnostic. **Mandatory in both playbooks** — it is what stops the attention section from being a volume-ranked list where a flat-but-huge code outranks a real step change, and it is also the **noise gate**: its `attention` set (`NEW` + `ACCELERATING`), plus at most 2 wins, is all that renders visibly with charts — everything else collapses into a fold. | +| [`scripts/agg.js`](assets/scripts/agg.js) | Per-key per-dim top-N rollup with WoW deltas. | +| [`scripts/find-suspect-prs.ps1`](assets/scripts/find-suspect-prs.ps1) | Parallel `git log -S` + `--grep` for a symbol. `-Repos` selects which repos to scan (broker/common, or authenticator). | +| [`scripts/visual-smoke.ps1`](assets/scripts/visual-smoke.ps1) | Optional Playwright layout smoke test — catches rendered-layout bugs (text bleed, cards touching) that HTML/CSS validation can't see. | +| [`templates/index-template.html`](assets/templates/index-template.html) | Canonical layout for the combined index page. | + +App-specific assets (queries, cheatsheets, report templates) are listed in each playbook. --- -## Hard rules - -- **Never `sum(countDevices)`.** Always `dcount_hll(hll_merge(countDevicesHll))`. Summing the per-row distinct count double-counts. -- **Always wrap view names in `materialized_view('Xxx')`** and use the canonical `Metrics`/`Updated` variants (see cheatsheet § 2). -- **Never sum percentiles.** Latency is a TDigest sketch — `percentile_tdigest(tdigest_merge(responseTimeTDigest), N, typeof(long))` only. -- **Always apply `MergeAccountType` / `MergeIsSharedDevice` / `MergeUiRequiredExceptions`** so this report agrees with the dashboard. -- **Confirm the week bucket label matches the user's intent** before writing the rest of the queries (Sunday-aligned). -- **Do NOT filter the partial in-progress week at the source in the 60-day trend queries** — the chart ends today and wants that partial week as its final bar. Exclude it from the regression/improvement **delta math** instead by running `bucket-trends.js --end=<startofweek(today)> --include-partial-end`: the `--end` cutoff drops the partial week from first/last/delta classification while `--include-partial-end` keeps it in the emitted `series`. Skipping `--end` (or the cutoff) would make `bucket-trends.js` show every error as a fake −99% improvement. The per-row `wow-table-sparkline-series.kql` is the exception — it keeps 8 complete weeks (`<SPARK_END>` = `startofweek(today)`, with the partial week filtered at the source) so no WoW row ends on a misleading partial dip. -- **Never carry a numeric telemetry value forward between runs.** Every KPI, table cell, delta %, device/request count, sparkline point, and verdict number must be re-pulled from Kusto for *this* run — never copied from a previous report, from a checkpoint/summary, from notes, or from memory. Telemetry shifts between runs and stale numbers read as fabricated. Near-miss precedent: a `no_tokens_found` count was about to be carried as ~23.7M when the actual current-window value was ~4.86M — a ~5× error that only the re-pull caught. If a number isn't backed by a query result file in this run's `_data/<end-date>/`, it does not go in the report. -- **Never hardcode the "Generated" date.** It is the *run* date in **UTC**, auto-stamped by `bootstrap-report.ps1` (which uses `(Get-Date).ToUniversalTime()`). If you rebuild the body programmatically, derive it live with a **UTC-date** formatter (`new Date().toISOString().slice(0,10)` in Node, `[datetime]::UtcNow.ToString('yyyy-MM-dd')` in PowerShell) — never paste a literal, and stay on UTC so the assembler can never stamp a different day than `bootstrap-report.ps1`. The v8 "Generated 2026-06-15 on a 2026-06-18 file" bug came from a hardcoded string in the assembler. (Reporting-week / baseline / 60d window dates are author-set and verified against the user's intended Sunday bucket — see template-readme "Date fields".) -- **Originator pre-check is mandatory.** A card cannot claim `Originator: Broker` without first running [`assets/queries/error-message-and-location.kql`](assets/queries/error-message-and-location.kql) and reading the throw site + top 3 `error_message` strings. If the throw site is in `common/ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult}` AND the message starts with `AADSTS`, the originator is **eSTS, not broker** — see the AADSTS reference in [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md). -- **WoW-movers pass is mandatory.** The 60d bucketer's `--peak-floor` silently drops sub-10K-device codes, so [`assets/queries/wow-movers.kql`](assets/queries/wow-movers.kql) MUST be run as a separate pass for both `error_code` and `error_type` (per Step 3d). Its output is **merged into the single 🔴 WoW regressions callout**, sorted by current-week device count descending, with rows tagged `NEW` / `60d↑` / originator chip. Do not render a separate "emerging" callout. Skipping the pass is how the Apr 26 `Failed to parse JWT` spike (7 → 3,461 devs over 7 weeks) hid for two reports running. -- **Section 2 callouts are at-a-glance, Section 4 is the deep dive.** WoW / Slow-burn / Wins items in Section 2 use the `.item` flat-row pattern (no nested cards, no per-item left bars — the parent `.callout` border is the only severity affordance). Each row is a single line of metric chips + a one-line body + an `Attribution card →` link to the corresponding `.attr-card` in Section 4. Do NOT duplicate the dim slicing, PR analysis, or detailed verdict between the two sections — Section 4 is where that lives. See [`assets/templates/template-readme.md`](assets/templates/template-readme.md) for the CSS class reference and the example `.item` markup. -- **Never use bash/PowerShell regex to bulk-edit balanced HTML.** This skill has burned twice on regex strip scripts that ate matched-pair `</div>` closes, producing inception-style nested-callout bugs that take a depth-tracking script to find. If you need a structural change to the HTML, make a targeted, single-occurrence string replacement (with explicit before/after context) or rewrite the affected block end-to-end. Never run a `-replace` across the whole file expecting it to leave balance intact. -- **Denominator caveat must cite evidence, not hand-wave.** If you flag a large all-spans device-count shift, run [`assets/queries/broker-version-share-wow.kql`](assets/queries/broker-version-share-wow.kql) (single WoW snapshot) or [`assets/queries/broker-version-share.kql`](assets/queries/broker-version-share.kql) (time-series) and name the version cohort the shift moved with. Do not write "recurring telemetry-shape artifact" without backing data; if you don't have it, drop the callout. -- **"Recovery" still merits a PR citation.** When an error pins to a single old broker version and recovers as that version retires, look for the **fix PR in the version that replaced it** before calling it a "natural rolloff." Often the fix is real and just under-credited. -- **Never report WoW-only verdicts** for errors that are flat-or-down WoW but rising on 60d — always cross-check both windows. -- **Never page** based on a regression that turns out to be a downstream of a denominator shift; always include the auth-only-denominator number alongside the all-spans number. -- **Always cite PRs** with full GitHub URLs (the repo URL patterns above), not bare commit SHAs. -- **Filename collision rule.** If a report file already exists for the same Sunday bucket, do not silently overwrite. Open the existing report, list its top-3 findings, and explicitly state in chat what changed in the new data before regenerating. A second run on the same week without a delta is wasted work. -- **No `devs` / `reqs` in user-facing strings.** All UI text — callouts, table headers, KPI labels, verdicts, badges — must say `devices` and `requests`. Internal variable / column / file names in scripts and JSON can stay short. -- **Do not create a separate Markdown summary** of the report — the HTML *is* the deliverable. -- **Do not commit** the report file. It lives in `$env:USERPROFILE\android-oce-reports\` (outside the workspace) precisely so it can't be staged accidentally. +## Shared hard rules + +These apply to **both** reports. App-specific hard rules (HLL device counting, TDigest percentiles, +`Merge*` helpers, the Originator pre-check, Authenticator `*DCount` columns, volume floors) live in +the playbooks — and **they are not interchangeable**. The Broker's "never `sum(countDevices)`" rule +is *actively wrong* on the Authenticator side, where `sum(SucceededDCount)` is the correct idiom. +Never carry a convention across the two playbooks. + +- **Never carry a numeric telemetry value forward between runs.** Every KPI, table cell, delta %, + device/request count, sparkline point, and verdict number must be re-pulled from Kusto for *this* + run — never copied from a previous report, from a checkpoint/summary, from notes, or from memory. + Telemetry shifts between runs and stale numbers read as fabricated. Near-miss precedent: a + `no_tokens_found` count was about to be carried as ~23.7M when the actual current-window value + was ~4.86M — a ~5× error that only the re-pull caught. If a number isn't backed by a query result + file in this run's `_data/<app>-<curEnd>/`, it does not go in the report. +- **Never hardcode the "Generated" date.** It is the *run* date in **UTC**, auto-stamped by + `bootstrap-report.ps1`. If you rebuild a body programmatically, derive it live with a **UTC-date** + formatter (`new Date().toISOString().slice(0,10)` in Node, + `[datetime]::UtcNow.ToString('yyyy-MM-dd')` in PowerShell) — never paste a literal. +- **⚠️ UTF-8 trap — DO NOT use PowerShell `@'...'@` heredocs to compose HTML containing emojis, + em-dashes, arrows, or middle dots.** PowerShell silently strips multi-byte UTF-8 characters when + piping heredocs to `Set-Content` / `Out-File`. Use Node (`fs.writeFileSync`), + `[IO.File]::WriteAllText($path, $text, [System.Text.UTF8Encoding]::new($false))`, or explicit + Unicode-pair literals (`[char]0xD83D + [char]0xDCCA` for 📊). The validator's `U+FFFD` check + catches the worst case (mojibake) but cannot detect characters silently stripped to nothing. +- **Never use regex to bulk-edit balanced HTML.** This skill has burned twice on regex strip scripts + that ate matched-pair `</div>` closes, producing inception-style nested-callout bugs that take a + depth-tracking script to find. Make targeted, single-occurrence string replacements (with explicit + before/after context) or rewrite the affected block end-to-end. +- **No shorthand in user-facing text.** All UI text — callouts, table headers, KPI labels, verdicts, + badges — says `devices` and `requests`, never `devs` / `reqs`. Internal variable, column, and file + names can stay short. +- **A moved metric is a question, not a verdict.** Never publish a regression verdict without the + app's diagnostic ladder having been walked (Broker: Originator pre-check + dim slicing; + Authenticator: volume floor + rate normalisation + error-reason decomposition). +- **Filename collision rule.** If a report already exists for the same end-date, do not silently + overwrite. Open it, list its top-3 findings, and explicitly state in chat what changed in the new + data before regenerating. A second run on the same window without a delta is wasted work. +- **Do not create a separate Markdown summary** of either report — the HTML *is* the deliverable. +- **Do not commit** any report. They live in `$env:USERPROFILE\android-oce-reports\` precisely so + they can't be staged accidentally. --- -## Output checklist - -- [ ] New `oncall-wow-report-YYYY-MM-DD.html` (where `YYYY-MM-DD` is the resolved `curEnd` — the end-date of the rolling 7-day window) exists at `$env:USERPROFILE\android-oce-reports\` (NOT at repo root). If a file for this end-date already existed, the chat session explicitly stated what changed before regenerating. -- [ ] All sections present and populated (incl. 🚚 Traffic Attribution — even if “None this week”) -- [ ] **60-day trend bucketing run on the full cross-product** — `{error_code, error_type} × {devices, requests}` = 4 runs — union of regressions reported. Per-request retry storms (e.g. small device pool, exploding request count) are flagged on both axes. Source KQL spans the literal last 60 days ending today (no source-side partial-week filter); the partial current week is excluded from delta classification via `bucket-trends.js --end=<startofweek(today)> --include-partial-end` and charted as the final bar. -- [ ] **WoW-movers pass run** ([`wow-movers.kql`](assets/queries/wow-movers.kql)) for BOTH `error_code` and `error_type`. Its output rows are **merged into the single 🔴 WoW regressions callout in Section 2** (sorted by curr-week devices descending), each row tagged `NEW` / `60d↑` / originator chip. No separate "emerging" callout. Every row carries throw-site, dominant message, originator, and a next step. If the WoW callout is empty (rare), render "None this week" rather than omit. -- [ ] **Both error-codes AND error-types WoW tables have `Δ requests %` and `Δ devices %` columns**, the 60d sparkline, and a status pill. Any row crossing threshold on either metric is in the regression list. -- [ ] Every WoW regression AND every 60d regression — **for both `error_code` and `error_type`** — has its own spike-attribution card with all 7 dimensions sliced. Cards are built from [`assets/templates/spike-card.html`](assets/templates/spike-card.html). -- [ ] **Every `error_type` regression card includes the 8th-dimension sub-code decomposition** showing the top 3–5 contributing `error_code`s with their Δ vs prior week, and links to those sub-codes' own attribution cards. -- [ ] **Originator pre-check has been run for every broker-tagged card** ([`error-message-and-location.kql`](assets/queries/error-message-and-location.kql)). Throw site and top 3 `error_message` strings are populated from real data, not from the code map. AADSTS-prefixed messages are tagged `eSTS`, not `Broker`. -- [ ] **Every regression card's Code Attribution block populates Originator + Top throw site + Wrapper + Caller hot-spots + Underlying cause + Top error_messages + Likely PRs (with confidence/why-it's-the-suspect) + Next step (with named owner)**. For type cards, the wrapper field focuses on the type's catch-and-rethrow site (e.g. `BaseException`, `ServiceException` constructor). Shallow PR-only attribution is not acceptable. -- [ ] Non-broker errors are explicitly tagged `environmental` / `non-broker` with confidence `none` — not invented broker PRs. -- [ ] Traffic analysis covers totals, per-app, per-span, requests-per-device ratio (per error AND overall), and a sampling-change check. -- [ ] **Every material traffic shift (>10% on any segment, up or down) has a reasoning paragraph** that names the dominant span/app/active-broker/broker-version, and either cites a causal PR (with confidence) — span removed/added, `goAsync()` refactor, sampling change, caller-side SDK release, ECS flight ramp — or explicitly says "no PR identified, suspect X" rather than leaving it unexplained. -- [ ] Denominator caveat (if used) is backed by [`broker-version-share-wow.kql`](assets/queries/broker-version-share-wow.kql) or [`broker-version-share.kql`](assets/queries/broker-version-share.kql) evidence naming the responsible version cohort. No hand-waving. -- [ ] Auth-only denominator used for all reliability %s, denominator caveat called out at top. -- [ ] No `\bdevs\b` or `\breqs\b` in user-facing text. (`Select-String -Pattern '\bdevs\b|\breqs\b' -CaseSensitive:$false` returns 0.) -- [ ] **Sparklines rendered.** Every `.kpi` tile in the Top-line health section has a `data-spark` array with 8–9 weekly values. Every row in the 60-day trend tables and both WoW tables (codes + types) has a `data-trend` mini-spark. Note the 60-day trend `data-trend` arrays now end on the current partial week (≈9 points incl. the in-progress bar), while the WoW-table sparklines keep 8 complete weeks. The validator's chart-coverage check passes (KPI coverage ≥1/2 of tiles, total elements ≥15). Past failure mode: the v7 body rebuild dropped all sparklines silently — see `template-readme.md` § "Sparklines are MANDATORY". -- [ ] **Code-attribution depth.** Every `.attr-card`'s Code attribution block uses the full 8-field `<div class="origin-row">` structure (Originator / Top throw site / Wrapper / Caller hot-spots / Underlying cause / Top error_messages / Likely PRs / Next step) per [`assets/docs/code-attribution-template.md`](assets/docs/code-attribution-template.md). A `pr-list`-only stub is **not acceptable** — the validator hard-fails this. Past failure mode (v7 third pass): all 10 cards shipped with PR-only stubs and lost the throw-site / wrapper / underlying-cause analysis. -- [ ] No stale text from previous weeks. (`Select-String -Pattern 'EXAMPLE CONTENT BELOW'` returns 0 — that's the unfinished-section sentinel. The template no longer ships `{{TOKEN}}` placeholders since v2; if the file still contains any `{{`, that's also a leftover.) -- [ ] `get_errors` clean on the HTML file. +## Router checklist + +- [ ] Mode resolved (`both` default) and stated in chat before work starts. +- [ ] Window resolved **once** and the same `-EndDate` passed to every bootstrap call. +- [ ] Each selected playbook read **in full** by the agent that owns it. +- [ ] In `both` mode, the two playbooks ran as **parallel** sub-agents, not interleaved in one context. +- [ ] Each report passed its own `validate-report.ps1 -App <app>` run. +- [ ] `oce-index-<curEnd>.html` built (in `both` mode) **after** both validators passed. +- [ ] File paths reported in chat; no report content pasted into chat; nothing committed. diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/docs/authapp-kusto-cheatsheet.md b/.github/skills/oncall-weekly-telemetry-report/assets/docs/authapp-kusto-cheatsheet.md new file mode 100644 index 00000000..b2b2eddd --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/docs/authapp-kusto-cheatsheet.md @@ -0,0 +1,235 @@ +# Authenticator Kusto cheatsheet + +Everything needed to write correct Authenticator KQL for the weekly OCE report. +Companion to [`kusto-cheatsheet.md`](kusto-cheatsheet.md), which covers the **Broker** side. + +> ## ⚠️ Read this first: Broker conventions do not transfer +> +> The two apps share a report skill, a report look, and nothing else in their data model. +> Every one of the Broker's headline hard rules is **wrong** here: +> +> | Broker rule | Authenticator reality | +> |---|---| +> | Never `sum(countDevices)` — always `dcount_hll(hll_merge(countDevicesHll))` | There are no HLL columns. Distinct devices are **pre-computed**: `sum(SucceededDCount)` is correct and is what the dashboard does. | +> | Never sum percentiles — use `percentile_tdigest(tdigest_merge(...))` | There are no TDigest sketches. The only latency source is the raw `brokeroperations` table, where plain `percentile()` is correct. | +> | Always apply `MergeAccountType` / `MergeIsSharedDevice` / `MergeUiRequiredExceptions` | These helper functions do not exist in this database. | +> | Time column is `EventInfo_Time` | Materialized views use **`EventDate`**. Raw tables use `EventInfo_Time` — except `brokeroperations`, which the dashboard filters on **`PipelineInfo_IngestionTime`**. | +> | Wrap views in `materialized_view('Xxx')` | Authenticator MVs are referenced by **bare name**: `Passkey_WebAuthN_Registration_MV_V1`. | +> | Unit of analysis is a flat `error_code` / `error_type` table | Unit of analysis is a **scenario funnel**: Initiated → Succeeded / Failed / Unknown. | +> | 7 slicing dimensions | **3** dimensions: `AppVersion`, `OsLevel`, `DeviceInfoMake`. | +> +> If you find yourself pattern-matching a Broker query into this database, stop and re-read. + +--- + +## 1. Connection + +| | | +|---|---| +| Cluster | `https://idsharedeus2.eastus2.kusto.windows.net` | +| Database | `d496be22d62a46b0a3cf67ea2e736fd8` | +| Auth | `az login` (same tenant as Broker) | + +Via the fallback REST helper: + +```pwsh +.\assets\scripts\run-kql.ps1 -App authapp -QueryFile .\assets\queries\authapp\scenario-outcomes-wow.kql -OutFile $data\scenarios.json +# equivalent to: +# -Cluster https://idsharedeus2.eastus2.kusto.windows.net -Database d496be22d62a46b0a3cf67ea2e736fd8 +``` + +Output shape is the array form: `{ "results": { "items": [ [colNames…], [row…], … ] } }` — the +same shape `bucket-trends.js` and `agg.js` already parse. + +--- + +## 2. The scenario → MV → column catalog + +Outcome columns each have a `…DCount` distinct-device twin. +**Registration / Authentication MVs expose only `Initiated / Succeeded / Failed` (+`DCount`) and +`TotalUniqueDevices`. There is NO `Cancelled` and NO `PartiallySucceeded` column** — do not +invent one. PN MVs carry only an initiated counter; the terminal outcome lives in the paired +`_Results_MV_V1`. + +| Scenario | Outcome MV | Initiate column | +|---|---|---| +| Passkey WebAuthN Registration | `Passkey_WebAuthN_Registration_MV_V1` | `Initiated` | +| Passkey InApp Registration | `Passkey_InApp_Registration_MV_V1` | `Initiated` | +| Passkey WebAuthN Authentication | `Passkey_WebAuthN_Authentication_MV_V1` | `Initiated` | +| Entra MFA Registration (QR) | `Entra_MFA_Registration_QR_Code_Flow_MV_V1` | `Initiated` | +| Entra MFA Registration (No-QR) | `Entra_MFA_Registration_Manual_Flow_MV_V1` **∪** `Entra_MFA_Registration_Non_QR_Code_Flow_MV_V1` | `Initiated` | +| Entra PSI Registration | `Entra_PSI_Registration_MV_V1` | `Initiated` | +| Entra PSI PN Registration | `Entra_PSI_Push_Notification_Registration_MV_V1` | **`RegistrationStarted`** | +| MSA NGC Registration | `Entra_MSA_NGC_Registration_MV_V1` | `Initiated` | +| MSA SA Registration | `Entra_MSA_SA_Registration_MV_V1` | `Initiated` | + +Push-notification families (two-stage, no Succeeded/Failed on the init MV): + +| Family | PN init MV | init column | PN results MV (`FinalResult`) | reacted column | +|---|---|---|---|---| +| Entra MFA PN+CFA | `Entra_MFA_Push_Notification_And_CheckForAuth_MV_V1` | `NotificationInitiated` | `Entra_MFA_Push_Notification_And_CheckForAuth_Results_MV_V1` | `RequestTimeInitiated` | +| Entra PSI PN+CFA | `Entra_PSI_Push_Notification_And_CheckForAuth_MV_V1` | `NotificationInitiated` | `Entra_PSI_Push_Notification_And_CheckForAuth_Results_MV_V1` | `RequestTimeInitiated` | +| MSA NGC PN+CFA | `Entra_MSA_Push_Notification_And_CheckForAuth_MV_V1` | `NotificationReceivedInitiated` | `Entra_MSA_Push_Notification_And_CheckForAuth_Results_MV_V1` | `SessionTimeInitiated` | +| MSA SA PN+CFA | same as NGC | same | same | same | + +**MSA NGC vs SA split:** both the init MV and the results MV carry `IsNGC` +(`"true"` → NGC, `"false"` → SA). **Apply the filter on BOTH sides of the join** — filtering only +one side silently mixes the two populations and the funnel stops reconciling. + +`FinalResult ∈ {Approved, Denied, Error}`. Completion = (Approved + Denied) ÷ initiated. +Approved / Denied / Error percentages are shares of the **reacted** total, not of initiated. + +--- + +## 3. "Unknown" is a real metric, not a rounding error + +``` +Unknown = max(0, Initiated − (Succeeded + Failed)) +``` + +A session that started and never produced a terminal result in the window. Causes range from +genuine user abandonment (walked away from the biometric prompt) to the app being killed, to a +result landing after the window closed. + +This is the **single most Authenticator-specific signal in the report** and it has no Broker +analogue. Report it as its own rate. A scenario whose success rate is flat while Unknown climbs +is degrading — the failures just are not being recorded as failures. + +Caveat to state in the report: a small Unknown floor is expected from window-edge truncation. +Only a *change* in the Unknown rate is a finding. + +--- + +## 4. The Errors companion views + +Named by inserting `Errors` before `_MV_V1`: + +``` +Passkey_WebAuthN_Registration_MV_V1 -> Passkey_WebAuthN_Registration_Errors_MV_V1 +Entra_MFA_Push_Notification_And_CheckForAuth_MV_V1 + -> Entra_MFA_Push_Notification_And_CheckForAuth_Errors_MV_V1 +``` + +Uniform schema: + +``` +EventDate, Error, OsLevel, AppVersion, DeviceInfoMake, ErrorCount, ErrorDCount, TotalUniqueDevices +``` + +Some views also expose a pre-formatted `ErrorBeautified` — prefer it for display when present, +but group on `Error` so the grouping is stable across views. + +**⚠️ Counts only — no denominator.** Always pair with the outcome MV's `Initiated`. An error +count that rose 30% alongside a 30% rise in initiates is traffic growth, not a regression. This +is the most common false positive on the Authenticator side. + +--- + +## 5. Drilling below the Errors views + +When `Error` is a coarse bucket, the raw `passkeyoperations` table has the finer code. + +- `OperationName` — `PasskeyCredentialRequest{Initiated,Succeeded,Failed}` plus sub-operations + like `PasskeyBeginGetCredential*` +- `AppInfo_Version`, `DeviceInfo_Make`, `DeviceInfo_Id`, `EventInfo_Time` +- `osLevel = tostring(split(DeviceInfo_OsVersion, " ")[0])` +- `AllProperties` is a **JSON string** — `todynamic()` it before indexing. + +Useful `AllProperties` keys: + +| Key | Notes | +|---|---| +| `RequestType` | `CreatePasskeyCredentialRequest` = registration · `GetPasskeyCredentialRequest` = authentication | +| `PasskeyFlow` | `WEB_AUTH_N_REGISTRATION` / `WEB_AUTH_N_AUTHENTICATION` / `IN_APP_REGISTRATION` | +| `Error`, `ErrorSource` | finer than the Errors MV bucket | +| `IsCrossDevice` | cross-device passkey flows behave differently — separate them | +| `DeviceUnauthenticatedErrorCode` | Android `BiometricPrompt` code. **5 / 10 / 13 / 14 = user abandonment** (cancel, timeout, negative button). **1 / 7 / 9 = device/hard error** (hw unavailable, lockout). Misreading abandonment as failure is the classic Passkey false alarm. | +| `DeviceUnauthenticatedErrorMessage`, `Source` | | + +**Know what a metric counts before you drill:** + +```kql +.show materialized-view Passkey_WebAuthN_Registration_MV_V1 | project Query +``` + +reveals the source table and the `OperationName` / `RequestType` / `PasskeyFlow` filters — e.g. +Registration MVs count only `CreatePasskeyCredentialRequest`, Authentication only +`GetPasskeyCredentialRequest`. Query the wrong request family in the raw table and the numbers +will not reconcile with the MV. + +--- + +## 6. `brokeroperations` — the one raw table in the main path + +The Broker-API responsiveness section reads it directly. Three traps: + +1. **Time column is `PipelineInfo_IngestionTime`**, not `EventDate` and not `EventInfo_Time`. + The wrong column returns an empty or badly skewed window with no error. +2. `BrokerApiName` and `BrokerApiElapsedTimeMs` are **inside** `AdditionalProperties` and must be + extracted, not projected: + ```kql + | extend ApiName = extract("BrokerApiName=([^,}]+)", 1, tostring(AdditionalProperties)) + | extend ElapsedMs = toint(extract("BrokerApiElapsedTimeMs=([0-9]+)", 1, tostring(AdditionalProperties))) + ``` +3. It is the **slowest query in the run**. Keep the window at 14 days and the projection narrow. + If the Kusto MCP times out, fall back to `run-kql.ps1`. + +Operation names: `BrokerApiCallInitiated` / `BrokerApiCallCompleted` / `BrokerApiCallFailed`. + +**Cross-report rule:** a regression here is a *shared* finding. Check the companion Broker report +for the same window before attributing it to Authenticator, and say in the write-up which report +the evidence came from. + +--- + +## 7. Cheap version resolution + +Do **not** `union *` to find the live versions — it is enormous. Read the highest-volume MV: + +```kql +Entra_MFA_Push_Notification_And_CheckForAuth_MV_V1 +| where EventDate >= datetime(<CUR_START>) and EventDate < datetime(<CUR_END>) +| where isnotempty(AppVersion) +| summarize Devices = sum(NotificationInitiatedDCount) by AppVersion +| order by Devices desc +``` + +`AppVersion` looks like `6.2606.3817` — not the Broker's `16.1.0` shape. + +**This is also the report's active-device proxy.** It counts devices that emitted at least one +Entra MFA push-notification event. Label it as a telemetry-active-device count. It is **not** a +product DAU and must not be printed as one. + +--- + +## 8. Volume floor + +Treat any scenario with **< ~1,000 initiates** in the window as noise. Rate swings on tiny +denominators are the leading source of false regressions in this report. Tag such rows +`low-volume` in the scoreboard and exclude them from the regression callout — but never delete +them from the scoreboard, because a scenario *dropping* into low-volume is itself a signal. + +--- + +## 9. Crash and stability data is NOT in Kusto + +Crash clusters live in **App Center only**. Use +[`../../release-monitoring-report/assets/scripts/fetch-appcenter-crashes.js`](../../../release-monitoring-report/assets/scripts/fetch-appcenter-crashes.js). +The crash section is **gated on the App Center token being available** — when it is not, render +the section's "Not collected this run" empty state rather than omitting the section or, worse, +estimating a crash rate from Kusto. There is no Kusto proxy for crash rate; do not invent one. + +--- + +## 10. Weekly bucketing + +`startofweek()` is **Sunday-aligned**, same as on the Broker side: +`startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. Print the distinct week values from the +first weekly query of the run and eyeball them. Off-by-one-week is the most common silent error +in weekly-bucketed KQL and it survives every other check in the pipeline. + +The 60-day trend deliberately **includes** the partial current week (it is the chart's final bar) +and excludes it from delta classification via +`bucket-trends.js --end=<startofweek(curEnd)> --include-partial-end`. +The 8-week sparkline series deliberately **excludes** it at the source. Both behaviours are +intentional and are not the same thing. diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/authapp.md b/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/authapp.md new file mode 100644 index 00000000..0d6b108a --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/authapp.md @@ -0,0 +1,608 @@ +# Authenticator playbook — weekly OCE telemetry report + +> Invoked by [`SKILL.md`](../../SKILL.md) in `authapp` or `both` mode. **Read the router first** — +> it owns the reporting-window resolution, output paths, mode routing, and the shared hard rules +> (UTF-8 trap, never-carry-a-number-forward, never bulk-regex-edit HTML, never-commit-the-report). +> This file owns everything Authenticator-specific. + +**Cluster:** `https://idsharedeus2.eastus2.kusto.windows.net` · +**Database:** `d496be22d62a46b0a3cf67ea2e736fd8` · +**Time column:** `EventDate` (MVs) · **Distinct devices:** pre-computed `sum(XxxDCount)` + +> ### ⚠️ Do not carry Broker conventions into this report +> The Broker playbook's first four hard rules are **actively wrong** here. There are no HLL +> columns (`sum(SucceededDCount)` is correct), no TDigest sketches, no `Merge*` helper functions, +> and views are referenced by bare name, not `materialized_view('…')`. The slicing space is 3 +> dimensions, not 7. If a query you are about to run looks like a Broker query, stop. +> **Before writing any KQL, read [`../docs/authapp-kusto-cheatsheet.md`](../docs/authapp-kusto-cheatsheet.md).** + +## What this report is about + +The Broker report is organised around **error codes** — a flat, high-volume failure taxonomy. +The Authenticator report is organised around **scenario funnels**: 13 user-visible flows +(Passkey, Entra MFA, Entra PSI, MSA NGC/SA — registration, authentication, and push-notification +variants). + +> ### ⚠️ There are TWO funnel shapes. Do not apply one to the other. +> +> The 13 scenarios do **not** share a single outcome model. Assuming they do is a real, observed +> failure mode: a run mapped push-notification outcomes onto the registration vocabulary by +> guesswork (`Approved`→"success", `Cancelled`→"unknown") and reported invented rates. +> +> | Shape | Applies to | Outcome model | +> |---|---|---| +> | **Outcome funnel** | the 9 registration / authentication scenarios | `Initiated → Succeeded / Failed`, with `Unknown = max(0, Initiated − (Succeeded + Failed))` | +> | **Reaction funnel** | the 4 `… PN+CFA` push-notification scenarios | two-stage: `Initiated → Reacted`, then `Reacted` split by `FinalResult` | +> +> **The PN scenarios have no `Succeeded`, no `Failed`, and no `Unknown` column.** Their +> `FinalResult` domain is `Approved · Denied · Error · Cancelled · ""` — verified live across all +> four families. `Approved`/`Denied`/`Error` are always present; `Cancelled` and `""` appear in +> some families only (MFA has both, PSI has `""`, the MSA pair had neither in a 14-day window), so +> never write a `case()` that assumes a fixed set. `pn-completion-wow.kql` deliberately keeps only +> `{Approved, Denied, Error}`; `Cancelled` and empty together are ~0.1% of MFA reacted volume and +> are not reactions that complete an auth. The canonical rates are: +> +> ```text +> CompletionRate = (Approved + Denied) / Initiated # share of notifications acted on +> ApprovedRate = Approved / (Approved+Denied+Error) # share of REACTIONS, not of Initiated +> DeniedRate = Denied / (Approved+Denied+Error) +> ErrorRate = Error / (Approved+Denied+Error) +> ``` +> +> **Denied is not a failure.** A user declining a push is a correct, healthy outcome — often the +> security-positive one. Never fold `Denied` into a failure rate, and never headline a rising +> `DeniedRate` as a regression without saying what it means. +> +> A drop in `CompletionRate` with a flat `ApprovedRate` means notifications are not reaching or not +> being acted on (delivery / lifecycle) — **not** that auth is failing. Say which one in the report. + +That difference drives everything. A finding here is "the Entra PSI registration funnel lost 3 +points of success rate and the loss landed in Unknown, concentrated on Android 14 Samsung +devices" — not "error code X rose". Write the report in those terms. + +### The canonical 13 + +These labels are the contract between the queries, the template, and the validator. Use them +**verbatim** — the validator fails a report where any one of them is missing from the scoreboard, +and renaming one here without renaming it in the query pack silently drops a row. + +| # | Scenario | Source | +|---|---|---| +| 1 | `Passkey WebAuthN Registration` | `scenario-outcomes-wow.kql` | +| 2 | `Passkey InApp Registration` | `scenario-outcomes-wow.kql` | +| 3 | `Passkey WebAuthN Authentication` | `scenario-outcomes-wow.kql` | +| 4 | `Entra MFA Registration (QR)` | `scenario-outcomes-wow.kql` | +| 5 | `Entra MFA Registration (No-QR)` | `scenario-outcomes-wow.kql` (union of Manual + Non-QR MVs) | +| 6 | `Entra PSI Registration` | `scenario-outcomes-wow.kql` | +| 7 | `Entra PSI PN Registration` | `scenario-outcomes-wow.kql` (initiate column is `RegistrationStarted`) | +| 8 | `MSA NGC Registration` | `scenario-outcomes-wow.kql` | +| 9 | `MSA SA Registration` | `scenario-outcomes-wow.kql` | +| 10 | `Entra MFA PN+CFA` | `pn-completion-wow.kql` | +| 11 | `Entra PSI PN+CFA` | `pn-completion-wow.kql` | +| 12 | `MSA NGC PN+CFA` | `pn-completion-wow.kql` (`IsNGC == "true"`) | +| 13 | `MSA SA PN+CFA` | `pn-completion-wow.kql` (`IsNGC == "false"`) | + +Rows 12 and 13 read the **same** MV pair, split only by `IsNGC`. That filter has to be applied on +both sides of the init↔results join or the two funnels quietly contaminate each other. + +## Authenticator asset map + +| File | Purpose | +|---|---| +| [`authapp-report-template.html`](../templates/authapp-report-template.html) | Canonical layout — a realistic populated example report. **Edit in place**; do not restyle. Shares the Broker report's CSS/sparkline system. | +| [`authapp-kusto-cheatsheet.md`](../docs/authapp-kusto-cheatsheet.md) | Scenario→MV catalog, column names, the Unknown metric, Errors views, `brokeroperations` traps, volume floor | +| [`queries/authapp/`](../queries/authapp/) | The eight canonical KQL templates — see [`queries/authapp/README.md`](../queries/authapp/README.md) | +| [`bucket-trends.js`](../scripts/bucket-trends.js) | 60-day trend bucketing. Run with `--key=scenario`. | +| [`classify-novelty.js`](../scripts/classify-novelty.js) | Splits movers into NEW / ACCELERATING / ONGOING / VOLATILE / RECOVERY / IMPROVING / STABLE against their own baseline. Run with `--family-sep=none`. **This is what makes Section 3 readable** — and it is the noise gate: its `attention` set (`NEW` + `ACCELERATING`), plus at most 2 wins, is all that renders visibly with charts. | +| [`agg.js`](../scripts/agg.js) | Per-key per-dim top-N rollup with WoW deltas | +| [`find-suspect-prs.ps1`](../scripts/find-suspect-prs.ps1) | `git log -S` / `--grep` — run with `-Repos authenticator` | +| [`bootstrap-report.ps1`](../scripts/bootstrap-report.ps1) | Bootstrap the report. Run with `-App authapp`. | +| [`validate-report.ps1`](../scripts/validate-report.ps1) | Pre-publish validator. Run with `-App authapp`. | +| [`run-kql.ps1`](../scripts/run-kql.ps1) | Direct-REST Kusto helper. Run with `-App authapp`. Signature is `-Query <kql-string> -Out <path.json>` (**not** `-File`/`-OutFile`). | +| [`fetch-appcenter-crashes.js`](../../../release-monitoring-report/assets/scripts/fetch-appcenter-crashes.js) | Crash clusters — **App Center only, not Kusto** | + +--- + +## Required sections (in order) + +1. **Top-line health KPIs** — telemetry-active devices, total scenario initiates, overall success + rate, overall **Unknown rate**, Broker-API success rate, crashes per 1,000 devices. WoW delta + on each, inline SVG sparkline on each. + +2. **Scenario scoreboard** — one table, **all 13 scenarios, every week, no exceptions**. Even + scenarios that did not move get a row; a silent scenario disappearing from the table is + indistinguishable from a scenario that was never checked. Columns: scenario, initiated, + success rate, Δ success (pts), failure rate, unknown rate, Δ unknown (pts), devices, 8-week + sparkline, status pill. Rows under the volume floor carry a `low-volume` tag. + + > **The 4 push-notification rows cannot fill the success/failure/unknown columns — that is + > expected, not a gap.** PN has no success/failure/Unknown model (see the outcome-model warning + > above; `Denied` is a healthy outcome, so a "failure rate" would be a lie). For the 4 PN rows + > put the **completion rate** in the success-rate column, the **error rate** in the failure + > column, and a literal `n/a` in the unknown-rate and Δ-unknown cells. Do **not** leave the cells + > blank (blank reads as "not measured") and do **not** synthesise an Unknown bucket for them. + > Footnote the table once: *"PN scenarios report completion/error; they have no Unknown state."* + +3. **Needs attention** — callouts using the `.item` flat-row pattern, ordered by **novelty, not + volume** (see Step 4b). Render the classifier's `attention` set (`NEW` + `ACCELERATING`) at the + top level, plus **at most 2** wins, and nothing else; `ONGOING` goes in a collapsed fold. + Budget **≤ 8 visible rows total, wins included** (`validate-report.ps1` check 17 warns above it + and counts every visible `.item` row in the section, wins among them). + - **🔴 New this week** — classifier label `NEW`: a flat baseline that just stepped, and that + cleared the volume floor. If nothing is `NEW`, say *"nothing new this week"* — do not backfill + with `ACCELERATING` or `ONGOING` scenarios. + - **🟠 Getting worse** — label `ACCELERATING`: already degraded **and still sliding**. This is + the only multi-week bucket that stays visible, because "is it getting worse?" is the one + question a known issue can still answer usefully. Delete the callout if the set is empty. + + > **When the classifier and the headline delta disagree, keep the row here and show both.** + > The classifier's "not falling" gate runs on **complete Sun–Sat calendar weeks**; the headline + > percentage-point delta runs on the **rolling 7-day** window. Different bases, and they + > legitimately disagree. That is not a reason to demote the row, rename the group, or hedge the + > heading — keep it exactly **"Getting worse"** and resolve it *in the row body*: *"Down 2.1 pp + > across the last three complete weeks; the rolling window shows +0.4 pp as the slide flattens. + > Still 3.8 pp below its own 60-day median."* The sparkline settles it visually. Do **not** + > invent a "needs verification" group. + - **🔵 Ongoing / known** — label `ONGOING`: degraded but level or easing. **Collapse into a + `<details class="fold">`** with a one-line summary ("N scenarios still below baseline, none + accelerating") and each row's `weeksElevated`. Still in the report, no longer competing with + the finding. + - **🟢 Wins** — scenarios that improved, with the cause where identifiable. **Cap at 2 rows, and + they count against the ≤ 8 visible-row budget.** A win is worth showing; a list of wins is padding. + - `VOLATILE` / `RECOVERY` ride as clearly-labelled trailing rows in the 🔴 callout, never + headlining a percentage (check 15). + + **⚠️ Every visible row carries its own 9-week `.item-spark`** holding the scenario's success-rate + series — including the wins, because a recovery is a shape claim too. Check 16 hard-fails a + visible row without one. Rows inside the fold are exempt. + ```html + <span class="item-name">Passkey WebAuthN Authentication</span> + <span class="item-spark trend" data-trend="[96.4,96.2,96.3,96.4,96.1,96.3,96.2,94.1]" + data-w="120" data-h="22" data-color="#cf222e"></span> + <span class="spark-cap">9 wk</span> + ``` + Colour: `#cf222e` worsening, `#bc4c00` accelerating, `#1a7f37` improving. Tag each row with its + label (`tag-new` / `tag-accel` / `tag-ongoing`) plus `elevated Nw` where `weeksElevated > 1`. + + **A quiet week is a valid outcome.** If the classifier reports `quietWeek: true`, lead with the + quiet-week banner from the template, keep the fold closed, and keep the report short. Do not + promote the biggest degraded-but-flat scenario to have something to show. + + **Low-volume scenarios do not belong here at all**, however large the percentage move — a 2× + swing on 820 devices goes in the appendix. Promoting it is exactly what teaches a reader to skim. + + Each row: name + sparkline + inline metric chips + tags pushed right + a one-line body + an + `Attribution card →` foot link. **At-a-glance only** — the dimension slicing and verdict live + in Section 5. Do not duplicate them here. + + > **Every row body must be specific to that row** — which scenario, what moved, from what to what, + > and whether it's news. One generic sentence repeated across rows makes the section unreadable; + > `validate-report.ps1` fails the report for it. + +4. **60-day per-scenario trend** — weekly-bucketed sparkline per scenario, first→last delta, and + a classification pill (regression / spike / improvement / flat) from `bucket-trends.js`. + +5. **Error attribution cards** — one `.attr-card` per regressed scenario. Each card carries: + (a) top error reasons with WoW deltas and a NEW flag, (b) **all three** dimension bars + (`AppVersion`, `OsLevel`, `DeviceInfoMake`), (c) a 4-field attribution block — Likely cause / + Concentration / Suspect PRs / Next step with a named owner. + +6. **Unknown / abandonment** — its own section. Define the metric inline for the reader, then + table it per scenario with Δ pts, devices affected, and an 8-week sparkline. + +7. **Push-notification reacted split** — the 4 PN families: initiated, reacted, completion rate, + Approved / Denied / Error shares with a stacked split bar, WoW deltas on each. + +8. **Broker API responsiveness** — per `BrokerApiName`: volume, success rate, p50/p95/p99, devices, + WoW deltas. **Must carry a visible cross-check note** pointing at the companion Broker report. + + > **The note is mandatory even when there is nothing to correlate — "no matching finding" is + > itself the finding.** Three cases, all of which produce a note: + > - **Match found** — name the Broker error code/type and its delta: *"`acquireTokenSilent` + > success −1.8 pp here; the Broker report flags `ipc_return_null_cursor` +52% over the same + > window. Same root cause, tracked there."* + > - **No match** — say so explicitly and draw the conclusion: *"No Broker-side error code moved + > materially this window, so this looks client-side (Authenticator's own IPC path or a caller + > change) rather than a Broker regression."* A silent omission reads as "not checked". + > - **Companion report unavailable** (`authapp`-only run, or the Broker run failed) — state that + > plainly: *"Broker report not generated this run; Broker-side correlation not performed."* + > Never imply a cross-check happened when it did not. + +9. **Crash & stability (App Center)** — crashes per 1,000 active devices and top crash clusters. + Rendered as "Not collected this run" when the App Center token is unavailable. + +10. **Version adoption + PR attribution** — version share table plus PR cards. PR links use the + **Azure DevOps** URL pattern, not GitHub. + +11. **Appendix** — query provenance: the `.kql` files used, cluster, database, resolved windows. + +--- + +## Step-by-step workflow + +### Step 1 — Bootstrap + +```pwsh +.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\bootstrap-report.ps1 -App authapp +# Optional: explicit end-date (curEnd, exclusive) + force overwrite +# .\bootstrap-report.ps1 -App authapp -EndDate 2026-07-02 -Force +``` + +It creates `$env:USERPROFILE\android-oce-reports\authapp-wow-report-<curEnd>.html` from +[`authapp-report-template.html`](../templates/authapp-report-template.html), creates +`_data/authapp-<curEnd>/`, stamps the window into the header, prunes data folders older than 60 +days, and prints every resolved token value. **Copy those token values down** — every query in +`queries/authapp/` consumes them. + +In `both` mode the router has already bootstrapped this file with a shared `-EndDate`. Do not +re-bootstrap; you would reset the header and lose the shared window. + +Edit the bootstrapped file **in place**. The template ships as a realistic populated report, not +a skeleton — walk it top to bottom and replace every example date, KPI, table row, verdict, and +PR citation with current data. The CSS, sparkline JS, section order, and card markup are +canonical: do not redesign them. + +### Step 2 — Denominator first + +Run [`version-share-wow.kql`](../queries/authapp/version-share-wow.kql) **before anything else**. + +A build ramping from 5% to 60% of the population moves every scenario rate at once without a +single line of scenario code having changed. If the version mix moved materially, every +downstream verdict has to be read against that fact — and you want to know before you write +thirteen verdicts, not after. + +Record the top 3 versions and their share delta. If a cohort moved more than ~10 share points, +say so explicitly in Section 1 and reference it from every affected card. + +### Step 3 — The scoreboard + +Run [`scenario-outcomes-wow.kql`](../queries/authapp/scenario-outcomes-wow.kql) and +[`pn-completion-wow.kql`](../queries/authapp/pn-completion-wow.kql). Between them they cover all +13 scenarios in two round-trips. + +For each scenario compute Δ success rate and Δ unknown rate **in percentage points, not percent +of percent**. A move from 92% to 89% is `−3.0 pts`, never `−3.3%`. Mixing the two is the fastest +way to make a report untrustworthy. + +Apply the volume floor: **< ~1,000 initiates in the window is noise.** Tag the row `low-volume`, +keep it in the scoreboard, and keep it out of the regression callout. A 12-initiate scenario +going 100% → 50% is two users. + +> A scenario **dropping into** low-volume is itself a finding — instrumentation may have broken. +> Flag that case explicitly rather than letting the row quietly go grey. + +### Step 4 — 60-day trend + +Run [`scenario-60d-trend.kql`](../queries/authapp/scenario-60d-trend.kql), then: + +```pwsh +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js $data\scenario-60d.json ` + --key=scenario --metric=devs --end=<startofweek(curEnd)> --include-partial-end ` + --peak-floor=1000 --summary +``` + +The query maps `errs` / `devs` to **bad outcomes** (`Failed + Unknown`), so the bucketer's +"rising = regression" semantics come out correct with no script change. Run it for **both** +`--metric=devs` and `--metric=reqs` and report the union of what each flags — a scenario where +device count is flat but event count explodes is a retry storm and only shows on one axis. + +Do **not** filter the partial current week at the source; `--end` excludes it from the delta math +while `--include-partial-end` keeps it as the chart's final bar. + +### Step 4b — Classify novelty (mandatory) + +`bucket-trends.js` says *what moved*. It cannot say *whether the movement is news* — and ranking the +attention section by volume is a known, reported defect that buries real step-changes under flat-but-huge +rows. Add `--json=` to the Step 4 runs, then classify: + +```pwsh +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\classify-novelty.js ` + $data\bucket-trends-devices.json --summary --floor=1000 --family-sep=none ` + --json=$data\novelty-devices.json +``` + +`--family-sep=none` is required for AuthApp: scenario names contain spaces (`Entra MFA Registration (QR)`), +so prefix-clustering on `_` is meaningless here. AuthApp findings are per-scenario, not per-family. + +Each scenario is classified against **its own history** on complete weeks only (first match wins): +`VOLATILE` (cv > 0.60) → `RECOVERY` (bouncing off a suppressed week) → `NEW` (ratio > 1.15 **and** cv < 0.25) +→ rising over the window (climb > 1.15), which splits into `ACCELERATING` (still deteriorating: +ratio > 1.10 **and** recent-block ratio > 1.10 **and** not falling this week) vs `ONGOING` (elevated but +level) → `IMPROVING` (ratio < 0.8) → `STABLE`. + +**⚠️ The `ACCELERATING` / `ONGOING` split is the noise gate.** Both mean "degraded for a while"; +only `ACCELERATING` means "and getting worse", which is the only reason to re-surface a known issue. +`attention = NEW ∪ ACCELERATING` — that set, and only that set, is rendered visibly with sparklines. +Read it straight off the sidecar: + +```jsonc +{ "attention": ["Passkey WebAuthN Authentication"], + "attentionLabels": { "Passkey WebAuthN Authentication": "NEW" }, + "quietWeek": false, + "counts": { "NEW": 1, "ACCELERATING": 0, "ONGOING": 2, "STABLE": 9, "VOLATILE": 1, "IMPROVING": 0 } } +``` + +If your Needs-attention section is longer than the `attention` array, you promoted rows the +classifier did not — that is the defect this step exists to prevent. + +> **Which series get classified: the outcome funnels only — the PN funnel is NOT run through the +> classifier.** Feed `classify-novelty.js` the **9 outcome-funnel bad-outcome series** and nothing +> else. The 13 scenarios in Section 5's scoreboard are **9 outcome funnels + 4 push-notification +> families**; only the 9 are classifiable. The PN families (Section 7) are deliberately excluded for +> two reasons: their `FinalResult` set has **two shapes** across the window so a weekly series is not +> comparable week-to-week, and **`Denied` is a healthy outcome** — a rising `Denied` share is a user +> correctly rejecting a prompt, which the classifier would read as a regression. Never let a PN +> family appear in Section 3's `attention` set. +> +> PN still gets trend treatment, just not novelty classification: chart each family's **completion +> rate** in Section 7 with its own sparkline and report the WoW delta there. If a PN family moves +> enough to be this week's story, say so in Section 7 and, if it warrants top-level visibility, +> reference it from the Section 1 executive summary — not by inserting it into Section 3. + +`weeksElevated` is **derived from the 9-week series, never persisted** — it counts consecutive recent +weeks above the early-window baseline, so it is identical on any machine and needs no state file. +Where the whole visible window is elevated the classifier sets `sustainedFullWindow: true`; phrase +that as *"below baseline for the entire visible window"* rather than inventing a week count. + +**Rows carrying `suppressRatio: true` (`VOLATILE`, `RECOVERY`) must not carry a WoW-percentage chip at all — +replace it with a `vs 60d median` chip.** Their WoW % +is an artifact of a depressed prior week, and a caveat in the body does not undo a large number sitting in the +chip row (`validate-report.ps1` check 15 hard-fails it). Give the absolute level and where it sits against the 60-day median. +AuthApp scenarios are mostly low-variance (cv 0.02–0.2), so a `VOLATILE` scenario is itself worth a sentence — +it usually means instrumentation is flapping, not that users are failing. + +**Two different WoW bases exist — do not conflate them.** The report's headline Δ is the rolling 7-day window; +the classifier's `WoW` is calendar Sun–Sat weeks. They legitimately disagree. Use novelty as *context* +("flat for seven weeks, first slip this week"), never as a competing delta number. + +### Step 5 — Sparklines + +Run [`scenario-sparkline-series.kql`](../queries/authapp/scenario-sparkline-series.kql) once. It +returns 8 **complete** weeks per scenario and feeds every `data-spark` / `data-trend` array in +Sections 1, 2, 4, and 6. + +Sparklines are **mandatory**, not decorative — the validator fails a report whose sparkline +coverage drops, because a body rebuild silently dropping them has happened before. + +### Step 6 — Attribution, per regressed scenario + +For each scenario in the 🔴 or 🟠 callouts (volume floor cleared): + +**6a.** [`scenario-errors-wow.kql`](../queries/authapp/scenario-errors-wow.kql) with +`<ERRORS_MV>` set to that scenario's Errors companion (insert `Errors` before `_MV_V1`). + +> ⚠️ The Errors views carry **counts only, no denominator**. Always divide by the scenario's +> `Initiated` from Step 3 before calling anything a rate. An error count up 30% alongside +> initiates up 30% is traffic growth. This is the most common false positive in this report. + +**6b.** [`scenario-errors-by-dim.kql`](../queries/authapp/scenario-errors-by-dim.kql), filtered +via `<REASON_FILTER>` to the reasons that actually moved. Read the output as **concentration**: + +| Pattern | Reading | +|---|---| +| One `AppVersion` holds most of the delta | Client regression — go find the PR (Step 7) | +| One `DeviceInfoMake` / `OsLevel` holds most of the delta | OEM or OS-version specific — often a platform API behaviour change, not our code | +| Delta spread evenly across all three dims | Service-side or population change, **not** a client regression — say so and do not manufacture a PR | + +**6c.** Where the reason is a coarse bucket, drill into raw `passkeyoperations` per the cheatsheet +§ 5. For Passkey scenarios specifically, check `DeviceUnauthenticatedErrorCode` before calling +anything a failure: **5 / 10 / 13 / 14 are user abandonment** (cancel, timeout, negative button); +**1 / 7 / 9 are real device errors**. Reporting abandonment as failure is the classic Passkey +false alarm. + +**6d.** Write the 4-field attribution block. Every field must be populated: + +| Field | Requirement | +|---|---| +| **Likely cause** | A specific mechanism, not a restatement of the metric. "Success rate fell" is not a cause. | +| **Concentration** | Which dimension, which value, what share of the delta. Quote the number. | +| **Suspect PRs** | ADO PR links with a confidence level and one line on *why* it is the suspect. `none` is a legitimate answer — write it rather than inventing a PR. | +| **Next step** | An action with a **named owner**. | + +### Step 7 — PR attribution (Azure DevOps, not GitHub) + +The Authenticator source lives at `authenticator/` in this workspace, backed by +**Azure DevOps**: `msazure / One / AD-MFA-phonefactor-phoneApp-android`. + +```pwsh +.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\find-suspect-prs.ps1 ` + -Repos authenticator -Symbol <ClassOrMethodName> -Since <curStart-30d> +``` + +Cite PRs with the full ADO URL: + +``` +https://msazure.visualstudio.com/One/_git/AD-MFA-phonefactor-phoneApp-android/pullrequest/<id> +``` + +**Never emit a `github.com` URL for an Authenticator PR** — it will 404 and it signals the +attribution was pattern-matched from the Broker playbook rather than actually researched. + +Scope the search window to changes that shipped in the version cohort that moved (from Step 2), +not to the reporting window — a regression surfaces when a build reaches users, which lags the +merge by weeks. + +### Step 8 — Broker API responsiveness + +Run [`broker-api-responsiveness-wow.kql`](../queries/authapp/broker-api-responsiveness-wow.kql). +It is the slowest query in the run — start it while writing up Step 6. + +Traps (all three are silent failures): the time column is **`PipelineInfo_IngestionTime`**; +`BrokerApiName` and `BrokerApiElapsedTimeMs` must be **extracted** from `AdditionalProperties`; +if the Kusto MCP times out, fall back to `run-kql.ps1 -App authapp`. Its parameters are +`-Query <kql-string>` and `-Out <path.json>` — both mandatory, and it takes the **query text**, not +a `.kql` file path, so read the file in first: + +```pwsh +$S = '.github\skills\oncall-weekly-telemetry-report\assets\scripts' +$q = [IO.File]::ReadAllText("$A\queries\authapp\scenario-outcomes-wow.kql") # then substitute the window tokens +& "$S\run-kql.ps1" -App authapp -Query $q -Out "$D\scenario-outcomes-wow.json" +``` + +**This section is the seam between the two reports.** A regression here is a shared finding: +check the companion Broker report for the same window before attributing it to Authenticator, and +name which report the evidence came from. In `both` mode, if the Broker agent found a matching +error-code spike, both reports should reference each other. + +### Step 9 — Crash & stability + +```pwsh +# The script lives in the sibling release-monitoring-report skill and takes a SUBCOMMAND +# (groups | enrich | diff) plus --owner/--app/--version. There is no bare "--days" form. +# --version is the current production Authenticator version from Step 2's version-share query. +$FC = '.github\skills\release-monitoring-report\assets\scripts\fetch-appcenter-crashes.js' +node $FC groups --owner authapp-t7qc ` + --app Microsoft-Authenticator-Android-Prod-App-Center ` + --version <CURRENT_PROD_VERSION> --days 14 --top 15 ` + --out "$env:USERPROFILE\android-oce-reports\_data\authapp-<curEnd>\crash-groups.json" +``` + +Auth comes from `--token-file <path>`, `$APPCENTER_API_TOKEN`, or +`~/.android-release-reports/appcenter.token`, in that order. **If none is present the section is +skipped — that is expected and is not a failure.** Render the empty state (below) and move on; +never block the report on the crash layer. + +Crash data is **App Center only — there is no Kusto source.** If the App Center token is +unavailable, render the section's "Not collected this run" empty state. Do **not** omit the +section (the reader cannot tell omission from zero crashes) and do **not** estimate a crash rate +from Kusto. There is no proxy; inventing one is worse than the gap. + +Normalise to **crashes per 1,000 telemetry-active devices** using the Step 2 device count, so the +number is comparable week over week as the population changes. + +### Step 10 — Validate + +```pwsh +.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\validate-report.ps1 -App authapp +# defaults to the most recent authapp-wow-report-*.html under ~/android-oce-reports/ +``` + +Then verify by hand: +- Every PR link is an ADO URL and resolves. +- No stale example text from the template survives (scenario names from the template's sample + data, example version numbers, example PR ids). +- All 13 scenarios are present in the scoreboard. + +--- + +## Hard rules + +> **Shared hard rules live in the router** — [`SKILL.md` § Shared hard rules](../../SKILL.md). +> They apply here too and are NOT repeated below: never carry a telemetry number forward between +> runs · never hardcode the Generated date · never compose report HTML via a PowerShell `@'...'@` +> heredoc (UTF-8 strip) · never bulk-regex-edit balanced HTML · no `devs`/`reqs` in user-facing +> text · same-end-date collision requires an explicit delta statement · no separate Markdown +> summary · never commit the report. **Read them before writing any HTML.** +> +> The rules below are Authenticator-specific and do **not** transfer to the Broker playbook. + +- **Novelty classification is mandatory, and Section 3 is ordered by it — never by volume.** Run + [`classify-novelty.js`](../scripts/classify-novelty.js) (Step 4b) and lead with `NEW`. Volume-ranking + the attention section is a known, reported defect — it puts flat-but-huge rows above real step-changes. + If `NEW` is empty, write "nothing new this week"; do not backfill it with `ONGOING` scenarios. +- **Section 3's visible rows are the classifier's `attention` set plus at most 2 wins — nothing else.** + `NEW` + `ACCELERATING` visible with sparklines; `ONGOING` inside a collapsed `<details class="fold">`. + Budget **≤ 8 visible rows total, wins included** (check 17 counts wins). The Broker report this + replaces shipped 13 visible rows with zero charts while the section below it carried 38 — the reader + could not tell which row was that week's story. +- **Every visible attention row carries a 9-week `.item-spark`**, wins included. Check 16 hard-fails + otherwise. The series is already in the trend sidecar — no extra query. Charts belong beside the + claim they support. +- **A quiet week is a valid outcome — publish it as one.** If `quietWeek: true`, show the quiet-week + banner and keep the report short. Padding the list with the biggest degraded-but-flat scenario is the + exact behaviour that teaches readers to skim. +- **The 60-day per-scenario table is the scoreboard, not a catalog — charting all ~13 scenarios there + is correct.** Check 18 allows 16 visible charts for `-App authapp` (vs 6 for broker, whose universe is + 40–50 error codes). Do **not** fan per-error-reason charts into that section; those belong in the + attribution cards. +- **A `VOLATILE`/`RECOVERY` scenario must not carry a WoW-percentage chip — swap it for `vs 60d median`.** Their WoW % + is an artifact of a depressed prior week, not a regression. Tagging the row and caveating in the body is **not + sufficient** — the chip row is read first. A Broker run shipped `429` tagged `VOLATILE`, body reading *"classified + volatile"*, and still rendered `+401.8%` in `metric up` styling; `validate-report.ps1` check 15 now hard-fails that + shape on both apps. Report the absolute level and its position against the 60-day + median instead. AuthApp scenarios normally sit at cv 0.02–0.2, so a genuinely `VOLATILE` scenario usually + means flapping instrumentation — call that out rather than reporting it as a user-facing failure. +- **No boilerplate in Section 3.** Every row body must be specific to that scenario — what moved, from what + to what, and whether it's news. One generic sentence repeated across rows makes the section unreadable and + `validate-report.ps1` fails the report for it. +- **Never `dcount_hll` / `hll_merge` / `percentile_tdigest` / `materialized_view('…')` / + `MergeAccountType` here.** None of them exist in this database. Distinct devices are the + pre-computed `…DCount` columns: `sum(SucceededDCount)`. +- **The MV time column is `EventDate`.** Raw tables use `EventInfo_Time`, except + `brokeroperations`, which the dashboard filters on `PipelineInfo_IngestionTime`. Using the wrong + one returns an empty or skewed window with **no error** — this is a silent failure, always + sanity-check the row count of the first query. +- **Registration / Authentication MVs have exactly `Initiated / Succeeded / Failed` (+`DCount`) + and `TotalUniqueDevices`.** There is no `Cancelled` and no `PartiallySucceeded`. Do not invent + a column; if the numbers do not reconcile, the shortfall is Unknown. +- **The 4 `… PN+CFA` scenarios do NOT have these columns.** They are a two-stage reaction funnel + keyed on `FinalResult` (`Approved · Denied · Error · Cancelled · ""`). Never compute a + success/failure/Unknown rate for a PN scenario, never fold `Denied` into failures, and never + reuse a registration-shaped rate formula on them. See § There are TWO funnel shapes. +- **`Unknown = max(0, Initiated − (Succeeded + Failed))` is a first-class metric.** Report it. + A scenario with flat success rate and rising Unknown is degrading — the failures simply are not + being recorded as failures. A small Unknown floor from window-edge truncation is expected; only + a *change* is a finding. +- **Errors views carry counts only.** Always pair with the outcome MV's `Initiated` before + quoting a rate. Never report a bare error-count delta as a regression. +- **Volume floor: < ~1,000 initiates is noise.** Tag `low-volume`, keep in the scoreboard, keep + out of the regression callout. A scenario *dropping into* low-volume is a separate finding. +- **Apply the MSA `IsNGC` filter on BOTH sides of the PN join.** Filtering only the init side or + only the results side silently mixes NGC and SA populations and the funnel stops reconciling. +- **Rate deltas are in percentage points.** 92% → 89% is `−3.0 pts`, never `−3.3%`. +- **Passkey abandonment is not failure.** Check `DeviceUnauthenticatedErrorCode` — 5/10/13/14 are + user abandonment, 1/7/9 are device errors — before writing a Passkey regression verdict. +- **Only three dimensions exist** (`AppVersion`, `OsLevel`, `DeviceInfoMake`). Do not go looking + for calling-app, account-type, or shared-device slices; they are Broker concepts. +- **PR links are Azure DevOps URLs**, never `github.com`. +- **All 13 scenarios appear in the scoreboard every week**, including flat and low-volume ones. +- **A regression spread evenly across all three dims is not a client regression.** Say + "service-side or population change, no client concentration" rather than manufacturing a PR. +- **Crash data comes from App Center or not at all.** No Kusto proxy, no estimate, no omission — + render the empty state. +- **Broker-API findings are cross-checked against the Broker report** before attribution. + +--- + +## Output checklist + +- [ ] New `authapp-wow-report-YYYY-MM-DD.html` (where `YYYY-MM-DD` is the resolved `curEnd`) exists + at `$env:USERPROFILE\android-oce-reports\` (NOT at repo root). If a file for this end-date + already existed, the chat session explicitly stated what changed before regenerating. +- [ ] All 11 sections present and populated. Sections with nothing to report render an explicit + "None this week" / "Not collected this run" state rather than being omitted. +- [ ] **Version-share query ran first** and its result is reflected in Section 1. If a cohort moved + >10 share points, the caveat is stated and referenced from every affected card. +- [ ] **All 13 scenarios** are in the scoreboard, each with initiated, success rate, Δ pts, + unknown rate, Δ pts, devices, sparkline, and a status pill. +- [ ] Volume floor applied — `low-volume` rows tagged, excluded from the regression callout, and + any scenario that *dropped into* low-volume is flagged as its own finding. +- [ ] **60-day bucketing run on both axes** (`--metric=devs` AND `--metric=reqs`, `--key=scenario`), + union of regressions reported, partial week charted but excluded from delta classification. +- [ ] **Novelty classification run** ([`classify-novelty.js`](../scripts/classify-novelty.js), Step 4b, + `--family-sep=none`). Section 3 leads with `NEW`, `ACCELERATING` sits in the 🟠 Getting-worse + callout, `ONGOING` is inside a collapsed fold, and no `VOLATILE`/`RECOVERY` row headlines a + percentage. Every row body is specific — no sentence repeats across rows. +- [ ] **Attention section is short and charted.** Visible rows == the classifier's `attention` set + (`NEW` + `ACCELERATING`), ≤ 8 of them, each with an `.item-spark` 9-week sparkline (wins too). + If `quietWeek: true`, the quiet-week banner is shown and nothing was promoted to fill the gap. +- [ ] Every regressed scenario that cleared the volume floor has an attribution card with **all + three** dimension bars and a fully populated 4-field block (Likely cause / Concentration / + Suspect PRs / Next step with a named owner). "No PR identified, suspect X" is acceptable; + an empty field is not. +- [ ] Error counts are normalised against `Initiated` — no bare count deltas presented as rates. +- [ ] Unknown/abandonment section populated with per-scenario rates, Δ pts, and sparklines. +- [ ] PN section covers all 4 families with completion rate **and** the Approved/Denied/Error split. + MSA NGC and SA are separated, with `IsNGC` applied on both join sides. +- [ ] Broker API section populated and carries the cross-check note naming the companion report. +- [ ] Crash section either populated from App Center or rendered as "Not collected this run". + No Kusto-derived crash estimate anywhere. +- [ ] Every PR link is `https://msazure.visualstudio.com/One/_git/AD-MFA-phonefactor-phoneApp-android/pullrequest/<id>`. + (`Select-String -Pattern 'github\.com'` finds nothing in the PR sections.) +- [ ] Rate deltas expressed in **percentage points** throughout. +- [ ] Sparklines rendered — every KPI tile has `data-spark`; every scoreboard, trend, and unknown + row has `data-trend`. The validator's coverage check passes. +- [ ] No `\bdevs\b` or `\breqs\b` in user-facing text. +- [ ] No stale template example content (sample version numbers, sample PR ids, sample verdicts). +- [ ] `validate-report.ps1 -App authapp` passes. +- [ ] `get_errors` clean on the HTML file. diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/broker.md b/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/broker.md new file mode 100644 index 00000000..9d7d8a46 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/broker.md @@ -0,0 +1,778 @@ +# Broker playbook — weekly OCE telemetry report + +> Invoked by [`SKILL.md`](../../SKILL.md) in `broker` or `both` mode. **Read the router first** — +> it owns the reporting-window resolution, output paths, mode routing, and the shared hard rules +> (UTF-8 trap, never-carry-a-number-forward, never-commit-the-report). This file owns everything +> Broker-specific: required sections, the 7-step workflow, Broker hard rules, and the Broker +> output checklist. + +**Cluster:** `https://idsharedeus2.kusto.windows.net` · **Database:** `ad-accounts-android-otel` · +**Time column:** `EventInfo_Time` · **Distinct devices:** `dcount_hll(hll_merge(countDevicesHll))` + +**Before writing any KQL, read [`../docs/kusto-cheatsheet.md`](../docs/kusto-cheatsheet.md).** + +## Broker asset map + +| File | Purpose | +|---|---| +| [`report-template.html`](../templates/report-template.html) | Canonical layout — a real prior-week report kept verbatim. **Edit in place**; do not restyle. See [`template-readme.md`](../templates/template-readme.md). | +| [`template-readme.md`](../templates/template-readme.md) | Author guide — what to change per week, colour palette, CSS class quick-reference | +| [`kusto-cheatsheet.md`](../docs/kusto-cheatsheet.md) | Schemas, helper funcs, gotchas, ready-to-paste KQL, AADSTS reference | +| [`code-attribution-template.md`](../docs/code-attribution-template.md) | Per-card checklist for the deep code-attribution block | +| [`queries/`](../queries/) | Canonical KQL templates, one per query — see [`queries/README.md`](../queries/README.md) | +| [`templates/`](../templates/) | Copy-paste HTML snippets (`spike-card.html`, `traffic-attr-card.html`, `sparkline-footer.html`) | +| [`bucket-trends.js`](../scripts/bucket-trends.js) | Bucket error codes into 60-day regression / spike / improvement / flat. Run with `--metric=devs` AND `--metric=reqs`. | +| [`classify-novelty.js`](../scripts/classify-novelty.js) | Split movers into NEW / ACCELERATING / ONGOING / VOLATILE / RECOVERY / IMPROVING / STABLE against their own 7-week baseline, and cluster related codes into families. **This is what makes Section 2 readable** — without it the attention list is volume-ranked noise — and it is also the **noise gate**: its `attention` set (`NEW` + `ACCELERATING`), plus at most 2 wins, is all that renders visibly with charts. | +| [`agg.js`](../scripts/agg.js) | Per-error per-dim top-N rollup with WoW deltas | +| [`summarize-attribution.js`](../scripts/summarize-attribution.js) | Roll up 7-dim attribution slices for spike-attribution cards | +| [`find-suspect-prs.ps1`](../scripts/find-suspect-prs.ps1) | Parallel `git log -S` + `--grep` across `broker/` + `common/` for a class/method symbol | +| [`validate-report.ps1`](../scripts/validate-report.ps1) | Pre-publish validator. Run with `-App broker` (the default). | +| [`run-kql.ps1`](../scripts/run-kql.ps1) | Direct-REST Kusto helper — fallback when the Kusto MCP times out | +| [`bootstrap-report.ps1`](../scripts/bootstrap-report.ps1) | Bootstrap a report from the canonical template. Run with `-App broker`. | +| [`visual-smoke.ps1`](../scripts/visual-smoke.ps1) | Optional Playwright layout smoke test | + +--- +## Required sections (in order) + +1. **Top-line health KPIs** — total requests, total devices, silent-auth reliability %, interactive reliability %, p95 latency on the hot spans. WoW delta on each. Inline SVG sparklines. +2. **Things that need attention this week** — callouts: + - **Denominator caveat** — explain any large total-spans device-count shift caused by span-emission changes (e.g. `goAsync()` refactors). Always state which denominator the report uses (auth-only: `SilentAuthStats` ∪ `InteractiveAuthStats`). + - **🔴 Regressions — grouped by NOVELTY, never by volume.** Built from [`classify-novelty.js`](../scripts/classify-novelty.js) (Step 3e), unioned with [`assets/queries/wow-movers.kql`](../queries/wow-movers.kql) so small-but-recent spikes land in the same grouping. + + > **The section has a hard budget: ≤ 8 visible rows, counting the wins.** The classifier's + > `attention` set (`NEW` + `ACCELERATING`) is typically 3–6 series out of 40–50 — that is the + > whole point — and you may add **at most 2** "Real wins" rows on top. Everything else goes into + > a collapsed fold. A previous report shipped **13 visible rows and zero charts** while the + > 60-day section below it carried **38 charts**; a reader could not tell which of the 13 was this + > week's story. `validate-report.ps1` check 17 warns above 8 rows and it counts **every** visible + > `.item` row in Section 2, wins included — so budget accordingly. + > + > **This budget caps VISIBILITY, never COVERAGE.** Step 5's rule that every regressed code/type + > must get an attribution card still holds in full — the surplus cards go into a collapsed fold, + > they are never dropped. See the boxed note under Step 5 for the exact resolution. If you ever + > find yourself deleting a mandated card to hit 8 rows, you have misread both rules. + + Emit these sub-groups **in this order**, omitting any that are empty: + + 1. **🆕 New this week** — label `NEW`: a genuinely boring baseline (cv < 0.25) that took a clean + step up. **These lead the section.** Typically 0–3 items. If there are none, say *"nothing new + this week"* explicitly — do **not** promote an `ACCELERATING` or `ONGOING` item to fill the slot. + 2. **🟠 Getting worse** — label `ACCELERATING`: already elevated, but **still climbing** (rising + over the window, ≥ 10% above its own median, and not falling WoW). This is the "known issue is + deteriorating" bucket and it is the *only* multi-week category that stays visible. + + > **When the classifier and the headline WoW disagree, keep the row here and show both numbers.** + > The classifier's "not falling" gate runs on **complete Sun–Sat calendar weeks**; the report's + > headline `Δ WoW` runs on the **rolling 7-day** window. These are different bases and they + > legitimately disagree — a code can be `ACCELERATING` on calendar weeks while showing a small + > rolling-window decline. That is *not* a reason to demote it, rename the group, or hedge the + > heading. Keep the group heading exactly **"Getting worse"**, and resolve it *in the row body*: + > *"Up 18% over the last three complete weeks; the rolling 7-day window shows −4% as the ramp + > flattens. Still ~30% above its own 60-day median — watch, don't close."* The sparkline settles + > it visually, which is why the row has one. Do **not** invent a "needs verification" group. + 3. **🔵 Ongoing / known** — label `ONGOING`: elevated but flat. **These go inside a collapsed + `<details class="fold">`**, summarised by one line ("N codes remain elevated, none accelerating"). + They are still in the report — a reader can open the fold — but they no longer compete with the + finding. Give each the number of weeks it has been elevated (`weeksElevated` from the classifier) + so a reader can see it is old news at a glance. + 4. **🔁 Volatile** — label `VOLATILE` (`suppressRatio: true`): high-variance series where a WoW + percentage is an artifact of a depressed baseline. **Delete the `Δ WoW` chip from the row head + and put a `vs 60d median` chip in its place** — a caveat in the body does not undo a `+401.8%` + sitting in the chip row, because the chip is what the eye reads first: + ```html + <!-- WRONG: the caveat below is invisible next to this --> + <span class="metric up"><span class="m-label">Delta WoW</span><span class="m-value">+401.8%</span></span> + <!-- RIGHT: absolute level + position in the 60-day band --> + <span class="metric"><span class="m-label">vs 60d median</span><span class="m-value">-94.5%</span></span> + ``` + `validate-report.ps1` check 15 **hard-fails** any `VOLATILE`/`RECOVERY` row that still carries a WoW chip ≥ 25%. + 5. **↩️ Recovery** — label `RECOVERY`: returning to its normal band after a suppressed week. Explicitly *not* a regression. + + **⚠️ Every visible row carries its own 9-week sparkline.** The shape is what separates a step + change from ordinary variance, and it must sit *next to the claim* — not in a separate browsable + section further down. Use the `.item-spark` pattern from the template: + ```html + <span class="item-name">ipc_return_null_cursor</span> + <span class="item-spark trend" data-trend="[41200,40800,41500,41100,40900,41500,52100]" + data-w="120" data-h="22" data-color="#cf222e"></span> + <span class="spark-cap">9 wk</span> + ``` + Colour by direction: `#cf222e` worsening, `#1a7f37` improving, `#9a6700` volatile. The series is + the same `comparable` array `classify-novelty.js` already read from the trend sidecar — you do not + run another query for it. `validate-report.ps1` check 16 **hard-fails** any visible attention row + without one. Rows inside the collapsed fold are exempt. + + **Novelty chips.** Tag each row with its label so the grouping survives skimming: + `<span class="tag tag-new">NEW</span>`, `tag-accel` for `ACCELERATING`, `tag-ongoing` for + `ONGOING`, plus `elevated Nw` where the classifier reports `weeksElevated > 1`. + + **Families outrank individuals.** If the classifier emits a `families` entry, render it as ONE row + naming the family and its members — related codes moving together are one root cause, not N + findings. Within each sub-group, order by the classifier's `ORDER`/novelty ranking, **not** by + current-week device volume. + + **Quiet weeks are a valid, good outcome.** If the classifier reports `quietWeek: true` (empty + attention set), lead the section with the quiet-week banner from the template and keep the fold + closed. Do **not** manufacture a headline finding — resist the pull to promote the largest flat + code. A short report that says "nothing new" is more trustworthy than a long one that pads. + + Each row uses the `.item` flat-row pattern (see `assets/templates/template-readme.md` § "Section 2 callouts"): name + sparkline + inline metric chips + tags pushed right + one-line body + optional foot with `Attribution card →` link. **Section 2 rows are at-a-glance only** — no dim slicing or PR analysis here; that belongs in the Section 4 card. Tags: `60d↑` (also rising on 60d) plus an originator chip (`broker` / `eSTS` / `Android` / `env`). + + > **Every row's one-line body must say something specific to that row** — what changed, from what to what, and why it is or isn't alarming. A sentence that would read identically on any other row (*"movement needs owner triage; deep dive below"*) is worthless and the validator will fail the report for it. If you have nothing specific to say, the row does not belong in Section 2. + - **Real wins this week**, with PR links. These carry sparklines too — a recovery is a shape claim. + **Cap at 2 rows, and they count against Section 2's ≤ 8 visible-row budget.** A win is worth + showing; a list of wins is padding. + - **Traffic shape** — flat / surge / collapse summary. +3. **📈 60-day cross-check** — a **slow-burn detector, not a browsing list**. Built from the + `ErrorStatsMetrics` materialized view over the **literal last 60 days ending today** (final bar = + current in-progress week). **Run the bucketing pipeline FOUR times — the cross-product of + `{error_code, error_type} × {devices, requests}`** — and union the regression sets. An entry (code + OR type) is flagged if it regresses on either metric. Deltas are computed on complete weeks only; + the partial current week is charted but excluded from classification. + + - **% of devices** affected (`devicesHit / authActiveDevices`) — catches errors hitting more users. + - **% of requests** affected (`errRequests / authTotalRequests`) — catches per-device retry storms (fewer users, more traffic per user). The previous report would have missed `kdfv2_key_derivation_error` (262 → 5,374 requests on ~57 devices) without this dim. + + > **⚠️ This section exists to catch what a 7-day window structurally cannot see: something that has + > crept up ~5%/week for eight weeks and never triggers a WoW alarm.** That is its *only* job. + > + > **Chart only the codes it promotes** — series flagged as rising on 60d that are **not already in + > Section 2**. In a typical week that is **0–3 rows**, and an empty result is the normal, healthy + > outcome; say "no slow burns this week" and move on. Everything else — the full classification of + > all 40–50 series — goes into a collapsed `<details class="fold">` **with no chart column at all**. + > + > Rationale: a previous report rendered 38 charts here, ~93% of which duplicated rows already in the + > error tables below, while the attention section above had none. Reviewing 38 long-elevated series + > every week is exactly the noise that trains an on-call engineer to skim. `validate-report.ps1` + > check 18 **hard-fails** if this section renders more than 6 charts outside a fold. + + Categories: True 60d regression / Ephemeral 60d spike (peak-then-recover) / True 60d improvement / Flat. Every **promoted** rising entry — whether `error_code` or `error_type` — gets the same Spike Attribution + Code Attribution treatment (Step 4 / Step 5); entries already covered in Section 2 are not re-analysed here, just cross-referenced. + + Always apply `MergeUiRequiredExceptions(error_type)` before bucketing on type; otherwise the 6+ string variants of `UiRequiredException` will each be tracked separately and skew the buckets. +4. **🔎 Spike Attribution** — one card per WoW regression AND per 60-day regression, **for both `error_code` and `error_type` regressions**. Each card slices on **all 7 dimensions** (broker version, span, active broker pkg, calling app, account type AAD/MSA, shared-device mode, client SKU). Each card ends with a **deep Code Attribution block** (see Step 4 for the required fields) and a Traffic Attribution verdict. +5. **🚚 Traffic Attribution** — top-level section listing every error whose spike is fully or partly explained by traffic volume from a specific calling app, rather than a code regression. If none qualify this week, render the section with an explicit "None this week" note. +6. **Error codes — WoW with stable denominator** — full table with `Δ requests %` and `Δ devices %` columns and the 60d sparkline. +7. **Error types — WoW with stable denominator** — full table, **same columns and rigor as the error-codes table** (`Δ requests %`, `Δ devices %`, 60d sparkline, status pill). Any regressing type also gets a spike-attribution card in Section 4. For composite types (e.g. `ClientException` is the umbrella for many sub-codes), include a **decomposition card** that breaks the WoW Δ down into the top 3 contributing sub-codes — so a `ClientException` −5 pp drop is explicitly attributed to e.g. `−8.5 pp timed_out_execution` + `−3.4 pp unknown_authority` + `−0.15 pp illegal_argument_exception`. + +> **📌 Sections 6 and 7 keep a sparkline on EVERY row — this is a deliberate exemption from the +> "charts follow findings" rule, decided explicitly. Do not strip them as part of noise reduction.** +> These are **lookup tables**, not a browsing section: the reader arrives with a code in mind, scans +> the `Error code` column for it, and the 60-day sparkline is glanceable context in a cell their eye +> is already on. It costs no extra attention. The noise problem the redesign fixed was the *60-day +> trend catalog* — a section you had to read top-to-bottom, ~93% of whose rows duplicated these very +> tables. Checks 16/17/18 deliberately scope to Section 2 and the 60-day section only; the +> `$totalCharts` count that these ~62 charts dominate is now a floor-only guard ("the charts didn't +> vanish"), never a ceiling. +8. **📊 Traffic analysis** — total requests/devices (WoW + 60d), top calling apps, top spans, **requests-per-device ratio** per error and overall (a rising ratio = retry storm; a falling ratio = caching gain), sampling-rate change indicator. +9. **Latency** — p50/p95/p99 by hot span. +10. **Broker version adoption** — week-over-week version share. +11. **Appendix** — query list and methodology. + +--- + +## Step-by-step workflow + +### Step 1 — Bootstrap the new report file from the template + +This skill ships with a canonical template at [`assets/templates/report-template.html`](../templates/report-template.html) (a real prior report kept as the reference layout). **Use [`assets/scripts/bootstrap-report.ps1`](../scripts/bootstrap-report.ps1)** to handle all the boilerplate (rolling-window computation, `_data/broker-<curEnd>/` directory, header stamping, retention-pruning, collision detection): + +```pwsh +.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\bootstrap-report.ps1 -App broker +# Optional: explicit end-date (curEnd, exclusive upper bound) + force overwrite +# .\bootstrap-report.ps1 -App broker -EndDate 2026-07-02 -Force +``` + +> `-App broker` is the default and may be omitted, but pass it explicitly in `both` mode so the +> two bootstrap calls in the run transcript are unambiguous. + +What it does: +* Resolves the rolling 7-day window from the system clock in UTC (`curEnd = today`, `curStart = curEnd - 7d`, `prevStart = curEnd - 14d`) — or from `-EndDate` if passed. +* Creates `~/android-oce-reports/oncall-wow-report-<curEnd>.html` from the canonical template. +* Creates `~/android-oce-reports/_data/broker-<curEnd>/` for raw KQL JSON payloads. +* **Stamps the resolved window into the report's `<title>`, `<div class="meta">` block, and Generated banner** — you never hand-edit header dates. The resolved window is echoed in the report header for transparency. +* Prunes `_data/broker-<old-end-date>/` folders older than 60 days so the cache doesn't accumulate. +* **Collision detection (fail-safe):** an existing same-day report is silently re-bootstrapped only when it is *positively* identified as an unpopulated stub — it still carries the `OCE-UNPOPULATED-STUB` sentinel that bootstrap injects **and** its first KPI still equals the template's value. Anything else (sentinel removed, or KPIs edited) is treated as real work: **HARD HALT, exit 2**, requiring `-Force` to overwrite. `validate-report.ps1` refuses to pass a report that still carries the sentinel, so a published report can never be misclassified as a stub. + +Edit the bootstrapped file in place — the template ships as a real prior-week report (not a tokenized skeleton). **Walk top-to-bottom and replace every prior-week date / KPI value / table row / verdict / PR citation with current-week data.** The CSS, sparkline JS, section ordering, and attribution-card markup are canonical — do not redesign them. See [`assets/templates/template-readme.md`](../templates/template-readme.md) for the full guide on what to change vs leave alone, the sparkline color palette, the CSS class reference, and the two v8 layout traps. + +> **⚠️ UTF-8 trap — DO NOT use PowerShell `@'...'@` heredocs to compose HTML content containing emojis, em-dashes, arrows, or middle dots.** PowerShell silently strips multi-byte UTF-8 characters when piping heredocs to `Set-Content` / `Out-File`. Use Node.js (`fs.writeFileSync`), `[IO.File]::WriteAllText($path, $text, [System.Text.UTF8Encoding]::new($false))`, or explicit Unicode-pair literals (`[char]0xD83D + [char]0xDCCA` for 📊) instead. This trap cost ~30 min in v8 and required a full emoji-restoration pass — every callout icon, every section header emoji, every arrow link had to be re-injected. The validator's `U+FFFD` check catches the worst case (mojibake replacement char) but cannot detect characters that were silently stripped to nothing. + +Mark any unfinished card or table cell with the literal sentinel `EXAMPLE CONTENT BELOW` inside an HTML comment — the final-pass validator (Step 7) greps for it. + +If the template ever needs structural improvements (new section, new card style, etc.), update `assets/templates/report-template.html` in the skill folder and commit it so future weeks inherit the change. + +### Step 2 — Pull WoW reliability data + +Use the Kusto MCP tool against: +- **Cluster:** `https://idsharedeus2.kusto.windows.net` +- **Database:** `ad-accounts-android-otel` + +**Always prefer the canonical `materialized_view('XxxMetrics' or 'XxxUpdated')` variants** — these are what the production dashboard uses, are pre-aggregated and HLL-bucketed, and avoid the 240 s MCP timeout that plain `android_spans` queries hit. Full schema, gotchas, and query templates: [`assets/docs/kusto-cheatsheet.md`](../docs/kusto-cheatsheet.md). + +> **Fallback when the Kusto MCP times out:** use [`assets/scripts/run-kql.ps1`](../scripts/run-kql.ps1). It acquires a token via `az account get-access-token`, POSTs directly to `/v2/rest/query`, and writes the result as a JSON file the JS helpers (`bucket-trends.js`, `summarize-attribution.js`) can consume directly. The skill's MCP-vs-REST switch is roughly: try the MCP once; if it returns `McpError -32001 (timeout)`, switch to the REST helper for the rest of the run. Run multiple queries in parallel via PowerShell `Start-Job`: +> +> ```pwsh +> $queries = @{ 'reliability.json' = $reliabilityKql; '60d-codes.json' = $codesKql; ... } +> $jobs = @() +> foreach ($f in $queries.Keys) { +> $q = $queries[$f] +> $jobs += Start-Job -ScriptBlock { +> param($Q, $O) & "$using:skillRoot\assets\scripts\run-kql.ps1" -Query $Q -Out $O +> } -ArgumentList $q, $f +> } +> $jobs | Wait-Job | Receive-Job; $jobs | Remove-Job +> ``` + +| Need | View | +|------|------| +| Per-error-code / per-error-type / per-span counts | `materialized_view('ErrorStatsMetrics')` | +| Total broker requests / devices | `materialized_view('BrokerAdoptionStatsUpdated')` | +| Silent auth reliability | `SilentAuthStatsAllRequestsMetrics` + `SilentAuthStatsRequestsWithoutExpectedErrorMetrics` | +| Interactive auth reliability | `InteractiveAuthStatsAllRequestsMetrics` + `InteractiveAuthStatsRequestsWithoutExpectedErrorMetrics` | +| Latency (p50/p95/p99) | `materialized_view('PerfStatsUpdated')` — use `percentile_tdigest(tdigest_merge(responseTimeTDigest), N, typeof(long))` | +| Broker version share | `BrokerAdoptionStatsUpdated` | +| Calling app share | `AppStatsUpdated` | +| SKU share | `SkuStatsUpdated` | +| Spike-by-flight slicing | `Operations_ByFlight`, `ErrorCodeBySpan_ByFlight`, `ErrorType_ByFlight` | + +Time filter: always use `EventInfo_Time` on materialized views. Use `PipelineInfo_IngestionTime` only on raw `android_spans`. + +**Three rules that will silently corrupt your data if violated** (full detail in the cheatsheet): + +1. **Distinct devices are HLL-encoded.** Use `dcount_hll(hll_merge(countDevicesHll))`, never `sum(countDevices)`. Summing double-counts every device that appears in more than one row. +2. **Apply the dashboard helper functions** so this report agrees with the dashboard: `MergeAccountType(account_type)`, `MergeIsSharedDevice(is_shared_device)`, `MergeUiRequiredExceptions(error_type)`. +3. **Auth-only denominator for reliability %s:** sum `countRequests` from `SilentAuthStatsAllRequestsMetrics` ∪ `InteractiveAuthStatsAllRequestsMetrics` — not total broker spans. Total span counts are sensitive to `goAsync()` / receiver refactors and will give false WoW reliability swings. + +### Step 3 — Pull 60-day trend + +Don't pre-filter to a hand-picked top-N list — small-but-rising errors (e.g. `null_pointer_error` at ~67K devices) will fall off and never show up in the trend section. Instead pull every error code **and every error type** with a meaningful baseline across the window, then bucket each. + +#### 3a. Per-error-code trend + +Use [`assets/queries/60d-trend-codes.kql`](../queries/60d-trend-codes.kql) (template; replace `<TREND_START>` and `<TREND_END>` tokens. **`<TREND_START>` = `curEnd − 60d`** and **`<TREND_END>` = `curEnd` (today), exclusive** — the literal last 60 days. `bootstrap-report.ps1` prints the resolved values): + +```kql +materialized_view('ErrorStatsMetrics') +| where EventInfo_Time >= datetime(<TREND_START>) and EventInfo_Time < datetime(<TREND_END>) +| where isnotempty(error_code) and error_code != 'success' +| summarize errs = sum(countOverall), + devs = dcount_hll(hll_merge(countDevicesHll)) + by week = startofweek(EventInfo_Time), error_code +| order by error_code asc, week asc +``` + +**Do NOT filter the partial in-progress week here.** The chart wants it as the final bar (the window ends today). The partial week is excluded from the regression/improvement **delta math** by `bucket-trends.js` via `--end=<TREND_CLASS_END> --include-partial-end` (see 3c), not at the source — a partial week driving the delta would read as a fake −99% improvement, which is exactly why classification and display are split in the JS. + +#### 3b. Per-error-type trend (same rigor) + +```kql +materialized_view('ErrorStatsMetrics') +| extend unified_error_type = MergeUiRequiredExceptions(error_type) +| where EventInfo_Time >= datetime(<TREND_START>) and EventInfo_Time < datetime(<TREND_END>) +| where isnotempty(unified_error_type) +| summarize errs = sum(countOverall), + devs = dcount_hll(hll_merge(countDevicesHll)) + by week = startofweek(EventInfo_Time), unified_error_type +| order by unified_error_type asc, week asc +``` + +`MergeUiRequiredExceptions` is mandatory — without it the 6+ string variants of `UiRequiredException` (raw, fully-qualified, com.microsoft.identity.common.exception.*) each show as separate rows and skew the buckets. + +#### 3c. Run the bucketer 4 times (cross-product of `{code, type} × {devices, requests}`) + +`bucket-trends.js` defaults to grouping by `error_code`. For the type runs you MUST pass `--key=unified_error_type` so it picks up the right column from the type-trend JSON. + +```pwsh +# Error codes — by devices, then by requests. +# TREND_START = curEnd - 60d (literal 60d start) +# TREND_CLASS_END = startofweek(today) ("Trend delta cutoff" printed by bootstrap) +# --include-partial-end charts the current partial week while excluding it from deltas. +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end --metric=reqs + +# Error types — by devices, then by requests (note --key) +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end --key=unified_error_type +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end --key=unified_error_type --metric=reqs +``` + +`--end` is `<TREND_CLASS_END>` = `startofweek(today)` (exclusive) — the Sunday that opens the current in-progress week. Weeks at or after it (the partial current week) are excluded from delta classification; `--include-partial-end` keeps that week in the emitted `series` so the chart ends today. The script also auto-detects partial end-buckets and warns if `--end` is omitted, but passing it explicitly is safer. + +Take the **union** of all four regression sets. Both `error_code` and `error_type` regressions get a spike-attribution card in Step 5. + +It will print regression / spike / improvement / flat buckets, sorted by peak. The thresholds (in case you need to tune): + +- **True 60d regression:** `delta > +15%` and trajectory is monotonic-ish (no single-week spike dominating). +- **Ephemeral 60d spike:** peak week is ≥3× the mean of the surrounding weeks (peak-then-recover shape). +- **True 60d improvement:** `delta < −15%`. +- **Flat:** otherwise. +- Codes/types with peak weekly devices `< 10K` (or peak weekly requests `< 100K` when `--metric=reqs`) are filtered out (`--peak-floor=N` to override). + +**Why both axes matter:** +- *codes × requests:* in v5, `kdfv2_key_derivation_error` spiked +1,951% on requests across only ~57 devices — a per-device retry storm device-only bucketing would have missed. +- *types × either:* `error_type` is the umbrella (e.g. `ClientException`, `ServiceException`, `UiRequiredException`) — a moving type that doesn't map cleanly to one moving code is a strong signal of a *new* sub-code being introduced or an existing one being reclassified (the v5 `ClientException` −10% drop was driven by `timed_out_execution` reclassification under PR #141, which would have been invisible from the codes table alone). + +**Always present side-by-side WoW tables for BOTH error_code AND error_type** with `Δ requests %` and `Δ devices %` columns; flag any row where either crosses threshold. + +#### 3d. WoW movers query — MANDATORY pass to catch small-base movers + +The 60d bucketer's `--peak-floor=10000` exists for good reason (otherwise the 60d regression list would be 200+ tiny noise codes), but it **silently drops every code whose absolute weekly volume stays under 10K** — even if that code is brand-new or just spiked 5× WoW. Real examples this skill has missed in the past: + +- `Failed to parse JWT` — went `7 → 32 → 54 → 46 → 55 → 892 → 3,461` over 7 weeks (2-week-old NEW spike, real broker code in `IDToken.parseJWT:38`). Never crossed the 10K floor. +- `Code:-11` — sat at ~1,030 devs/week for 7 weeks then jumped to 2,433 (+165% WoW). Sub-floor. +- `SSLHandshakeException` — devices flat at 260 but requests +186% WoW (per-device retry storm). The bucketer's reqs-axis floor (100K) just barely captures it but the device floor doesn't. + +To catch these, **always** run [`assets/queries/wow-movers.kql`](../queries/wow-movers.kql) **as a separate pass after the 60d bucketing**: + +```kql +// inputs: <CUR_END> = curEnd (exclusive), <CUR_START> = curEnd - 7d, +// <PREV_START> = curEnd - 14d. Printed by bootstrap-report.ps1. +// floor: cDev>=500 OR cReq>=5000 move: |Δd|>=25% OR |Δr|>=50% OR new-this-window +``` + +Run it **twice — once for `error_code`, once for `error_type`**. **Merge its output rows into the same regression callout as the standard WoW table**, then let Step 3e's novelty classification decide their grouping and order. The size split is implementation detail; what a reader needs first is *"is this new?"*, not *"is this big?"*. Do **not** sort the merged list by device count — that is exactly how a flat-but-huge code ends up above a real step change. + +For each WoW mover (regardless of size), you still owe the full Code Attribution treatment (Step 4). The dim-slicing pass (Step 5) is allowed to be deferred for sub-1K-device spikes if the throw-site + dominant message already pin the originator unambiguously — but say so explicitly in the card ("dims not yet sliced — file the bug first; pull dims if it persists"). + +### Step 3e — Classify novelty (what is actually NEW vs already-known) + +**This step is mandatory and it is what makes Section 2 readable.** `bucket-trends.js` tells you *what moved*; it cannot tell you *whether the movement is news*. Without this pass the attention section degenerates into a volume-ranked list where a flat-but-huge code leads and the real step change sits at position #9. + +Run it over each sidecar `bucket-trends.js` wrote (`--json=<path>`): + +```powershell +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\classify-novelty.js <codes-devs-buckets.json> --summary --json=<codes-devs-novelty.json> +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\classify-novelty.js <types-devs-buckets.json> --summary --json=<types-devs-novelty.json> +``` + +Each key is classified against **its own history**, using complete weeks only (first match wins): + +| Label | Rule | What it means for the report | +|---|---|---| +| `VOLATILE` | `cv > 0.60` | Series swings wildly. **Suppress the ratio** — a WoW % here is meaningless. | +| `RECOVERY` | prior week `< median × 0.5` **and** now back near median | Bounce-back off a suppressed week, not a regression. **Suppress the ratio.** | +| `NEW` | `ratio > 1.15` **and** `cv < 0.25` | Boring baseline, clean step up. **This is the news.** Visible + charted. | +| `ACCELERATING` | rising over the window (`climb > 1.15`) **and** `ratio > 1.10` **and** `recentRatio > 1.10` **and** not falling WoW | Known issue that is **still deteriorating**. Visible + charted. | +| `ONGOING` | rising over the window, but level or easing now | Elevated and flat. **Collapse into the fold** with its `weeksElevated` count. | +| `IMPROVING` | `ratio < 0.8` | | +| `STABLE` | otherwise | | + +**⚠️ The `ACCELERATING` / `ONGOING` split is the whole noise fix.** Both are multi-week elevated +series, and lumping them together is what produced a 13-row attention list. `ACCELERATING` answers +"is this getting *worse*?" — the only reason a known issue deserves the on-call engineer's eye a +second time. Everything else that is merely still-elevated goes in the fold. + +**The classifier decides what gets a chart.** `attention = NEW ∪ ACCELERATING` — that set, and only +that set, is rendered visibly with sparklines. The sidecar exposes it directly: + +```jsonc +{ "attention": ["ipc_return_null_cursor", "access_denied"], // render these, with charts + "attentionLabels": { "ipc_return_null_cursor": "NEW", "access_denied": "ACCELERATING" }, + "quietWeek": false, // true => nothing to headline + "counts": { "NEW": 3, "ACCELERATING": 1, "ONGOING": 9, "STABLE": 35, "VOLATILE": 3, "IMPROVING": 2 } } +``` + +On the 2026-07-30 fixture that is **5 attention rows out of 53 series**. If your attention section is +much longer than the `attention` array, you promoted rows the classifier did not. + +**`weeksElevated` is derived, never persisted.** It counts consecutive recent weeks above the +*early-window baseline* (`median` of the first third), so it is identical on any machine and needs no +state file. Its known limit: with 7–9 weeks of history you cannot distinguish "elevated for 7 weeks" +from "normal at a high level" — the classifier sets `sustainedFullWindow: true` for those, and the +correct phrasing is *"elevated for the entire visible window"*, not a hard week count. + +**Two guards that exist because they were violated in real runs:** +- **`ratio > 1.10` on `ACCELERATING`** — `IntuneAppProtectionPolicyRequiredException` (cv 0.08, flat, + **down 3.7% WoW**, only 4.9% above its own median) was labelled `ACCELERATING` by a slow drift in + block means and led the whole types list. A code within 10% of its own median is not this week's + story regardless of slope. +- **`cur >= prev * 0.95`** — a series that is *falling* this week cannot be "getting worse", even if + the multi-week trend is up. + +**Why `cv < 0.25` gates `NEW`:** a series must have been genuinely boring before a jump counts as news. Without that guard a jittery code that happens to be up this week gets promoted over a real step change. + +**⚠️ The trap this exists to kill — a big WoW % off an anomalous baseline is not a regression.** Real 2026-07-30 data: + +``` +429 300,664 299,965 892,839 974,980 11,512 32,530 2,724 → 16,531 cv=1.07 +temporarily_unavailable 30,257 29,962 37,263 6,168 41,971 141 71 → 36,153 cv=0.80 +``` + +`429` was reported as **+397.8% WoW** — but it is **94.5% *below* its own 60-day median**; the ratio is measured off a 2,724 floor after a collapse from ~975K. `temporarily_unavailable` was reported as **+400.7%** — it merely returned to its normal ~36K band after two suppressed weeks. Both were headlined. Neither is a regression. **A WoW percentage is meaningless whenever the *prior* week was itself anomalous.** + +Meanwhile the week's actual story classified as `NEW` and was buried at report positions #6/#9/#10: + +``` +ipc_return_null_cursor 43,093 43,950 43,552 44,571 43,759 42,117 41,473 → 52,129 cv=0.02 +ipc_operation_not_supported_on_server 19,503 19,957 20,074 21,457 21,466 20,849 20,575 → 24,050 cv=0.03 +ipc_connection_error 8,317 8,685 8,517 8,634 8,104 8,410 8,593 → 10,133 cv=0.02 +IPC FAMILY TOTAL 70,641 → 86,312 (+22.2%) +``` + +Three codes, each flat for seven straight weeks, all stepping up in the *same* week — one root cause in the IPC layer, reported as **one** finding. (`BrokerCommunicationException`, `NEW` on the type axis at +21.8%, is the same incident seen through the type dimension — say so rather than filing it twice.) + +**Families.** The classifier clusters keys sharing a prefix before `_` when ≥2 members share the same label. Report a family as ONE row. Error *types* are CamelCase and produce no families under `_` — that is correct, not a bug. + +**⚠️ Two different WoW bases exist — do not conflate them.** The report headline ΔWoW is a **rolling 7-day** window (`[CUR_START, CUR_END)` vs the 7 days before). The classifier's `WoW` is **calendar Sun–Sat weeks**. They legitimately disagree — `authorization_pending` read **+3.5%** rolling and **−37.1%** weekly on the same data. Use novelty as *history and context* ("flat for seven weeks, first step this week"), **never** as a competing delta number, or the report will appear to contradict its own tables. + +> **The division of labour, stated plainly so you do not have to derive it:** +> +> | Use the **rolling 7-day** numbers for… | Use the **calendar-week** classifier for… | +> |---|---| +> | Every KPI tile, table cell, and Δ% chip | Which rows are promoted (`attention` set) | +> | Any number a reader can see | Which label a row carries (NEW / ACCELERATING / …) | +> | The sentence "X rose N% this week" | The sentence "…and it has been climbing for six weeks" | +> +> **Rule: every *number* in the report comes from the rolling window; the classifier contributes +> *selection and narrative*, never a figure.** The one place the two meet is a row that is +> `ACCELERATING` on calendar weeks while the rolling delta is flat or negative — keep it in +> "Getting worse", keep the heading verbatim, and resolve it in the row body by stating both +> numbers and letting the sparkline settle it. Do not invent a hedged sub-group for these. + +--- + + +> ⚠️ **HARD RULE — Originator pre-check.** Before claiming `Originator: Broker` on any card, you MUST run [`assets/queries/error-message-and-location.kql`](../queries/error-message-and-location.kql) for that error code (or type) and read **(a) the throw-site stack and (b) the top 3 `error_message` strings**. Most broker error codes flow through `common/ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult, clientExceptionFromException}` — which intentionally bridge eSTS responses into broker exceptions. **If the throw site is in any of those three methods AND the error_message starts with `AADSTS`, the originator is eSTS, not broker.** See the AADSTS reference table in [`assets/docs/kusto-cheatsheet.md`](../docs/kusto-cheatsheet.md). Cards that skip this step must be marked low-confidence, not high. +> +> **Window:** use the FULL 7-day rolling window (`<CUR_START>` → `<CUR_END>`) on `PipelineInfo_IngestionTime`, NOT a narrower 3–5 day slice — low-volume types (e.g. `SSLHandshakeException`, `IntuneAppProtectionPolicyRequiredException`) routinely return zero rows in a sub-window slice. If a code/type still returns nothing, fall back to the prior 14 days (`<PREV_START>` → `<CUR_END>`) before declaring "no data". + +For every regression card, the Code Attribution block **must** populate the following fields. Shallow PR-citation only is not acceptable. Use [`assets/docs/code-attribution-template.md`](../docs/code-attribution-template.md) as the per-card checklist. + +| Field | What goes in it | How to find it | +|---|---|---| +| **Originator** | Where the error physically originates: broker code / common / Android system (WebView / Conscrypt / Keystore) / 3rd-party lib (Nimbus JWT, okhttp) / eSTS server / environmental (enterprise TLS interception). Use the colour-coded `origin-tag` spans (`origin-broker`, `origin-android`, `origin-thirdparty`, `origin-env`). | Grep the error string across `broker/`, `common/`, `msal/`. If no match, it's not our code — search the Android SDK or call out as eSTS-returned. | +| **Top throw site** | Fully-qualified file:line where the exception is constructed, plus the % of cases that throw from this single site. | Pull `error_location` / stack-prefix from `android_spans` for the spiking error code (one targeted query, narrow time window). Cite the dominant site. | +| **Wrapper** | Broker/common code that catches the originator's exception and re-throws it as the user-visible error code. Often `IDToken.parseJWT()`, `ServiceException(...)`, `ExceptionAdapter.exceptionFromAuthorizationResult()`. | Walk up the stack from the throw site — check for `try { ... } catch (X e) { throw new Y(...); }` patterns in broker/common. | +| **Caller hot-spots** | Top 1–3 callers of the wrapper, with device counts. Helps identify the specific code path the regression flows through. | `android_spans` slice by `error_location` (or `error.stack_trace` first frame inside our code). | +| **Underlying cause** | The proximate cause one level deeper (e.g. "99% `CertificateException` from `TrustManagerImpl.verifyChain`", "84% `no_such_algorithm` from `ProviderFactory.getMessageDigest`"). | `android_spans` slice by `error.cause` or `error_message` first 80 chars. | +| **Top error_messages** | Top 3–5 distinct `error_message` strings with counts. Often reveals the 3rd-party library or environmental signal (e.g. `net::ERR_SSL_PROTOCOL_ERROR`, Zscaler-issued cert names). | `summarize count() by tostring(error_message)` on raw `android_spans` filtered to the spike. | +| **Likely PRs** | 1–3 PRs with confidence rating (high / medium / low / none), full GitHub URL, commit SHA, author, AB#, and a 1-sentence **why-it's-the-suspect** justification (not just the title). Use the `pr-card` markup. | See PR-grep below. **Cite confidence honestly** — "none" is a valid verdict for environmental errors. | +| **Next step** | Concrete action with a named owner: who runs the next slice, who files the bug, what flight to flip, what correlation IDs to pull. | Pulled from PR authors / CODEOWNERS for the affected file. | + +#### PR-grep workflow + +**Read the full PR window first, then reason — don't `--grep` blind.** The 4-week window across `broker/` and `common/` typically returns <30 PRs total, small enough to read end-to-end. Targeted `--grep` matches will miss PRs whose titles don't mention the error string (most of them). **The recommended order is:** + +1. **Run plain `git log` on both repos** for the 4-week window. Read the resulting list end-to-end before any greps. +2. **Cross-reference titles + dates** against the Originator pre-check throw-site class. +3. **Only when you have a specific symbol** to chase (e.g. the throw-site class identified in step 2), reach for `find-suspect-prs.ps1` to do the symbol-targeted parallel pickaxe + grep. + +The historical mistake (pre-v8) was to jump straight to `find-suspect-prs.ps1` without reading the window first, which silently dropped PRs whose titles didn't mention the symbol. + +```pwsh +# Step 1: read the full 4-week window +cd c:\Users\shjameel\Repos\android-complete\broker +git --no-pager log --since='<windowStart>' --until='<windowEnd>' --pretty=format:'%h | %ai | %an | %s' --no-merges + +cd ..\common +git --no-pager log --since='<windowStart>' --until='<windowEnd>' --pretty=format:'%h | %ai | %an | %s' --no-merges +``` + +For each candidate PR, **read the diff** to confirm it touches the throw site / wrapper class identified in the Originator pre-check. Don't cite a PR just because the title mentions a related concept. + +```pwsh +# Step 3 (optional): symbol-targeted focused follow-up. Use ONLY after step 1 gave +# you a specific class/method name to chase from the Originator pre-check. +# Searches both repos in parallel via `git log -S` (pickaxe on diff) AND `--grep` (subject). +# Returns a unified table: repo | date | author | sha | PR# | URL | subject. +.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\find-suspect-prs.ps1 ` + -Symbol 'ExceptionAdapter' -Since 2026-04-01 -Until 2026-05-09 +``` + +#### Repo URL patterns for citations + +| Repo | URL pattern | +|------|-------------| +| `common/` | `https://github.com/AzureAD/microsoft-authentication-library-common-for-android/pull/<num>` | +| `broker/` | `https://msft.ghe.com/security/ad-accounts-for-android/pull/<num>` | +| `msal/` | `https://github.com/AzureAD/microsoft-authentication-library-for-android/pull/<num>` | +| `adal/` | `https://github.com/AzureAD/azure-activedirectory-library-for-android/pull/<num>` | + +#### Non-broker errors + +For errors with no broker code in the stack (Android system errors like `Code:-10`/`Code:-11`, OEM-specific keystore failures, eSTS-returned codes, environmental TLS interception), explicitly cite **"⚪ None — not in scope"** with confidence `none`, and explain *why* in the why-it's-the-suspect line. Do not invent broker PRs to fill the slot. Tag these errors as `environmental` or `non-broker` so they're tracked but don't page. + +### Step 5 — Spike attribution dimensions + +**Coverage rule: every `error_code` AND every `error_type` that lands in either the WoW regression list OR the 60-day regression list MUST get a spike-attribution card.** No silent skips. + +> **⚠️ Coverage and the ≤ 8 visible-row budget are NOT in conflict — they govern different things.** +> This is the most-reported ambiguity in the playbook, so read it carefully: +> - The **≤ 8 budget (§2) limits what is VISIBLE at the top level.** It is about what the on-call +> engineer is asked to read first. +> - The **coverage rule here limits what may be OMITTED.** It is about what must exist somewhere in +> the document, so a regression can never silently vanish. +> +> **Resolution: cards beyond the budget go into a collapsed fold, they do not get dropped.** Render +> the `attention` set (plus ≤ 2 wins) as visible cards, and put every remaining mandated card in a +> `<details>` fold titled *"Full attribution coverage (N more codes/types)"*. Coverage is satisfied +> by the card **existing and being reachable**, not by it being expanded on load. A run with 12 +> mandated cards and 7 visible rows is correct and expected — that is the design working, not a +> budget violation. +> +> Never resolve this the other way: do **not** expand Section 2 past 8 rows to fit the cards, and do +> **not** skip a mandated card to protect the budget. + +**`ErrorStatsMetrics` already carries `account_type` and `is_shared_device`** (use the `MergeAccountType` / `MergeIsSharedDevice` helpers to normalize) — so you do **not** need a fallback to raw `android_spans` for these dims. Earlier versions of this skill claimed otherwise; that was wrong. The only dim that requires `android_spans` is `DeviceInfo_OsVersion` (OEM/version slicing). + +Slice on **all 7 dimensions** for each spike. **Preferred for 2-week WoW attribution: one union query that covers all 7 dims for all regressions in a single round-trip** — see [`assets/queries/attr-union-by-dim.kql`](../queries/attr-union-by-dim.kql). Typical payload for 8 codes × 2 weeks × 7 dims is ~800 KB, well under the MCP limit. Pipe the result into `summarize-attribution.js --union <file.json>` (which prints per-dim top-N share + Δ devices + Δ requests for every code). Fall back to the per-dim form ([`attr-codes-by-dim.kql`](../queries/attr-codes-by-dim.kql)) only when (a) you need a wider time window, or (b) the union response exceeds payload size. + +For `error_type` cards, swap `error_code in (codes)` for `unified_error_type in (types)` and aggregate by the `MergeUiRequiredExceptions(error_type)` extension — otherwise everything else is identical. + +> **Low-volume fallback (extends Step 4's pre-check fallback to the 7-dim union):** when a code/type returns sparse dim rows in the 7-day rolling window — typical for sub-1k-device entries like `TimeoutCancellationException`, `JsonSyntaxException`, `kdfv2_key_derivation_error` — widen the union query to **14 days** (use `<PREV_START>` as the lower bound so the window becomes `[curEnd − 14d, curEnd)`) before declaring "broad — needs targeted slice". The added week of context usually surfaces enough rows to compute concentration percentages. If a code STILL has no concentration after 14 days, mark every dim cell as "not sliced — sub-window volume; file the bug first, slice on persistence" — do NOT fabricate "Broad" verdicts. + +| # | Dimension | Source | Cross-check | +|---|-----------|--------|-------------| +| 1 | Broker version | `ErrorStatsMetrics` group by `broker_version` | Cross-reference `BrokerAdoptionStatsUpdated` to see if the version's request share *also* moved that week — if yes, the spike is rollout-driven, not code-driven | +| 2 | Span name | `ErrorStatsMetrics` group by `span_name` | A single span hosting >60% of the error → strong code-path signal | +| 3 | Active broker package | `ErrorStatsMetrics` group by `active_broker_package_name` | E.g. CompanyPortal vs Authenticator vs LTW | +| 4 | Calling package | `ErrorStatsMetrics` group by `calling_package_name` | If 1–2 callers dominate, this is likely a traffic-attribution case (see Step 6) | +| 5 | Account type (AAD vs MSA) | `ErrorStatsMetrics`, `extend t = MergeAccountType(account_type)` group by `t` | If the split deviates significantly from fleet (~85% AAD / 15% MSA), call it out | +| 6 | Shared device mode | `ErrorStatsMetrics`, `extend s = MergeIsSharedDevice(is_shared_device)` group by `s` | Shared-device fleets have very different error profiles | +| 7 | OS version | [`assets/queries/os-version-slice.kql`](../queries/os-version-slice.kql) — raw `android_spans`, group by `DeviceInfo_OsVersion` | **On-demand only** — slice OS-version when EITHER (a) the wrapper class is in `ExceptionAdapter.clientExceptionFromException` (catch-all wrapping a system exception, where the OEM/version often is the cause), OR (b) the error code is one of `Code:-6`, `Code:-10`, `Code:-11`, `unknown_crypto_error`, `io_error`, `null_pointer_error`. Otherwise mark the dim row as "not sliced this week — no OEM concentration suspected" and move on. Slicing OS-version on every card wastes a raw-spans query without changing the verdict. | + +#### Type cards have one extra required dimension: sub-code decomposition + +Because `error_type` is an umbrella over many `error_code` values, every `error_type` regression card MUST also include an **8th dimension: sub-code breakdown** showing the top 3–5 `error_code`s rolled up under that type, with their device counts and Δ vs prior week. This lets the reader see whether the type-level move is driven by one sub-code or many — and routes the deep Code Attribution work to the right sub-code. + +```kql +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); +let target_types = dynamic(['ClientException', 'ServiceException']); +materialized_view('ErrorStatsMetrics') +| extend unified_error_type = MergeUiRequiredExceptions(error_type) +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd +| where unified_error_type in (target_types) +| extend week = iff(EventInfo_Time >= curStart, curStart, prevStart) +| summarize devs = dcount_hll(hll_merge(countDevicesHll)), + errs = sum(countOverall) + by week, unified_error_type, error_code +| order by unified_error_type asc, week asc, devs desc +``` + +Cite the dominant sub-codes inline in the type card's verdict (e.g. *"`ClientException` −10.2% drop is dominated by −8.5 pp `timed_out_execution` + −3.4 pp `unknown_authority`"*) and link to those sub-codes' own attribution cards. The deep Code Attribution block (Step 4) for the type card itself focuses on the **wrapper / catch-and-rethrow** path that defines the type (e.g. `BaseException.java`, `ServiceException.java` constructors), not on each sub-code. + +Feed the union JSON output into the summarizer (one round-trip): + +```pwsh +# Union mode (preferred). attr-union.json comes from attr-union-by-dim.kql. +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\summarize-attribution.js ` + --union attr-union.json --top=5 +# For type cards, add --key=unified_error_type +``` + +Legacy per-dim mode (one JSON per dimension) is still supported for the rare wider-time-window case: + +```pwsh +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\summarize-attribution.js ` + --label=span span.json ` + --label=calling_app app.json ` + --label=active_broker ab.json ` + --label=broker_version ver.json ` + --label=acct_type acct.json ` + --label=shared_dev shared.json ` + --label=client_sku sku.json +``` + +Ready-to-paste KQL for both forms: union → [`assets/queries/attr-union-by-dim.kql`](../queries/attr-union-by-dim.kql); per-dim → [`assets/docs/kusto-cheatsheet.md` § 8c](../docs/kusto-cheatsheet.md). + +**Concentration thresholds** (paint the dim bar red): +- > 80% in a single value → strong attribution (one root cause) +- 60–80% → medium attribution +- < 60% → broad / cross-cutting → say so explicitly, don't fabricate a single cause + +### Step 6 — Traffic analysis + traffic attribution + +Do this section in three parts. Traffic changes (up *or* down) need the same level of root-cause reasoning as error spikes — a uniform "−9% requests across all top apps with flat devices" is **not** a satisfactory verdict on its own; explain *why*. + +**6a. Top-line traffic shape.** Compare WoW *and* 60d for both totals and per-segment: + +```kql +materialized_view('BrokerAdoptionStatsUpdated') +| where EventInfo_Time > ago(70d) +| summarize totalReq = sum(countRequests), + totalDev = dcount_hll(hll_merge(countDevicesHll)) + by week = startofweek(EventInfo_Time) +| order by week asc +``` + +For each of the following, report direction + magnitude: +- Total requests (WoW %, 60d %) +- Total devices (WoW %, 60d %) +- Requests-per-device ratio (a drop often means a benign caching improvement; a spike often means a retry storm) +- Top 10 calling apps (`AppStatsUpdated`) — which apps drove the change? +- Top spans by request volume — did one span explode or collapse? +- Sampling-rate change indicator: if total spans moved >20% but auth-only device count moved <5%, suspect a sampling/instrumentation change. + +**6b. Reasoning for material traffic shifts (>10% on any segment).** For every span/app/active-broker that moved meaningfully WoW *or* 60d, run this slicing-and-correlation pass: + +| # | Question | How to check | +|---|---|---| +| 1 | **Is the move concentrated in one span?** | Slice top-10 spans by `Δreq` absolute and `Δreq %`. A >50% move on a single span almost always points to a code change (span added / removed / sampled / `goAsync()`-ed). | +| 2 | **Is the move concentrated in one calling app?** | Slice `AppStatsUpdated` WoW. A single app moving >20% in requests with flat devices = client-side caching/retry change in that app — escalate to that app's owners, not broker. | +| 3 | **Is the move concentrated in one active broker pkg?** | Slice `BrokerAdoptionStatsUpdated` by `active_broker_package_name`. AppManager (LTW) vs Authenticator vs Intune CP often diverge during a rollout. | +| 4 | **Is the move concentrated in one broker version?** | Cross-check against rollout share. If a span dropped −80% on `16.0.1` but is flat on `15.1.0`, the cause is in the 16.0.1 diff. | +| 5 | **Did anything else co-move?** | A span dropping while `OnUpgradeReceiver`-style downstream spans also drop (`SecretKeyWrapping`, `WrappedKeyAlgorithmIdentifier` in v5) confirms a single upstream change. | + +For every meaningful shift, **search for a causal PR** in the repos likely to affect telemetry shape: + +```pwsh +# Broker (span add/remove, goAsync, scope changes, sampling/exporter config) +cd c:\Users\shjameel\Repos\android-complete\broker +git log --since='<last8wks>' --oneline -i ` + --grep='span|goAsync|receiver|telemetr|otel|trace|metric|sampl|exporter' + +# Common (instrumentation surfaces) +cd ..\common +git log --since='<last8wks>' --oneline -i ` + --grep='span|telemetr|otel|trace|sampl|instrument' +``` + +**Causal PR categories that meaningfully shift traffic counts** (flag any of these): + +- **Span removed / renamed / scope-narrowed** → drops the span's count to zero or partial +- **`goAsync()` / `BroadcastReceiver` refactor** → broadcast may complete before async work flushes the span (this is the v5 PR #88 / `OnUpgradeReceiver` story — call it out as a precedent) +- **Sampling-rate change** in broker `Otel*` / `Telemetry*` exporter config or `common/` instrumentation → uniformly scales counts up or down across many spans +- **New span added** in a hot path → request counts for that span jump from ~0 to material +- **Caller-side SDK change** (MSAL/MSAL_CPP/OneAuth release) that batches or caches requests → uniform per-app request drop with flat devices +- **Flight rollout** (ECS) that gates a code path on/off → bursty changes in a specific span on specific dates + +Cite the suspect PR(s) with the same confidence ratings used in Code Attribution (high / medium / low / none) and the same `pr-card` markup. If you can't pin one down, say so explicitly — *"uniform 5–22% per-app request drop with flat devices, no telemetry-platform PR identified, suspect caller-side SDK change in MSAL release X.Y"* is acceptable; "traffic is flat" without checking is not. + +**6c. Per-error traffic attribution (is the *error* spike traffic-driven?).** For every error code flagged in Step 5 as a regression, additionally check whether the spike is *traffic-driven* rather than *failure-rate-driven*: + +```kql +let curEnd = datetime(<CUR_END>); +let curStart = datetime(<CUR_START>); +let prevStart = datetime(<PREV_START>); +let target_code = "<error_code>"; +materialized_view('ErrorStatsMetrics') +| where EventInfo_Time >= prevStart and EventInfo_Time < curEnd and error_code == target_code +| extend week = iff(EventInfo_Time >= curStart, curStart, prevStart) +| summarize errs = sum(countOverall), + devs = dcount_hll(hll_merge(countDevicesHll)) + by week, calling_package_name +| order by week asc, devs desc +``` + +If the spike is concentrated in a single calling app whose **overall** request volume also rose that week (cross-check `AppStatsUpdated`), and the **per-request failure rate is essentially flat**, classify the spike as a **traffic-attribution case** rather than a code regression: + +> Example: "`no_account_found` +60% devices this week is fully explained by Outlook's request volume rising 65% — the per-Outlook-request failure rate is unchanged. No broker code change is implicated." + +Add a top-level **🚚 Traffic Attribution** section that lists every error matched to a traffic-driven origin, mirroring the Code Attribution section. **Each card must include**: the dominant calling app(s) with their WoW request-volume delta, the per-app per-request failure rate (now vs prior — show it's flat), and the recommended owner to route to (typically the calling app's team, not broker). If no errors qualify in a given week, render the section with an explicit "None this week" note rather than omitting it. + +### Step 7 — Validate & write + +Run the bundled validator FIRST — it covers all the silent-failure cases this skill has tripped on in the past: + +```pwsh +.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\validate-report.ps1 -App broker +# defaults to most-recent oncall-wow-report-*.html under ~/android-oce-reports/ +# pass -Path explicitly to validate a specific file +``` + +The validator hard-fails on: +1. Stale `{{...}}` tokens or `EXAMPLE CONTENT BELOW` / `EXAMPLE_*` sentinels. +2. `devs` / `reqs` in user-facing text (KQL inside `<pre><code>` is exempted). +3. `U+FFFD` replacement characters (catches mojibake from emoji edits). +4. Unbalanced `<div>` depth in the Section 2 attention block (catches the inception-style nested-callout bug from past runs). +5. A second callout opening before the previous one closes (nested-callout sanity check). +6. **Chartless KPI grid** — if more than half the `.kpi` tiles lack a `data-spark` element (catches the v7 regression where the body was rebuilt without sparklines). Also warns when total chart count (sparks + trends + inline svgs) is < 15. +7. **Code-attribution depth** — each `.attr-card`'s "Code attribution" block must contain an `Originator` row (proxy for the full 8-field structure: Originator / Top throw site / Wrapper / Caller hot-spots / Underlying cause / Top error_messages / Likely PRs / Next step). Catches the v7-third-pass regression where cards shipped with a `pr-list`-only stub. +8. **Attribution-card layout guards (v8)** — the CSS must define `.attr-card { margin-bottom: 16px }` AND `.dim-row` overflow rules (`text-overflow: ellipsis` + `min-width: 0`). Catches the "cards touching" and "text bleeding out of dim boxes" regressions from a stale `<head>` block. +9. **Fabricated-sparkline heuristic (v8)** — warns when a `data-trend` array's peak value is < 100 (almost certainly hand-rolled rather than sourced from real data). See [`assets/queries/wow-table-sparkline-series.kql`](../queries/wow-table-sparkline-series.kql) for the canonical KQL that pulls real 8-week series for every code in the WoW tables. Its `<SPARK_START>` / `<SPARK_END>` tokens are the last **8 complete** Sun-Sat weeks (`<SPARK_END>` = `startofweek(today)`, exclusive) — deliberately distinct from the trend-chart's `<TREND_START>` / `<TREND_END>` (literal last 60 days ending today). Per-row sparklines stay on complete weeks so a partial final point doesn't create a misleading dip in every WoW row. + +Then: +- **Run the visual smoke test (recommended)** — catches rendered-layout bugs that pure HTML/CSS validation can't see: + + ```pwsh + .\.github\skills\oncall-weekly-telemetry-report\assets\scripts\visual-smoke.ps1 + # Opens the report at 1400px in headless Chromium via Playwright, captures a + # full-page screenshot to ~/android-oce-reports/_visual/, and runs DOM-based + # checks for: + # - element overflow inside .dim / .attr-card (catches "text bleeding out") + # - adjacent .attr-card pairs with gap < 8px (catches "cards touching") + # First run auto-installs Playwright + Chromium into %LOCALAPPDATA%\oce-skill-playwright + ``` +- Run `get_errors` on the HTML file (no errors expected — pure HTML/CSS). +- Verify no stale phrases from prior weeks remain (`Select-String` for retracted hypotheses, prior week's PR numbers). +- Verify every PR link in the new file is reachable (the file paths just before the link should match what `git log` returned). + +--- + +## Hard rules + +> **Shared hard rules live in the router** — [`SKILL.md` § Shared hard rules](../../SKILL.md). +> They apply here too and are NOT repeated below: never carry a telemetry number forward between +> runs · never hardcode the Generated date · never compose report HTML via a PowerShell `@'...'@` +> heredoc (UTF-8 strip) · never bulk-regex-edit balanced HTML · no `devs`/`reqs` in user-facing +> text · same-end-date collision requires an explicit delta statement · no separate Markdown +> summary · never commit the report. **Read them before writing any HTML.** +> +> The rules below are Broker-specific and do **not** transfer to the Authenticator playbook. + + +- **Never `sum(countDevices)`.** Always `dcount_hll(hll_merge(countDevicesHll))`. Summing the per-row distinct count double-counts. +- **Always wrap view names in `materialized_view('Xxx')`** and use the canonical `Metrics`/`Updated` variants (see cheatsheet § 2). +- **Never sum percentiles.** Latency is a TDigest sketch — `percentile_tdigest(tdigest_merge(responseTimeTDigest), N, typeof(long))` only. +- **Always apply `MergeAccountType` / `MergeIsSharedDevice` / `MergeUiRequiredExceptions`** so this report agrees with the dashboard. +- **Confirm the week bucket label matches the user's intent** before writing the rest of the queries (Sunday-aligned). +- **Do NOT filter the partial in-progress week at the source in the 60-day trend queries** — the chart ends today and wants that partial week as its final bar. Exclude it from the regression/improvement **delta math** instead by running `bucket-trends.js --end=<startofweek(today)> --include-partial-end`: the `--end` cutoff drops the partial week from first/last/delta classification while `--include-partial-end` keeps it in the emitted `series`. Skipping `--end` (or the cutoff) would make `bucket-trends.js` show every error as a fake −99% improvement. The per-row `wow-table-sparkline-series.kql` is the exception — it keeps 8 complete weeks (`<SPARK_END>` = `startofweek(today)`, with the partial week filtered at the source) so no WoW row ends on a misleading partial dip. +- **Originator pre-check is mandatory.** A card cannot claim `Originator: Broker` without first running [`assets/queries/error-message-and-location.kql`](../queries/error-message-and-location.kql) and reading the throw site + top 3 `error_message` strings. If the throw site is in `common/ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult}` AND the message starts with `AADSTS`, the originator is **eSTS, not broker** — see the AADSTS reference in [`assets/docs/kusto-cheatsheet.md`](../docs/kusto-cheatsheet.md). +- **WoW-movers pass is mandatory.** The 60d bucketer's `--peak-floor` silently drops sub-10K-device codes, so [`assets/queries/wow-movers.kql`](../queries/wow-movers.kql) MUST be run as a separate pass for both `error_code` and `error_type` (per Step 3d). Its output is **merged into the single regression callout** and then grouped by Step 3e's novelty labels. Do not render a separate "emerging" callout. Skipping the pass is how the Apr 26 `Failed to parse JWT` spike (7 → 3,461 devs over 7 weeks) hid for two reports running. +- **Novelty classification is mandatory, and Section 2 is ordered by it — never by volume.** Run [`classify-novelty.js`](../scripts/classify-novelty.js) (Step 3e) and lead with `NEW`. Ranking the attention list by device count is a known, reported defect: it put `IntuneAppProtectionPolicyRequiredException` (ΔWoW **+0.1%**, classifier says `ONGOING` and *falling*) at #1 while the genuinely new `ipc_*` family sat at #6/#9/#10. If the `NEW` bucket is empty, write "nothing new this week" — do not backfill it with `ONGOING` items. +- **Section 2's visible rows are the classifier's `attention` set plus at most 2 wins — nothing else.** + `NEW` + `ACCELERATING` visible with sparklines; `ONGOING` inside a collapsed `<details class="fold">`. + Budget: **≤ 8 visible rows total, wins included** (validator check 17 warns above it and counts wins). + The failure mode this replaces is measured, not hypothetical: 13 visible rows, 0 charts, and the + 60-day section below carrying 38. +- **Every visible attention row carries a 9-week `.item-spark`.** Validator check 16 hard-fails + otherwise. The series comes from the trend sidecar you already loaded — no extra query. Charts belong + beside the claim they support; a separate browsable chart section is the noise, not the signal. +- **The 60-day section is a detector, not a catalog.** Chart only the slow burns it *promotes* (rising + on 60d and absent from Section 2) — typically 0–3, often zero. The full classification goes in a fold + with no charts. Validator check 18 hard-fails above 6 visible charts there. +- **A quiet week is a valid outcome — publish it as one.** If `quietWeek: true`, say so plainly and + keep the report short. Padding the attention list with the biggest flat code to look thorough is the + exact behaviour that trains readers to skim. +- **A `VOLATILE`/`RECOVERY` row must not carry a `Δ WoW` chip at all — swap it for `vs 60d median`.** These carry `suppressRatio: true` because their WoW % is an artifact of a depressed prior week, not a regression. Tagging the row `VOLATILE` and caveating in the body is **not sufficient**: a naive run shipped `429` tagged `VOLATILE` with the body reading *"large percentage move but classified volatile"* — and still rendered `+401.8%` in `metric up` styling, which is the first thing a reader sees. `429` at "+401.8%" while sitting 94.5% *below* its own 60-day median is the canonical failure. `validate-report.ps1` check 15 hard-fails this. +- **A family is one finding, not N — and that includes the type axis.** When related codes move together (classifier `families`), emit **one** row whose `item-name` is the family and whose body names the members. Reconcile across axes too: if a `NEW` type is the umbrella for a `NEW` code family, that is still **one** row. A naive run emitted four rows — `BrokerCommunicationException`, `ipc_return_null_cursor`, `ipc_operation_not_supported_on_server_side`, `ipc_connection_error` — for a single IPC incident, which re-creates the wall-of-codes problem this section exists to fix. Correct shape: + ```html + <span class="item-name">ipc_* / BrokerCommunicationException<span class="kind">family</span></span> + ``` + with the body reading *"Three IPC codes stepped up together off a flat 7-week baseline (…null_cursor 41.5K→52.1K, …not_supported 20.6K→24.1K, …connection_error 8.6K→10.1K; family +22.2%). `BrokerCommunicationException` is the same incident seen on the type axis."* One row, one owner, one attribution card. +- **No boilerplate in Section 2.** Every row's one-line body must be specific to that row — what changed, from what to what, why it is or isn't alarming. Reusing one generic sentence across rows (*"Current-window movement needs owner triage; deep dive below has originator and dimensions."*) makes the section unreadable and `validate-report.ps1` fails the report for it. +- **Section 2 callouts are at-a-glance, Section 4 is the deep dive.** WoW / Slow-burn / Wins items in Section 2 use the `.item` flat-row pattern (no nested cards, no per-item left bars — the parent `.callout` border is the only severity affordance). Each row is a single line of metric chips + a one-line body + an `Attribution card →` link to the corresponding `.attr-card` in Section 4. Do NOT duplicate the dim slicing, PR analysis, or detailed verdict between the two sections — Section 4 is where that lives. See [`assets/templates/template-readme.md`](../templates/template-readme.md) for the CSS class reference and the example `.item` markup. +- **Denominator caveat must cite evidence, not hand-wave.** If you flag a large all-spans device-count shift, run [`assets/queries/broker-version-share-wow.kql`](../queries/broker-version-share-wow.kql) (single WoW snapshot) or [`assets/queries/broker-version-share.kql`](../queries/broker-version-share.kql) (time-series) and name the version cohort the shift moved with. Do not write "recurring telemetry-shape artifact" without backing data; if you don't have it, drop the callout. +- **"Recovery" still merits a PR citation.** When an error pins to a single old broker version and recovers as that version retires, look for the **fix PR in the version that replaced it** before calling it a "natural rolloff." Often the fix is real and just under-credited. +- **Never report WoW-only verdicts** for errors that are flat-or-down WoW but rising on 60d — always cross-check both windows. +- **Never page** based on a regression that turns out to be a downstream of a denominator shift; always include the auth-only-denominator number alongside the all-spans number. +- **Always cite PRs** with full GitHub URLs (the repo URL patterns above), not bare commit SHAs. + +--- + +## Output checklist + +- [ ] New `oncall-wow-report-YYYY-MM-DD.html` (where `YYYY-MM-DD` is the resolved `curEnd` — the end-date of the rolling 7-day window) exists at `$env:USERPROFILE\android-oce-reports\` (NOT at repo root). If a file for this end-date already existed, the chat session explicitly stated what changed before regenerating. +- [ ] All sections present and populated (incl. 🚚 Traffic Attribution — even if “None this week”) +- [ ] **60-day trend bucketing run on the full cross-product** — `{error_code, error_type} × {devices, requests}` = 4 runs — union of regressions reported. Per-request retry storms (e.g. small device pool, exploding request count) are flagged on both axes. Source KQL spans the literal last 60 days ending today (no source-side partial-week filter); the partial current week is excluded from delta classification via `bucket-trends.js --end=<startofweek(today)> --include-partial-end` and charted as the final bar. +- [ ] **WoW-movers pass run** ([`wow-movers.kql`](../queries/wow-movers.kql)) for BOTH `error_code` and `error_type`. Its output rows are **merged into the single regression callout in Section 2**. Every row carries throw-site, dominant message, originator, and a next step. If the callout is empty (rare), render "None this week" rather than omit. +- [ ] **Novelty classification run** ([`classify-novelty.js`](../scripts/classify-novelty.js)) on every `bucket-trends.js` sidecar. Section 2 is grouped 🆕 New → 🟠 Getting worse → 🔵 Ongoing (collapsed fold) → 🔁 Volatile → ↩️ Recovery, **not** sorted by device count. No `VOLATILE`/`RECOVERY` row headlines a percentage. Families are reported as one row. Every row body is specific — no sentence repeats across rows. +- [ ] **Attention section is short and charted.** Visible rows == the classifier's `attention` set (`NEW` + `ACCELERATING`), ≤ 8 of them, each with an `.item-spark` 9-week sparkline. `ONGOING` rows live inside a collapsed fold. If `quietWeek: true`, the quiet-week banner is shown and nothing was promoted to fill the gap. +- [ ] **60-day section charts only promoted slow burns** (rising on 60d, not already in Section 2 — often zero). Full classification is inside a `<details>` fold with no chart column. ≤ 6 visible charts. +- [ ] **Both error-codes AND error-types WoW tables have `Δ requests %` and `Δ devices %` columns**, the 60d sparkline, and a status pill. Any row crossing threshold on either metric is in the regression list. +- [ ] Every WoW regression AND every 60d regression — **for both `error_code` and `error_type`** — has its own spike-attribution card with all 7 dimensions sliced. Cards are built from [`assets/templates/spike-card.html`](../templates/spike-card.html). +- [ ] **Every `error_type` regression card includes the 8th-dimension sub-code decomposition** showing the top 3–5 contributing `error_code`s with their Δ vs prior week, and links to those sub-codes' own attribution cards. +- [ ] **Originator pre-check has been run for every broker-tagged card** ([`error-message-and-location.kql`](../queries/error-message-and-location.kql)). Throw site and top 3 `error_message` strings are populated from real data, not from the code map. AADSTS-prefixed messages are tagged `eSTS`, not `Broker`. +- [ ] **Every regression card's Code Attribution block populates Originator + Top throw site + Wrapper + Caller hot-spots + Underlying cause + Top error_messages + Likely PRs (with confidence/why-it's-the-suspect) + Next step (with named owner)**. For type cards, the wrapper field focuses on the type's catch-and-rethrow site (e.g. `BaseException`, `ServiceException` constructor). Shallow PR-only attribution is not acceptable. +- [ ] Non-broker errors are explicitly tagged `environmental` / `non-broker` with confidence `none` — not invented broker PRs. +- [ ] Traffic analysis covers totals, per-app, per-span, requests-per-device ratio (per error AND overall), and a sampling-change check. +- [ ] **Every material traffic shift (>10% on any segment, up or down) has a reasoning paragraph** that names the dominant span/app/active-broker/broker-version, and either cites a causal PR (with confidence) — span removed/added, `goAsync()` refactor, sampling change, caller-side SDK release, ECS flight ramp — or explicitly says "no PR identified, suspect X" rather than leaving it unexplained. +- [ ] Denominator caveat (if used) is backed by [`broker-version-share-wow.kql`](../queries/broker-version-share-wow.kql) or [`broker-version-share.kql`](../queries/broker-version-share.kql) evidence naming the responsible version cohort. No hand-waving. +- [ ] Auth-only denominator used for all reliability %s, denominator caveat called out at top. +- [ ] No `\bdevs\b` or `\breqs\b` in user-facing text. (`Select-String -Pattern '\bdevs\b|\breqs\b' -CaseSensitive:$false` returns 0.) +- [ ] **Sparklines rendered.** Every `.kpi` tile in the Top-line health section has a `data-spark` array with 8–9 weekly values. Every **visible** Section 2 attention row has an `.item-spark` (validator check 16). Every row in the WoW tables (codes + types) has a `data-trend` mini-spark. Note the 60-day trend `data-trend` arrays now end on the current partial week (≈9 points incl. the in-progress bar), while the WoW-table sparklines keep 8 complete weeks. Past failure mode: the v7 body rebuild dropped all sparklines silently — see `template-readme.md` § "Sparklines are MANDATORY". +- [ ] **Code-attribution depth.** Every `.attr-card`'s Code attribution block uses the full 8-field `<div class="origin-row">` structure (Originator / Top throw site / Wrapper / Caller hot-spots / Underlying cause / Top error_messages / Likely PRs / Next step) per [`assets/docs/code-attribution-template.md`](../docs/code-attribution-template.md). A `pr-list`-only stub is **not acceptable** — the validator hard-fails this. Past failure mode (v7 third pass): all 10 cards shipped with PR-only stubs and lost the throw-site / wrapper / underlying-cause analysis. +- [ ] No stale text from previous weeks. (`Select-String -Pattern 'EXAMPLE CONTENT BELOW'` returns 0 — that's the unfinished-section sentinel. The template no longer ships `{{TOKEN}}` placeholders since v2; if the file still contains any `{{`, that's also a leftover.) +- [ ] `get_errors` clean on the HTML file. + diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md b/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md index a48c5276..f3cdbdd7 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md @@ -36,7 +36,7 @@ rationale. | [`broker-version-share-wow.kql`](broker-version-share-wow.kql) | Single WoW snapshot of version share — fastest evidence for cohort transitions | Denominator caveat callout | | [`60d-trend-codes.kql`](60d-trend-codes.kql) | Feeds `bucket-trends.js` for codes (Sun-Sat weekly buckets over the literal last 60 days ending today; final bar = partial current week) | 60-day trend analysis | | [`60d-trend-types.kql`](60d-trend-types.kql) | Feeds `bucket-trends.js` for types (Sun-Sat weekly buckets over the literal last 60 days ending today; final bar = partial current week) | 60-day trend analysis | -| [`wow-movers.kql`](wow-movers.kql) | **MANDATORY second pass** — catches small-base codes that spiked sharply in the current window (below the 60d bucketer's reporting threshold). Run for both `error_code` and `error_type`. **Merge its output rows into the single 🔴 WoW regressions callout** alongside the standard WoW table; tag rows that were absent or near-zero in the prior window with `NEW`. Do not render a separate "emerging" callout. | 🔴 WoW regressions callout (Section 2) | +| [`wow-movers.kql`](wow-movers.kql) | **MANDATORY second pass** — catches small-base codes that spiked sharply in the current window (below the 60d bucketer's reporting threshold). Run for both `error_code` and `error_type`. **Merge its output rows into the same regression callout** alongside the standard WoW table, then group them by `classify-novelty.js` labels (🆕 New / 📈 Ongoing / 🔁 Volatile / ↩️ Recovery) — **never by device count**. Do not render a separate "emerging" callout. | Section 2 regression callouts | | [`attr-union-by-dim.kql`](attr-union-by-dim.kql) | **PREFERRED for WoW.** All 7 dims for N codes (or types) in ONE round-trip; pipe through `summarize-attribution.js --union`. | Spike attribution cards | | [`attr-codes-by-dim.kql`](attr-codes-by-dim.kql) | Per-dim form (run 7 times). Fall back to this only when the union exceeds payload size. | Spike attribution cards | | [`attr-types-by-dim.kql`](attr-types-by-dim.kql) | Per-dim form for type regressions | Spike attribution cards | diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/README.md b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/README.md new file mode 100644 index 00000000..842a38fe --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/README.md @@ -0,0 +1,63 @@ +# Authenticator query pack + +Canonical KQL for the **Authenticator app** half of the weekly OCE report. +Read [`../../docs/authapp-kusto-cheatsheet.md`](../../docs/authapp-kusto-cheatsheet.md) before +writing or editing anything here. + +**Cluster** `https://idsharedeus2.eastus2.kusto.windows.net` +**Database** `d496be22d62a46b0a3cf67ea2e736fd8` + +> ⚠️ These are **not** the Broker cluster/database, and Authenticator conventions are **not** +> Broker conventions. Do not carry `dcount_hll(hll_merge(...))`, `percentile_tdigest(...)`, +> `MergeAccountType`, or `EventInfo_Time` into these queries — none of them apply here. + +## Token convention + +All files use angle-bracket tokens, replaced before execution. +`bootstrap-report.ps1 -App authapp` prints every resolved value. + +| Token | Meaning | +|---|---| +| `<CUR_START>` | `curEnd − 7d` — start of the reporting window | +| `<CUR_END>` | `curEnd` — **exclusive** upper bound | +| `<PREV_START>` | `curEnd − 14d` — start of the baseline window (its end is always `<CUR_START>`) | +| `<TREND_START>` / `<TREND_END>` | literal last 60 days ending today — partial final week **included** | +| `<SPARK_START>` / `<SPARK_END>` | last 8 **complete** Sun-Sat weeks — `<SPARK_END>` = `startofweek(curEnd)`, partial week **excluded at the source** | +| `<ERRORS_MV>` | a `*_Errors_MV_V1` view name | +| `<REASON_FILTER>` | optional `\| where Error in (...)` line, or blank | + +## Files + +| File | Returns | Used by report section | +|---|---|---| +| [`scenario-outcomes-wow.kql`](scenario-outcomes-wow.kql) | one row per (Scenario, Window) — Initiated / Succeeded / Failed / **Unknown**, rates, and device twins, for all 9 single-MV scenarios in one round-trip | 1 Health · 2 Scoreboard · 3 Attention · 6 Unknown | +| [`pn-completion-wow.kql`](pn-completion-wow.kql) | one row per (PN family, Window) — initiated, reacted, completion rate, Approved/Denied/Error split, for all 4 PN families | 2 Scoreboard · 7 PN split | +| [`scenario-60d-trend.kql`](scenario-60d-trend.kql) | `week, scenario, errs, devs` (+ extras) — **directly consumable by `bucket-trends.js --key=scenario`** | 4 60-day trend | +| [`scenario-sparkline-series.kql`](scenario-sparkline-series.kql) | 8 complete weeks per scenario — success rate + bad-outcome volume | sparklines in 2 / 4 / 6 | +| [`scenario-errors-wow.kql`](scenario-errors-wow.kql) | error reasons WoW for **one** scenario's `*_Errors_MV_V1` | 5 Attribution | +| [`scenario-errors-by-dim.kql`](scenario-errors-by-dim.kql) | the same reasons sliced across all 3 dims (AppVersion / OsLevel / DeviceInfoMake) in one round-trip | 5 Attribution | +| [`broker-api-responsiveness-wow.kql`](broker-api-responsiveness-wow.kql) | per-`BrokerApiName` volume, success rate, p50/p95/p99 — raw `brokeroperations` table | 8 Broker API | +| [`version-share-wow.kql`](version-share-wow.kql) | AppVersion share WoW — the denominator check | 1 Health · 10 Adoption | + +## Query order for a run + +1. `version-share-wow.kql` **first**. If the version mix moved materially, every downstream rate + change has to be read against that. Running it last means re-reading every verdict. +2. `scenario-outcomes-wow.kql` + `pn-completion-wow.kql` — the scoreboard. +3. `scenario-60d-trend.kql` → `bucket-trends.js --key=scenario --end=<startofweek(curEnd)> --include-partial-end --peak-floor=1000`. +4. `scenario-sparkline-series.kql` — one pass, feeds every sparkline in the report. +5. For each scenario that regressed **and cleared the volume floor**: `scenario-errors-wow.kql`, + then `scenario-errors-by-dim.kql` filtered to the reasons that actually moved. +6. `broker-api-responsiveness-wow.kql` — slowest query, run it while step 5 is being written up. + +## The volume floor + +A scenario with fewer than **~1,000 initiates** in the window is noise. A 12-initiate scenario +going 100% → 50% is two users, not an incident. Tag such rows `low-volume` and keep them out of +the regression callout. This has no Broker equivalent — Broker error codes are high-volume by +construction; Authenticator scenario funnels are not. + +## Adding a query + +Keep the header comment block: purpose, cluster, database, tokens, and any trap specific to the +view. The header is the only documentation a future run will read before pasting the query. diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/broker-api-responsiveness-wow.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/broker-api-responsiveness-wow.kql new file mode 100644 index 00000000..335a9b30 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/broker-api-responsiveness-wow.kql @@ -0,0 +1,62 @@ +// Authenticator — BROKER API RESPONSIVENESS, week over week. +// +// This is the seam between the two reports. Authenticator calls into the Broker via a small set +// of named APIs; this query measures how those calls behave FROM THE AUTHENTICATOR SIDE. +// A regression here should always be cross-checked against the Broker report for the same window +// before it is attributed to Authenticator — and vice-versa. Cite the sibling report explicitly. +// +// ⚠️ RAW TABLE — not a materialized view. Two consequences: +// 1. The time column is PipelineInfo_IngestionTime (NOT EventDate, NOT EventInfo_Time). Using +// the wrong one silently returns an empty or wildly wrong window. +// 2. It is expensive. Keep the window at 14 days, keep the projection narrow, and expect this +// to be the slowest query in the run. If the Kusto MCP times out, use run-kql.ps1. +// Scale check (verified live): ~113M BrokerApi* events/day, so a 14-day window scans ~1.6B +// rows. A 2-day probe returns in seconds; if the full window struggles, narrow to 7 days and +// compare against the prior 7 rather than dropping the query. +// +// BrokerApiName and BrokerApiElapsedTimeMs live inside the AdditionalProperties string and must be +// extracted, not projected. +// +// ✅ VERIFIED LIVE — table name, all three OperationName literals, both extract() patterns, and +// DeviceInfo_Id all confirmed to exist and return data. Facts worth knowing: +// • The three operations are genuinely distinct: BrokerApiCallCompleted always carries +// BrokerApiSuccess=true, and failures arrive as their own BrokerApiCallFailed operation. +// Do NOT try to derive failures from a BrokerApiSuccess=false filter — that set is empty. +// • BrokerApiElapsedTimeMs is present ONLY on Completed/Failed, never on Initiated. The +// extract yields null there and percentile() ignores nulls, so the latency numbers are +// correctly computed over resolved calls only. This is intended — do not "fix" it. +// • Initiated exceeds Completed+Failed by ~2%. That gap is real abandonment, not a bug; it is +// why SuccessRate is computed against Initiated rather than against Completed+Failed. +// • Typical API mix: getBrokerAccounts.Entra (~80% of volume, P50 ~468ms, P95 ~3.6s), +// getAllEntries (~18%, P50 ~106ms), then a long tail. installCert is low-volume but slow +// (P95 tens of seconds) — do not flag it as a regression without a real WoW delta. +// +// Cluster: https://idsharedeus2.eastus2.kusto.windows.net +// Database: d496be22d62a46b0a3cf67ea2e736fd8 +// +// Tokens: <CUR_START> <CUR_END> <PREV_START> +let curStart = datetime(<CUR_START>); +let curEnd = datetime(<CUR_END>); +let prevStart = datetime(<PREV_START>); +brokeroperations +| where PipelineInfo_IngestionTime >= prevStart and PipelineInfo_IngestionTime < curEnd +| where OperationName in ("BrokerApiCallInitiated", "BrokerApiCallCompleted", "BrokerApiCallFailed") +| extend + ApiName = extract("BrokerApiName=([^,}]+)", 1, tostring(AdditionalProperties)), + ElapsedMs = toint(extract("BrokerApiElapsedTimeMs=([0-9]+)", 1, tostring(AdditionalProperties))), + Window = iff(PipelineInfo_IngestionTime >= curStart, "current", "previous") +| where isnotempty(ApiName) +| summarize + Initiated = countif(OperationName == "BrokerApiCallInitiated"), + Completed = countif(OperationName == "BrokerApiCallCompleted"), + Failed = countif(OperationName == "BrokerApiCallFailed"), + Devices = dcount(DeviceInfo_Id), + P50Ms = percentile(ElapsedMs, 50), + P95Ms = percentile(ElapsedMs, 95), + P99Ms = percentile(ElapsedMs, 99) + by ApiName, Window +| extend + SuccessRate = round(case(Initiated > 0, todouble(Completed) / todouble(Initiated) * 100, 0.0), 2), + FailureRate = round(case(Initiated > 0, todouble(Failed) / todouble(Initiated) * 100, 0.0), 2) +| project ApiName, Window, Initiated, Completed, Failed, SuccessRate, FailureRate, Devices, P50Ms, P95Ms, P99Ms +| order by ApiName asc, Window desc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/pn-completion-wow.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/pn-completion-wow.kql new file mode 100644 index 00000000..9445aa91 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/pn-completion-wow.kql @@ -0,0 +1,91 @@ +// Authenticator — WEEK-OVER-WEEK push-notification completion + reacted split, all 4 PN families. +// +// PN families have no Succeeded/Failed columns. The funnel is two-stage: +// stage 1 <PN_INIT_MV> notifications initiated / received (the denominator) +// stage 2 <PN_RESULTS_MV> the subset the user REACTED to, split by FinalResult +// +// CompletionRate = (Approved + Denied) / NotificationInitiated +// ApprovedRate / DeniedRate / ErrorRate are shares of the REACTED total, not of initiated. +// +// ✅ FinalResult — full distinct set enumerated live, do not assume it is only three values: +// Approved (~97% of reacted) Denied Error Cancelled "" (empty) +// The filter below deliberately keeps only {Approved, Denied, Error}. Cancelled and empty are +// ~0.1% of MFA volume and are NOT reactions that complete an auth, so excluding them is +// correct — and they are excluded from the CompletionRate numerator either way. If that share +// ever grows materially, surface it rather than silently dropping it. +// +// A drop in CompletionRate with a flat ApprovedRate means notifications are not reaching or not +// being acted on (delivery / lifecycle), NOT that auth is failing. Say which one in the report. +// +// Cluster: https://idsharedeus2.eastus2.kusto.windows.net +// Database: d496be22d62a46b0a3cf67ea2e736fd8 +// +// Tokens: <CUR_START> <CUR_END> <PREV_START> (baseline window end is always <CUR_START>) +// +// MSA note: BOTH the init MV and the results MV carry IsNGC ("true" → NGC, "false" → SA). +// The filter must be applied on BOTH sides or the two halves of the funnel disagree. +// +// KQL NOTE: in a `let` lambda, TABULAR parameters must be declared BEFORE scalar ones, so both +// helpers take `(T, label)` — not `(label, T)`. See scenario-outcomes-wow.kql's gotcha header. +// Verified live: the `join ... on Scenario, Window` below needs NO $left./$right. qualification. +let curStart = datetime(<CUR_START>); +let curEnd = datetime(<CUR_END>); +let prevStart = datetime(<PREV_START>); +let win = (d:datetime) { iff(d >= curStart, "current", "previous") }; +let inits = (T:(EventDate:datetime, Ini:long, IniD:long), label:string) { + T + | where EventDate >= prevStart and EventDate < curEnd + | summarize Initiated = sum(Ini), InitiatedDevices = sum(IniD) by Window = win(EventDate) + | extend Scenario = label +}; +let reacted = (T:(EventDate:datetime, FinalResult:string, React:long, ReactD:long), label:string) { + T + | where EventDate >= prevStart and EventDate < curEnd + | where FinalResult in ("Approved", "Denied", "Error") + | summarize R = sum(React), RD = sum(ReactD) by Window = win(EventDate), FinalResult + | summarize + Approved = sumif(R, FinalResult == "Approved"), + Denied = sumif(R, FinalResult == "Denied"), + Error = sumif(R, FinalResult == "Error"), + ApprovedDevices = sumif(RD, FinalResult == "Approved"), + DeniedDevices = sumif(RD, FinalResult == "Denied"), + ErrorDevices = sumif(RD, FinalResult == "Error") + by Window + | extend Scenario = label +}; +let allInits = + union + inits((Entra_MFA_Push_Notification_And_CheckForAuth_MV_V1 + | project EventDate, Ini = NotificationInitiated, IniD = NotificationInitiatedDCount), "Entra MFA PN+CFA"), + inits((Entra_PSI_Push_Notification_And_CheckForAuth_MV_V1 + | project EventDate, Ini = NotificationInitiated, IniD = NotificationInitiatedDCount), "Entra PSI PN+CFA"), + inits((Entra_MSA_Push_Notification_And_CheckForAuth_MV_V1 + | where IsNGC == "true" + | project EventDate, Ini = NotificationReceivedInitiated, IniD = NotificationReceivedInitiatedDCount), "MSA NGC PN+CFA"), + inits((Entra_MSA_Push_Notification_And_CheckForAuth_MV_V1 + | where IsNGC == "false" + | project EventDate, Ini = NotificationReceivedInitiated, IniD = NotificationReceivedInitiatedDCount), "MSA SA PN+CFA"); +let allReacted = + union + reacted((Entra_MFA_Push_Notification_And_CheckForAuth_Results_MV_V1 + | project EventDate, FinalResult, React = RequestTimeInitiated, ReactD = RequestTimeInitiatedDCount), "Entra MFA PN+CFA"), + reacted((Entra_PSI_Push_Notification_And_CheckForAuth_Results_MV_V1 + | project EventDate, FinalResult, React = RequestTimeInitiated, ReactD = RequestTimeInitiatedDCount), "Entra PSI PN+CFA"), + reacted((Entra_MSA_Push_Notification_And_CheckForAuth_Results_MV_V1 + | where IsNGC == "true" + | project EventDate, FinalResult, React = SessionTimeInitiated, ReactD = SessionTimeInitiatedDCount), "MSA NGC PN+CFA"), + reacted((Entra_MSA_Push_Notification_And_CheckForAuth_Results_MV_V1 + | where IsNGC == "false" + | project EventDate, FinalResult, React = SessionTimeInitiated, ReactD = SessionTimeInitiatedDCount), "MSA SA PN+CFA"); +allInits +| join kind=leftouter (allReacted) on Scenario, Window +| extend Reacted = Approved + Denied + Error +| extend + CompletionRate = round(case(Initiated > 0, todouble(Approved + Denied) / todouble(Initiated) * 100, 0.0), 2), + ApprovedRate = round(case(Reacted > 0, todouble(Approved) / todouble(Reacted) * 100, 0.0), 2), + DeniedRate = round(case(Reacted > 0, todouble(Denied) / todouble(Reacted) * 100, 0.0), 2), + ErrorRate = round(case(Reacted > 0, todouble(Error) / todouble(Reacted) * 100, 0.0), 2) +| project Scenario, Window, Initiated, Reacted, Approved, Denied, Error, + CompletionRate, ApprovedRate, DeniedRate, ErrorRate, + InitiatedDevices, ApprovedDevices, DeniedDevices, ErrorDevices +| order by Scenario asc, Window desc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-60d-trend.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-60d-trend.kql new file mode 100644 index 00000000..5ccab3b3 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-60d-trend.kql @@ -0,0 +1,54 @@ +// Authenticator — 60-DAY WEEKLY TREND per scenario, shaped for bucket-trends.js. +// +// Emits the four columns bucket-trends.js requires — week, scenario, errs, devs — where the +// "error" metric is the BAD-OUTCOME count (Failed + Unknown). That mapping is deliberate: it +// makes bucket-trends.js's rising == regression semantics correct for Authenticator without any +// script change. Extra columns (initiated, initiatedDevices, successRate) ride along for the +// chart and are ignored by the bucketer. +// +// node bucket-trends.js <file.json> --key=scenario --metric=devs \ +// --end=<startofweek(curEnd)> --include-partial-end --peak-floor=1000 +// +// Cluster: https://idsharedeus2.eastus2.kusto.windows.net +// Database: d496be22d62a46b0a3cf67ea2e736fd8 +// +// Tokens: <TREND_START> = curEnd - 60d <TREND_END> = curEnd (exclusive). +// Do NOT filter the partial in-progress week at the source — the chart wants it as its final bar. +// Exclude it from delta classification with bucket-trends.js --end=<startofweek(curEnd)>. +// See scenario-outcomes-wow.kql's "KQL GOTCHAS" header for why the lambda takes the tabular +// parameter FIRST, why `P()` spells out every column instead of using `(T:(*))`, and why the +// union argument carries its own parentheses. All four rules were learned against live Kusto. +let tStart = datetime(<TREND_START>); +let tEnd = datetime(<TREND_END>); +let outcome = (T:(EventDate:datetime, Ini:long, Suc:long, Fai:long, IniD:long, SucD:long, FaiD:long), label:string) { + T + | where EventDate >= tStart and EventDate < tEnd + | summarize Initiated = sum(Ini), Succeeded = sum(Suc), Failed = sum(Fai), + InitiatedD = sum(IniD), SucceededD = sum(SucD), FailedD = sum(FaiD) + by week = startofweek(EventDate) + | extend scenario = label +}; +let P = (T:(EventDate:datetime, Initiated:long, Succeeded:long, Failed:long, InitiatedDCount:long, SucceededDCount:long, FailedDCount:long)) { + T | project EventDate, Ini = Initiated, Suc = Succeeded, Fai = Failed, + IniD = InitiatedDCount, SucD = SucceededDCount, FaiD = FailedDCount +}; +union + outcome(P(Passkey_WebAuthN_Registration_MV_V1), "Passkey WebAuthN Registration"), + outcome(P(Passkey_InApp_Registration_MV_V1), "Passkey InApp Registration"), + outcome(P(Passkey_WebAuthN_Authentication_MV_V1), "Passkey WebAuthN Authentication"), + outcome(P(Entra_MFA_Registration_QR_Code_Flow_MV_V1), "Entra MFA Registration (QR)"), + outcome(P((union Entra_MFA_Registration_Manual_Flow_MV_V1, Entra_MFA_Registration_Non_QR_Code_Flow_MV_V1)), + "Entra MFA Registration (No-QR)"), + outcome(P(Entra_PSI_Registration_MV_V1), "Entra PSI Registration"), + outcome(P(Entra_MSA_NGC_Registration_MV_V1), "MSA NGC Registration"), + outcome(P(Entra_MSA_SA_Registration_MV_V1), "MSA SA Registration"), + outcome((Entra_PSI_Push_Notification_Registration_MV_V1 + | project EventDate, Ini = RegistrationStarted, Suc = RegistrationSucceeded, Fai = RegistrationFailed, + IniD = RegistrationStartedDCount, SucD = RegistrationSucceededDCount, FaiD = RegistrationFailedDCount), + "Entra PSI PN Registration") +| extend Unknown = case(Initiated > (Succeeded + Failed), Initiated - (Succeeded + Failed), 0) +| extend UnknownD = case(InitiatedD > (SucceededD + FailedD), InitiatedD - (SucceededD + FailedD), 0) +| extend errs = Failed + Unknown, devs = FailedD + UnknownD +| extend successRate = round(case(Initiated > 0, todouble(Succeeded) / todouble(Initiated) * 100, 0.0), 2) +| project week, scenario, errs, devs, initiated = Initiated, initiatedDevices = InitiatedD, successRate +| order by scenario asc, week asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-errors-by-dim.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-errors-by-dim.kql new file mode 100644 index 00000000..05d67898 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-errors-by-dim.kql @@ -0,0 +1,53 @@ +// Authenticator — error reasons sliced across ALL THREE dimensions in ONE round-trip. +// +// The Errors MVs carry exactly three usable dimensions: AppVersion, OsLevel, DeviceInfoMake. +// (There is no calling-app / account-type / shared-device dimension on the Authenticator side — +// do not go looking for the Broker's 7-dim slice here; it does not exist.) +// +// Use this once per REGRESSED scenario, after scenario-errors-wow.kql has identified which +// reasons moved. Read the output as concentration: if one AppVersion or one OEM holds a +// disproportionate share of the delta, that is the lead. If the delta is spread evenly across +// all three dims, it is a service-side or population change, not a client regression. +// +// Cluster: https://idsharedeus2.eastus2.kusto.windows.net +// Database: d496be22d62a46b0a3cf67ea2e736fd8 +// +// ⚠️ The Errors MVs come in three column shapes — see the mapping table at the top of +// scenario-errors-wow.kql and fill the same three column tokens here. +// ⚠️ KQL: tabular lambda params must be declared BEFORE scalar ones, hence `slice(T, dimName)`. +// See the gotcha header in scenario-outcomes-wow.kql. +// +// Tokens: <ERRORS_MV> <REASON_COL> <ERROR_COUNT_COL> <ERROR_DCOUNT_COL> <CUR_START> <CUR_END> <PREV_START> +// <REASON_FILTER> e.g. | where Reason in ("SomeReason", "OtherReason") +// must come AFTER normalisation, so it filters on `Reason` +// leave blank to slice every reason (heavier payload) +let curStart = datetime(<CUR_START>); +let curEnd = datetime(<CUR_END>); +let prevStart = datetime(<PREV_START>); +let src = + <ERRORS_MV> + | where EventDate >= prevStart and EventDate < curEnd + | project AppVersion, OsLevel, DeviceInfoMake, + Reason = tostring(<REASON_COL>), + ErrCnt = tolong(<ERROR_COUNT_COL>), + ErrDev = tolong(<ERROR_DCOUNT_COL>), + Window = iff(EventDate >= curStart, "current", "previous") + <REASON_FILTER>; +let slice = (T:(Reason:string, Window:string, DimValue:string, ErrCnt:long, ErrDev:long), dimName:string) { + T + | summarize Errors = sum(ErrCnt), Devices = sum(ErrDev) by Reason, DimValue, Window + | summarize CurErrors = sumif(Errors, Window == "current"), + PrevErrors = sumif(Errors, Window == "previous"), + CurDevices = sumif(Devices, Window == "current"), + PrevDevices= sumif(Devices, Window == "previous") + by Reason, DimValue + | extend Dim = dimName +}; +union + slice((src | project Reason, Window, DimValue = tostring(AppVersion), ErrCnt, ErrDev), "AppVersion"), + slice((src | project Reason, Window, DimValue = tostring(OsLevel), ErrCnt, ErrDev), "OsLevel"), + slice((src | project Reason, Window, DimValue = tostring(DeviceInfoMake), ErrCnt, ErrDev), "DeviceInfoMake") +| extend DeltaDevices = CurDevices - PrevDevices +| project Reason, Dim, DimValue, CurErrors, PrevErrors, CurDevices, PrevDevices, DeltaDevices +| where CurDevices > 0 or PrevDevices > 0 +| order by Reason asc, Dim asc, CurDevices desc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-errors-wow.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-errors-wow.kql new file mode 100644 index 00000000..b2db578e --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-errors-wow.kql @@ -0,0 +1,69 @@ +// Authenticator — WEEK-OVER-WEEK error-reason breakdown for ONE scenario. +// +// The outcome MVs say a scenario got worse; this says WHY. Every scenario has an Errors companion +// MV, named by inserting "Errors" before "_MV_V1": +// Passkey_WebAuthN_Registration_MV_V1 -> Passkey_WebAuthN_Registration_Errors_MV_V1 +// Entra_MFA_Push_Notification_And_CheckForAuth_MV_V1 +// -> Entra_MFA_Push_Notification_And_CheckForAuth_Errors_MV_V1 +// +// ⚠️⚠️ THE ERRORS MVs ARE **NOT** UNIFORM. Verified live 3 distinct column shapes exist. +// You MUST normalise per-MV with an explicit `project` before doing anything else. +// +// Shape A — 10 of 12 MVs (the default; assume this unless listed below): +// Reason = ErrorBeautified Errors = ErrorCount Devices = ErrorDCount +// Entra_PSI_Registration_Errors_MV_V1, Entra_MFA_Registration_QR_Code_Flow_Errors_MV_V1, +// Entra_MFA_Registration_Errors_Manual_MV_V1, Entra_MFA_Registration_Errors_Non_QR_Code_Flow_MV_V1, +// Entra_MSA_NGC_Registration_Errors_MV_V1, Entra_MSA_SA_Registration_Errors_MV_V1, +// Passkey_InApp_Registration_Failures_MV_V1, Entra_PSI_Push_Notification_Registration_Errors_MV_V1, +// Entra_MFA_Push_Notification_And_CheckForAuth_Errors_MV_V1, +// Entra_PSI_Push_Notification_And_CheckForAuth_Errors_MV_V1 +// +// Shape B — Passkey WebAuthN only (column is `Error`, NOT `ErrorBeautified`): +// Reason = Error Errors = ErrorCount Devices = ErrorDCount +// Passkey_WebAuthN_Authentication_Errors_MV_V1, Passkey_WebAuthN_Registration_Errors_MV_V1 +// +// Shape C — MSA push notification only (completely different family): +// Reason = ErrorDetails Errors = SessionTimeInitiated Devices = SessionTimeInitiatedDCount +// Entra_MSA_Push_Notification_And_CheckForAuth_Errors_MV_V1 +// +// ❌ Do NOT try to paper over this with `column_ifexists` after a `union` — union merges the +// schemas first, so the "missing" column then EXISTS (full of nulls) and the fallback never +// fires. Verified live: that silently returns Reason="" and Errors=0. Normalise per leg. +// ❌ Do NOT wrap the normalisation in a `let f = (T:(*)) {...}` lambda either — a wildcard tabular +// parameter cannot be followed by `project` on named columns (SEM0100). +// See the gotcha header in scenario-outcomes-wow.kql. +// +// ⚠️ COUNTS ONLY. This MV has no denominator. Always pair the numbers with the scenario's +// Initiated from scenario-outcomes-wow.kql before calling anything a rate. A raw error-count rise +// that tracks an equal initiated rise is traffic growth, not a regression. +// +// Cluster: https://idsharedeus2.eastus2.kusto.windows.net +// Database: d496be22d62a46b0a3cf67ea2e736fd8 +// +// Tokens: <ERRORS_MV> <REASON_COL> <ERROR_COUNT_COL> <ERROR_DCOUNT_COL> <CUR_START> <CUR_END> <PREV_START> +// Shape A -> ErrorBeautified / ErrorCount / ErrorDCount +// Shape B -> Error / ErrorCount / ErrorDCount +// Shape C -> ErrorDetails / SessionTimeInitiated / SessionTimeInitiatedDCount +let curStart = datetime(<CUR_START>); +let curEnd = datetime(<CUR_END>); +let prevStart = datetime(<PREV_START>); +<ERRORS_MV> +| where EventDate >= prevStart and EventDate < curEnd +| project EventDate, + Reason = tostring(<REASON_COL>), + ErrCnt = tolong(<ERROR_COUNT_COL>), + ErrDev = tolong(<ERROR_DCOUNT_COL>) +| summarize Errors = sum(ErrCnt), Devices = sum(ErrDev) + by Reason, Window = iff(EventDate >= curStart, "current", "previous") +| summarize + CurErrors = sumif(Errors, Window == "current"), + PrevErrors = sumif(Errors, Window == "previous"), + CurDevices = sumif(Devices, Window == "current"), + PrevDevices= sumif(Devices, Window == "previous") + by Reason +| extend + DeltaErrorsPct = round(case(PrevErrors > 0, (todouble(CurErrors) - PrevErrors) / PrevErrors * 100, 0.0), 1), + DeltaDevicesPct = round(case(PrevDevices > 0, (todouble(CurDevices) - PrevDevices) / PrevDevices * 100, 0.0), 1), + IsNew = PrevErrors == 0 and CurErrors > 0 +| project Reason, CurErrors, PrevErrors, DeltaErrorsPct, CurDevices, PrevDevices, DeltaDevicesPct, IsNew +| order by CurDevices desc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-outcomes-wow.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-outcomes-wow.kql new file mode 100644 index 00000000..92e488f8 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-outcomes-wow.kql @@ -0,0 +1,82 @@ +// Authenticator — WEEK-OVER-WEEK outcome distribution for every single-MV scenario, in ONE round-trip. +// +// This is the workhorse behind the report's scenario scoreboard (Section 1) and the regression +// list (Section 2). It returns one row per (Scenario, Window) with raw-event and distinct-device +// counts side by side, so the report can quote both a volume axis and a reach axis without a +// second query — the same dual-axis discipline the Broker report uses for devices vs requests. +// +// Cluster: https://idsharedeus2.eastus2.kusto.windows.net +// Database: d496be22d62a46b0a3cf67ea2e736fd8 +// +// Tokens (printed by bootstrap-report.ps1 -App authapp): +// <CUR_START> curEnd - 7d +// <CUR_END> curEnd (EXCLUSIVE upper bound) +// <PREV_START> curEnd - 14d (the baseline window's end is always <CUR_START>) +// +// NOTES +// * Outcome MVs expose ONLY Initiated / Succeeded / Failed (+ …DCount twins) and +// TotalUniqueDevices. There is NO Cancelled and NO PartiallySucceeded column. +// * Any shortfall (Initiated > Succeeded + Failed) is bucketed as "Unknown" — a session that +// started but never produced a terminal result in-window. Unknown is a first-class Authenticator +// signal (user abandonment / silent drop) with no Broker equivalent; report it, do not hide it. +// * Push-notification families are NOT in this query — they have no Succeeded/Failed columns. +// Use pn-completion-wow.kql for those. +// +// KQL GOTCHAS THIS QUERY WAS BITTEN BY (verified live — do not "simplify" these away): +// 1. In a `let` lambda, TABULAR parameters must be declared BEFORE scalar ones. `(label:string, +// T:(...))` fails with "Tabular parameters must be declared first" — hence `outcome(T, label)`. +// 2. A wildcard tabular parameter `(T:(*))` CANNOT be followed by `project` on named columns: +// the columns aren't statically known, so you get "Failed to resolve scalar expression named +// 'EventDate'". The `P()` shim therefore declares every column explicitly. +// 3. `union` as a function ARGUMENT needs its own parentheses — `P(union A, B)` parses the comma +// as an argument separator. Write `P((union A, B))`. +// 4. Entra_PSI_Push_Notification_Registration_MV_V1 does NOT use the Initiated/Succeeded/Failed +// naming. Its columns are RegistrationStarted / RegistrationSucceeded / RegistrationFailed +// (+ …DCount twins) — plus a RegistrationCancelled it does not expose to us here. Cancellations +// therefore fall into Unknown, which is semantically correct: a cancel IS an abandonment. +let curStart = datetime(<CUR_START>); +let curEnd = datetime(<CUR_END>); +let prevStart = datetime(<PREV_START>); +let outcome = (T:(EventDate:datetime, Ini:long, Suc:long, Fai:long, IniD:long, SucD:long, FaiD:long), label:string) { + T + | where EventDate >= prevStart and EventDate < curEnd + | summarize + Initiated = sum(Ini), + Succeeded = sum(Suc), + Failed = sum(Fai), + InitiatedDevices= sum(IniD), + SucceededDevices= sum(SucD), + FailedDevices = sum(FaiD) + by Window = iff(EventDate >= curStart, "current", "previous") + | extend Scenario = label +}; +// Rename shim for the MVs that DO use the standard Initiated/Succeeded/Failed naming. +let P = (T:(EventDate:datetime, Initiated:long, Succeeded:long, Failed:long, InitiatedDCount:long, SucceededDCount:long, FailedDCount:long)) { + T | project EventDate, Ini = Initiated, Suc = Succeeded, Fai = Failed, + IniD = InitiatedDCount, SucD = SucceededDCount, FaiD = FailedDCount +}; +union + outcome(P(Passkey_WebAuthN_Registration_MV_V1), "Passkey WebAuthN Registration"), + outcome(P(Passkey_InApp_Registration_MV_V1), "Passkey InApp Registration"), + outcome(P(Passkey_WebAuthN_Authentication_MV_V1), "Passkey WebAuthN Authentication"), + outcome(P(Entra_MFA_Registration_QR_Code_Flow_MV_V1), "Entra MFA Registration (QR)"), + outcome(P((union Entra_MFA_Registration_Manual_Flow_MV_V1, Entra_MFA_Registration_Non_QR_Code_Flow_MV_V1)), + "Entra MFA Registration (No-QR)"), + outcome(P(Entra_PSI_Registration_MV_V1), "Entra PSI Registration"), + outcome(P(Entra_MSA_NGC_Registration_MV_V1), "MSA NGC Registration"), + outcome(P(Entra_MSA_SA_Registration_MV_V1), "MSA SA Registration"), + // PSI push-notification registration uses the Registration* column family (see gotcha 4). + outcome((Entra_PSI_Push_Notification_Registration_MV_V1 + | project EventDate, Ini = RegistrationStarted, Suc = RegistrationSucceeded, Fai = RegistrationFailed, + IniD = RegistrationStartedDCount, SucD = RegistrationSucceededDCount, FaiD = RegistrationFailedDCount), + "Entra PSI PN Registration") +| extend Unknown = case(Initiated > (Succeeded + Failed), Initiated - (Succeeded + Failed), 0) +| extend UnknownDevices = case(InitiatedDevices > (SucceededDevices + FailedDevices), InitiatedDevices - (SucceededDevices + FailedDevices), 0) +| extend + SuccessRate = round(case(Initiated > 0, todouble(Succeeded) / todouble(Initiated) * 100, 0.0), 2), + FailureRate = round(case(Initiated > 0, todouble(Failed) / todouble(Initiated) * 100, 0.0), 2), + UnknownRate = round(case(Initiated > 0, todouble(Unknown) / todouble(Initiated) * 100, 0.0), 2) +| project Scenario, Window, Initiated, Succeeded, Failed, Unknown, + SuccessRate, FailureRate, UnknownRate, + InitiatedDevices, SucceededDevices, FailedDevices, UnknownDevices +| order by Scenario asc, Window desc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-sparkline-series.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-sparkline-series.kql new file mode 100644 index 00000000..f1b93f40 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-sparkline-series.kql @@ -0,0 +1,63 @@ +// Authenticator — 8-COMPLETE-WEEK series per scenario, for the per-row table sparklines. +// +// Deliberately distinct from scenario-60d-trend.kql: +// * 60d trend = literal last 60 days ending TODAY, partial final week INCLUDED (chart bar). +// * this query = last 8 COMPLETE Sun-Sat weeks, partial week FILTERED AT THE SOURCE. +// Row sparklines must not end on a partial week or every row shows a fake final-week cliff. +// +// <SPARK_END> = startofweek(curEnd), EXCLUSIVE. <SPARK_START> = <SPARK_END> - 56d. +// +// ⚠️ <SPARK_START> MUST land exactly on a Sunday (a startofweek boundary). Because <SPARK_END> is +// itself startofweek(curEnd), subtracting exactly 56d always lands on one — so use that formula +// and nothing else. If <SPARK_START> is off by even one day, this query does NOT error: the +// first bucket silently becomes partial and its rate is subtly wrong. Verified live — +// a one-day-late start moved week 1 of Entra MFA Registration (No-QR) from 45.56% to 45.87%, +// which is exactly the kind of phantom "improvement" that would get written up as a finding. +// +// ✅ Verified live: with correct boundaries this returns exactly 8 buckets per scenario and the +// last bucket is a complete week (no trailing cliff). +// +// Emits one row per (scenario, week) with both the success-rate series (what the scoreboard +// sparkline draws) and the bad-outcome volume series (what the regression rows draw). +// +// Cluster: https://idsharedeus2.eastus2.kusto.windows.net +// Database: d496be22d62a46b0a3cf67ea2e736fd8 +// See scenario-outcomes-wow.kql's "KQL GOTCHAS" header for why the lambda takes the tabular +// parameter FIRST, why `P()` spells out every column instead of using `(T:(*))`, and why the +// union argument carries its own parentheses. All four rules were learned against live Kusto. +let sStart = datetime(<SPARK_START>); +let sEnd = datetime(<SPARK_END>); +let outcome = (T:(EventDate:datetime, Ini:long, Suc:long, Fai:long, IniD:long, SucD:long, FaiD:long), label:string) { + T + | where EventDate >= sStart and EventDate < sEnd + | summarize Initiated = sum(Ini), Succeeded = sum(Suc), Failed = sum(Fai), + InitiatedD = sum(IniD), SucceededD = sum(SucD), FailedD = sum(FaiD) + by week = startofweek(EventDate) + | extend scenario = label +}; +let P = (T:(EventDate:datetime, Initiated:long, Succeeded:long, Failed:long, InitiatedDCount:long, SucceededDCount:long, FailedDCount:long)) { + T | project EventDate, Ini = Initiated, Suc = Succeeded, Fai = Failed, + IniD = InitiatedDCount, SucD = SucceededDCount, FaiD = FailedDCount +}; +union + outcome(P(Passkey_WebAuthN_Registration_MV_V1), "Passkey WebAuthN Registration"), + outcome(P(Passkey_InApp_Registration_MV_V1), "Passkey InApp Registration"), + outcome(P(Passkey_WebAuthN_Authentication_MV_V1), "Passkey WebAuthN Authentication"), + outcome(P(Entra_MFA_Registration_QR_Code_Flow_MV_V1), "Entra MFA Registration (QR)"), + outcome(P((union Entra_MFA_Registration_Manual_Flow_MV_V1, Entra_MFA_Registration_Non_QR_Code_Flow_MV_V1)), + "Entra MFA Registration (No-QR)"), + outcome(P(Entra_PSI_Registration_MV_V1), "Entra PSI Registration"), + outcome(P(Entra_MSA_NGC_Registration_MV_V1), "MSA NGC Registration"), + outcome(P(Entra_MSA_SA_Registration_MV_V1), "MSA SA Registration"), + outcome((Entra_PSI_Push_Notification_Registration_MV_V1 + | project EventDate, Ini = RegistrationStarted, Suc = RegistrationSucceeded, Fai = RegistrationFailed, + IniD = RegistrationStartedDCount, SucD = RegistrationSucceededDCount, FaiD = RegistrationFailedDCount), + "Entra PSI PN Registration") +| extend Unknown = case(Initiated > (Succeeded + Failed), Initiated - (Succeeded + Failed), 0) +| extend UnknownD = case(InitiatedD > (SucceededD + FailedD), InitiatedD - (SucceededD + FailedD), 0) +| extend + successRate = round(case(Initiated > 0, todouble(Succeeded) / todouble(Initiated) * 100, 0.0), 2), + errs = Failed + Unknown, + devs = FailedD + UnknownD +| project week, scenario, successRate, errs, devs, initiated = Initiated, initiatedDevices = InitiatedD +| order by scenario asc, week asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/version-share-wow.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/version-share-wow.kql new file mode 100644 index 00000000..fc05de1d --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/version-share-wow.kql @@ -0,0 +1,43 @@ +// Authenticator — APP VERSION SHARE, week over week (rollout / denominator check). +// +// Purpose: before you call any scenario rate change a regression, check whether the version mix +// moved. A new build ramping from 5% to 60% of the population can move every scenario's rate at +// once without a single line of scenario code having changed. This is the Authenticator analogue +// of the Broker report's broker-version-share check, and it carries the same hard rule: +// a denominator caveat must CITE this query's output and name the version cohort — never hand-wave. +// +// ⚠️ This is a telemetry-active-device proxy, NOT a product DAU. It counts devices that emitted +// at least one Entra MFA push-notification event in the window. Label it that way in the +// report; do not print it as "daily active users". +// +// The MFA PN+CFA view is used as the population source because it is the highest-volume +// Authenticator MV — resolving version share from it is far cheaper than a `union *` scan. +// +// Cluster: https://idsharedeus2.eastus2.kusto.windows.net +// Database: d496be22d62a46b0a3cf67ea2e736fd8 +// +// Tokens: <CUR_START> <CUR_END> <PREV_START> +let curStart = datetime(<CUR_START>); +let curEnd = datetime(<CUR_END>); +let prevStart = datetime(<PREV_START>); +let pop = + Entra_MFA_Push_Notification_And_CheckForAuth_MV_V1 + | where EventDate >= prevStart and EventDate < curEnd + | where isnotempty(AppVersion) + | summarize Devices = sum(NotificationInitiatedDCount) + by AppVersion, Window = iff(EventDate >= curStart, "current", "previous"); +let totals = pop | summarize WindowDevices = sum(Devices) by Window; +pop +| join kind=inner (totals) on Window +| extend SharePct = round(todouble(Devices) / todouble(WindowDevices) * 100, 2) +| summarize + CurDevices = sumif(Devices, Window == "current"), + PrevDevices = sumif(Devices, Window == "previous"), + CurSharePct = sumif(SharePct, Window == "current"), + PrevSharePct= sumif(SharePct, Window == "previous") + by AppVersion +| extend + DeltaSharePts = round(CurSharePct - PrevSharePct, 2), + DeltaDevicesPct = round(case(PrevDevices > 0, (todouble(CurDevices) - PrevDevices) / PrevDevices * 100, 0.0), 1) +| project AppVersion, CurDevices, CurSharePct, PrevDevices, PrevSharePct, DeltaSharePts, DeltaDevicesPct +| order by CurDevices desc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 index 4d3a8fcb..687b5480 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 @@ -1,10 +1,15 @@ <# .SYNOPSIS - Bootstrap a new OCE weekly report file from the canonical template. + Bootstrap a new OCE weekly report file from an app's canonical template. .DESCRIPTION - Implements SKILL.md Step 1 as a script so the workflow doesn't drift across - runs. The reporting window is a ROLLING 7-DAY window ending at start-of-day + Implements the SKILL.md router's bootstrap step as a script so the workflow + doesn't drift across runs. Serves BOTH apps -- pass -App broker (default) or + -App authapp; everything below is app-parameterised except the window math, + which is deliberately identical so a `both`-mode run can hand the same + -EndDate to two bootstraps and get two reports covering the same interval. + + The reporting window is a ROLLING 7-DAY window ending at start-of-day (UTC) on -EndDate (defaults to today): curStart = EndDate - 7d (inclusive) @@ -21,9 +26,10 @@ before the invocation with no user prompting. The script also: - 1. Creates ~/android-oce-reports/_data/<end-date>/ for raw query payloads. - 2. Copies the canonical template into - ~/android-oce-reports/oncall-wow-report-<end-date>.html. + 1. Creates ~/android-oce-reports/_data/<app>-<end-date>/ for raw payloads. + 2. Copies the app's canonical template into + ~/android-oce-reports/<prefix>-wow-report-<end-date>.html, where + <prefix> is 'oncall' for broker and 'authapp' for authapp. 3. Stamps the resolved window into the <title>, the <div class="meta"> block, and the "Generated <strong>...</strong>" banner so the header can never drift from what was actually queried (the resolved window @@ -33,9 +39,15 @@ fingerprint markers still match the canonical template), silently re-bootstrap -- nothing to preserve. - Otherwise HALT and require -Force. - 5. Prunes _data/<old-end-date>/ folders older than -DataRetentionDays + 5. Prunes _data/<app>-<old-end-date>/ folders older than -DataRetentionDays (default 60). +.PARAMETER App + Which report to bootstrap: 'broker' (default) or 'authapp'. Selects the + template, output filename, data folder, title, and telemetry-source label. + The two apps read DIFFERENT clusters and databases and their instructions do + not transfer -- see the playbook for the app you are bootstrapping. + .PARAMETER EndDate End of the reporting window (yyyy-MM-dd, UTC). Exclusive upper bound: data is queried up to but not including 00:00 UTC on this date. Defaults to @@ -46,14 +58,20 @@ Skip the collision check and overwrite any existing file. .PARAMETER DataRetentionDays - How many days of _data/<end-date>/ folders to keep before pruning. Default 60. + How many days of _data/<app>-<end-date>/ folders to keep before pruning. + Default 60. Pruning is scoped to the -App being bootstrapped, so a broker + run never deletes authapp payloads (or vice versa). .PARAMETER SkillRoot Path to the skill folder. Defaults to the location of this script's parent. .EXAMPLE .\bootstrap-report.ps1 - # Default: rolling 7 days ending today (UTC), halt on collision. + # Default: broker, rolling 7 days ending today (UTC), halt on collision. + +.EXAMPLE + .\bootstrap-report.ps1 -App authapp + # Authenticator report for the same rolling window. .EXAMPLE .\bootstrap-report.ps1 -EndDate 2026-07-09 -Force @@ -70,6 +88,8 @@ #> [CmdletBinding()] param( + [ValidateSet('broker','authapp')] + [string]$App = 'broker', [string]$EndDate, [switch]$Force, [int]$DataRetentionDays = 60, @@ -77,6 +97,30 @@ param( ) $ErrorActionPreference = 'Stop' +# --------------------------------------------------------------------------- +# Per-app profile +# +# Everything that differs between the two reports is declared here rather than +# branched inline, so adding a third app is a data change, not a code change. +# The window math deliberately stays outside this table: `both` mode depends on +# the two apps resolving identical windows from the same -EndDate. +# --------------------------------------------------------------------------- +$appProfiles = @{ + broker = @{ + Template = 'report-template.html' + FilePrefix = 'oncall' + TitleName = 'Android Broker' + SourceHtml = 'Source: <code>android_spans</code> materialized views' + } + authapp = @{ + Template = 'authapp-report-template.html' + FilePrefix = 'authapp' + TitleName = 'Authenticator Android' + SourceHtml = 'Source: Authenticator scenario materialized views' + } +} +$appProfile = $appProfiles[$App] + # --------------------------------------------------------------------------- # Locate the skill folder + canonical template # --------------------------------------------------------------------------- @@ -85,9 +129,9 @@ if (-not $SkillRoot) { # 2 levels to reach <skill>/assets/. Templates live at <skill>/assets/templates/. $SkillRoot = Split-Path -Parent (Split-Path -Parent $PSCommandPath) } -$template = Join-Path $SkillRoot 'templates\report-template.html' +$template = Join-Path $SkillRoot "templates\$($appProfile.Template)" if (-not (Test-Path $template)) { - throw "Canonical template not found at $template. Pass -SkillRoot if running outside the skill folder." + throw "Canonical template for -App $App not found at $template. Pass -SkillRoot if running outside the skill folder." } # --------------------------------------------------------------------------- @@ -129,6 +173,12 @@ $curEndDow = [int]$curEnd.DayOfWeek # Sun=0 .. Sat=6 $sixtyDayStart = $curEnd.AddDays(-60) # literal 60 days ending today $sixtyDayEnd = $curEnd # exclusive upper bound == today; chart includes the partial current week $trendClassEnd = $curEnd.AddDays(-$curEndDow) # startofweek(curEnd): weeks >= this are the in-progress (partial) week, excluded from delta classification +# Sparkline window: last 8 COMPLETE Sun-Sat weeks. sparkEnd is exclusive and is by construction a +# Sunday, so sparkEnd-56d is also exactly a Sunday. These are emitted below so the author never +# hand-computes them: an off-by-one sparkStart does NOT error in Kusto, it silently makes the first +# sparkline bucket partial and skews its rate (verified live against Entra MFA Registration No-QR). +$sparkEnd = $trendClassEnd +$sparkStart = $sparkEnd.AddDays(-56) # Sanity check: curEnd is an exclusive 00:00-UTC date boundary and must be today # (UTC) or earlier. Compare date-to-date -- a sub-day clock slack here would let @@ -144,11 +194,12 @@ $curStartStr = $curStart.ToString('yyyy-MM-dd') $prevStartStr = $prevStart.ToString('yyyy-MM-dd') $prevEndStr = $prevEnd.ToString('yyyy-MM-dd') -Write-Host "Resolved reporting window (UTC):" +Write-Host "Resolved reporting window (UTC) for -App $App :" Write-Host " Last 7 days: $curStartStr -> $curEndStr (exclusive upper bound)" Write-Host " Baseline: $prevStartStr -> $prevEndStr" Write-Host " 60-day trend: $($sixtyDayStart.ToString('yyyy-MM-dd')) -> $($sixtyDayEnd.ToString('yyyy-MM-dd')) (literal 60d ending today; chart includes current partial week)" Write-Host " Trend delta cutoff: weeks < $($trendClassEnd.ToString('yyyy-MM-dd')) (startofweek(curEnd); pass as bucket-trends.js --end)" +Write-Host " Sparkline (8 complete weeks): $($sparkStart.ToString('yyyy-MM-dd')) -> $($sparkEnd.ToString('yyyy-MM-dd')) (SPARK_START -> SPARK_END, exclusive; both land on Sunday -- do not adjust by hand)" # NOTE: Console output uses ASCII '->'; the HTML stamp below uses U+2192 arrows # and U+00B7 middle-dots to match the template's canonical visual style. This # is safe because $outText is written via [System.Text.UTF8Encoding]::new($false) @@ -159,8 +210,8 @@ Write-Host " Trend delta cutoff: weeks < $($trendClassEnd.ToString('yyyy-MM-dd' # Paths # --------------------------------------------------------------------------- $reportDir = Join-Path $env:USERPROFILE 'android-oce-reports' -$dataDir = Join-Path $reportDir "_data\$curEndStr" -$out = Join-Path $reportDir "oncall-wow-report-$curEndStr.html" +$dataDir = Join-Path $reportDir "_data\$App-$curEndStr" +$out = Join-Path $reportDir "$($appProfile.FilePrefix)-wow-report-$curEndStr.html" New-Item -ItemType Directory -Force $reportDir | Out-Null New-Item -ItemType Directory -Force $dataDir | Out-Null @@ -260,7 +311,7 @@ $todayStr = (Get-Date).ToUniversalTime().ToString('yyyy-MM-dd') $outText = [IO.File]::ReadAllText($out) # 1) <title>... -$newTitle = "Android Broker $dot On-Call Report $([char]0x2014) Last 7 days ending $curEndStr" +$newTitle = "$($appProfile.TitleName) $dot On-Call Report $([char]0x2014) Last 7 days ending $curEndStr" $outText = [regex]::Replace($outText, '[^<]*', "$newTitle") # 2) The
block up through the closing
immediately @@ -269,7 +320,7 @@ $newMeta = @"
Last 7 days: $curLabel  vs  $prevLabel  $dot  60-day trend: $sixtyDayLabel (last 60 days; final bar in progress)  $dot  - Source: android_spans materialized views  $dot  + $($appProfile.SourceHtml)  $dot  Generated $todayStr
"@ @@ -301,12 +352,15 @@ Write-Host "Stamped resolved window into and meta block. Generated=$toda $dataRoot = Join-Path $reportDir '_data' if (Test-Path $dataRoot) { $cutoff = (Get-Date).AddDays(-$DataRetentionDays) - $oldFolders = Get-ChildItem $dataRoot -Directory | Where-Object { + # Scope pruning to THIS app's folders. In `both` mode the two bootstraps run + # concurrently against the same _data root; an unscoped prune would let one + # app delete the other's freshly written payloads mid-run. + $oldFolders = Get-ChildItem $dataRoot -Directory -Filter "$App-*" | Where-Object { $_.FullName -ne $dataDir -and $_.LastWriteTime -lt $cutoff } if ($oldFolders) { - Write-Host "Pruning $($oldFolders.Count) _data folder(s) older than $DataRetentionDays days:" + Write-Host "Pruning $($oldFolders.Count) $App _data folder(s) older than $DataRetentionDays days:" $oldFolders | ForEach-Object { Write-Host " removing $($_.FullName) (last write $($_.LastWriteTime.ToString('yyyy-MM-dd')))" Remove-Item -Recurse -Force $_.FullName diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/build-index.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/build-index.ps1 new file mode 100644 index 00000000..7ef3bb39 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/build-index.ps1 @@ -0,0 +1,176 @@ +<# +.SYNOPSIS + Build the combined on-call index page from the two finished weekly reports. + +.DESCRIPTION + `both` mode produces two independent reports. This script emits a single + one-page digest that links to each and reproduces their headline KPI tiles, + so the on-call engineer has one URL to open and one place to scan. + + It is deliberately a SCRAPER, not an author: every value on the index is + lifted verbatim out of the report HTML. Nothing is recomputed, and no new + finding is introduced. That guarantee is what makes the index safe -- a + number can never disagree with the report it came from. + + Missing reports are not fatal. If only one app ran (or one failed), the + index renders that app's card and an explicit "not generated this run" + state for the other, so a half-delivered rotation is still publishable and + the gap is visible rather than silent. + +.PARAMETER EndDate + The reporting window end-date (yyyy-MM-dd) shared by both reports. Defaults + to today (UTC), matching bootstrap-report.ps1's default. + +.PARAMETER ReportDir + Folder holding the reports. Defaults to $env:USERPROFILE\android-oce-reports. + +.PARAMETER MaxKpis + How many headline KPI tiles to reproduce per app. Default 6. + +.PARAMETER SkillRoot + Path to the skill's assets folder. Defaults to this script's grandparent. + +.EXAMPLE + .\build-index.ps1 + .\build-index.ps1 -EndDate 2026-07-30 + +.OUTPUTS + Prints the absolute path of the index file. +#> +[CmdletBinding()] +param( + [string]$EndDate, + [string]$ReportDir, + [int]$MaxKpis = 6, + [string]$SkillRoot +) +$ErrorActionPreference = 'Stop' + +if (-not $EndDate) { $EndDate = [datetime]::UtcNow.Date.ToString('yyyy-MM-dd') } +if (-not $ReportDir) { $ReportDir = Join-Path $env:USERPROFILE 'android-oce-reports' } +if (-not $SkillRoot) { $SkillRoot = Split-Path -Parent (Split-Path -Parent $PSCommandPath) } + +$template = Join-Path $SkillRoot 'templates\index-template.html' +if (-not (Test-Path $template)) { + throw "Index template not found at $template. Pass -SkillRoot if running outside the skill folder." +} + +$apps = @( + @{ Key = 'broker'; Prefix = 'oncall'; Name = 'Android Broker' + Sub = 'Silent + interactive auth reliability, error-code attribution, latency, adoption' } + @{ Key = 'authapp'; Prefix = 'authapp'; Name = 'Authenticator Android' + Sub = 'Scenario funnels, unknown/abandonment, push-notification completion, crash & stability' } +) + +function Get-Kpis([string]$html, [int]$max) { + # Scrape .kpi tiles in document order. The report's first .kpi-grid is the + # headline grid by construction (both templates open with it), so taking the + # first $max tiles reproduces exactly what a reader sees above the fold. + # Matches both the multi-line and single-line .kpi forms the templates use. + $re = '(?s)<div class="kpi">\s*' + + '<div class="label">(.*?)</div>\s*' + + '<div class="value">(.*?)</div>' + + '(?:\s*<div class="delta ([a-z\-]*)">(.*?)</div>)?' + $out = @() + foreach ($m in [regex]::Matches($html, $re)) { + if ($out.Count -ge $max) { break } + $out += [pscustomobject]@{ + Label = $m.Groups[1].Value.Trim() + Value = $m.Groups[2].Value.Trim() + DeltaClass = if ($m.Groups[3].Success -and $m.Groups[3].Value) { $m.Groups[3].Value } else { 'delta-flat' } + Delta = $m.Groups[4].Value.Trim() + } + } + return $out +} + +function Get-MetaLine([string]$html) { + $m = [regex]::Match($html, '(?s)<div class="meta">(.*?)</div>') + if (-not $m.Success) { return $null } + # Collapse to a single line and strip tags -- the index shows it as plain text. + $t = [regex]::Replace($m.Groups[1].Value, '<[^>]+>', '') + $t = ($t -replace ' ', ' ') -replace '\s+', ' ' + return $t.Trim() +} + +$cards = New-Object System.Text.StringBuilder +$found = @() +$missing = @() +$windowLine = $null + +foreach ($app in $apps) { + $file = Join-Path $ReportDir "$($app.Prefix)-wow-report-$EndDate.html" + [void]$cards.AppendLine(' <div class="report-card">') + [void]$cards.AppendLine(' <div class="card-head">') + [void]$cards.AppendLine(" <h3>$($app.Name)</h3>") + + if (-not (Test-Path $file)) { + $missing += $app.Key + [void]$cards.AppendLine(' </div>') + [void]$cards.AppendLine(" <div class=`"card-sub`">$($app.Sub)</div>") + [void]$cards.AppendLine(" <p class=`"empty`">Not generated this run — no <code>$($app.Prefix)-wow-report-$EndDate.html</code> found in $ReportDir.</p>") + [void]$cards.AppendLine(' </div>') + Write-Warning "$($app.Name): report not found at $file - rendering an explicit empty state." + continue + } + + $found += $app.Key + $html = [IO.File]::ReadAllText($file) + + # Refuse to index an unpopulated stub -- publishing template numbers under a + # real-looking heading is worse than showing the gap. + if ($html.Contains('OCE-UNPOPULATED-STUB')) { + throw "$file still carries the OCE-UNPOPULATED-STUB sentinel (it was bootstrapped but never populated). Refusing to build an index over template data. Populate and validate the report first." + } + + if (-not $windowLine) { $windowLine = Get-MetaLine $html } + + $leaf = Split-Path $file -Leaf + [void]$cards.AppendLine(" <a class=`"open-link`" href=`"$leaf`">Open full report →</a>") + [void]$cards.AppendLine(' </div>') + [void]$cards.AppendLine(" <div class=`"card-sub`">$($app.Sub)</div>") + + $kpis = Get-Kpis $html $MaxKpis + if ($kpis.Count -eq 0) { + [void]$cards.AppendLine(' <p class="empty">No KPI tiles found to summarise — open the report directly.</p>') + Write-Warning "$($app.Name): no .kpi tiles matched in $leaf." + } else { + [void]$cards.AppendLine(' <div class="kpi-grid">') + foreach ($k in $kpis) { + [void]$cards.AppendLine(' <div class="kpi">') + [void]$cards.AppendLine(" <div class=`"label`">$($k.Label)</div>") + [void]$cards.AppendLine(" <div class=`"value`">$($k.Value)</div>") + if ($k.Delta) { [void]$cards.AppendLine(" <div class=`"delta $($k.DeltaClass)`">$($k.Delta)</div>") } + [void]$cards.AppendLine(' </div>') + } + [void]$cards.AppendLine(' </div>') + Write-Host "$($app.Name): reproduced $($kpis.Count) KPI tile(s) from $leaf" + } + [void]$cards.AppendLine(' </div>') +} + +if ($found.Count -eq 0) { + Write-Error "No reports found for end-date $EndDate in $ReportDir. Nothing to index. Run bootstrap-report.ps1 + the playbooks first." + exit 2 +} + +if (-not $windowLine) { $windowLine = "Reporting window ending $EndDate" } + +$out = Join-Path $ReportDir "oce-index-$EndDate.html" +$text = [IO.File]::ReadAllText($template) +$today = [datetime]::UtcNow.ToString('yyyy-MM-dd') + +$text = $text.Replace('<!--INDEX:TITLE-->', "Android Auth $([char]0x00B7) Weekly On-Call Digest $([char]0x00B7) $EndDate") +$text = $text.Replace('<!--INDEX:WINDOW-->', $windowLine) +$text = $text.Replace('<!--INDEX:GENERATED-->', $today) +$text = $text.Replace('<!--INDEX:CARDS-->', $cards.ToString().TrimEnd()) + +# UTF-8 without BOM -- same trap as the reports: a heredoc/Set-Content path would +# strip the multi-byte arrows and middle-dots stamped above. +[IO.File]::WriteAllText($out, $text, [System.Text.UTF8Encoding]::new($false)) + +Write-Host "Built index: $out" +if ($missing.Count -gt 0) { + Write-Warning "Index published with $($missing.Count) app(s) missing: $($missing -join ', '). Say so explicitly in chat." +} +Write-Output $out diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/classify-novelty.js b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/classify-novelty.js new file mode 100644 index 00000000..a0753b58 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/classify-novelty.js @@ -0,0 +1,332 @@ +#!/usr/bin/env node +/** + * classify-novelty.js -- Answer "is this NEW this week, or has it been broken for weeks?" + * + * WHY THIS EXISTS + * --------------- + * bucket-trends.js answers "which direction has this moved over 60 days?" That is a + * *direction* question. It cannot answer the question an on-call engineer actually asks + * when triaging: "which of these 20 moving error codes started THIS week, and which have + * I already been staring at for a month?" + * + * Without that split, the attention section degenerates into a volume-ranked list where a + * flat-but-huge error code outranks a genuine step change, and every card carries the same + * "needs owner triage" boilerplate. That is precisely the failure this script fixes. + * + * IT ALSO SUPPRESSES RATIO ARTIFACTS + * ---------------------------------- + * A week-over-week percentage is meaningless when the PRIOR week was itself anomalous. + * Real example from the 2026-07-30 broker run: + * + * 429 300,664 299,965 892,839 974,980 11,512 32,530 2,724 -> 16,531 + * temporarily_unavailable 30,257 29,962 37,263 6,168 41,971 141 71 -> 36,153 + * + * Both reported as ~+400% WoW and led the regression list. Both are noise: `429` is + * actually 94% BELOW its own 60-day median (it collapsed from ~975K in June), and + * `temporarily_unavailable` merely returned to its normal ~36K band after two suppressed + * weeks. Meanwhile the genuinely new regression that week -- the ipc_* family, flat for + * seven straight weeks (cv 0.02-0.03) then stepping up 22% together -- sat at positions + * #6, #9 and #10. + * + * A series with high variance gets `suppressRatio: true`. The playbook must NOT headline a + * percentage for those; report the absolute level and its position within the historical + * band instead. + * + * FAMILY CLUSTERING + * ----------------- + * Related codes that move together are one root cause, not N findings. When >=2 keys + * sharing a prefix all move the same direction in the same week, they are emitted as a + * `families` entry with a summed series. One story, one card. + * + * IT ALSO DECIDES WHAT DESERVES A CHART (noise control) + * ---------------------------------------------------- + * "Elevated" and "getting worse" are different questions, and conflating them is what made + * the report re-triage the same known issues every week. `ONGOING` is therefore split: + * + * ACCELERATING elevated AND still climbing right now -> promote, chart it, card it + * ONGOING elevated but PLATEAUED -> collapse to a counted line + * + * Only NEW + ACCELERATING form the `attention` set. Everything else is reference material. + * On the 2026-07-31 broker data this is 5 series out of 53, against 13 volume-ranked + * attention rows before -- and the 5 are the real ones. + * + * Two false positives the ACCELERATING gate exists to stop, both observed on real data: + * authorization_pending down 37% WoW, 16% BELOW its own median + * IntuneAppProtectionPolicyRequiredException flat (cv 0.08), +4.9% vs median, down 3.7% WoW + * Both had a rising block-mean, which alone is not evidence of anything. Hence the gate + * requires magnitude (ratio > 1.10), slope (recentRatio > 1.10) AND not-currently-falling. + * + * `weeksElevated` answers "how long has this been like this?" WITHOUT any persisted state -- + * it is derived from the same 9-week series, so two engineers on two machines get identical + * answers and there is nothing to commit, sync, or go stale. It counts consecutive recent + * weeks above the EARLY-window baseline (median of the first third). A series that never + * left that baseline band reports the full window with `sustainedFullWindow: true`, i.e. + * "at this level for as long as we can see" -- a standing condition, not this week's news. + * The reference is the baseline, NOT the current value: "within 20% of current" is + * meaningless for a flat series and made a code that stepped up this week claim it had been + * elevated for seven. + * + * Input: the --json sidecar written by bucket-trends.js. + * + * Usage: + * node classify-novelty.js <bucket-trends-sidecar.json> + * [--floor=N] # ignore keys whose current complete week is below N (default 5000) + * [--family-sep=_] # token separator for family detection ('none' disables) + * [--top=N] # rows printed per bucket (default 8) + * [--json=<path>] # structured sidecar for programmatic use + * [--summary] # counts + rows only, no series arrays + * + * Classification (evaluated in order, first match wins), computed on COMPLETE weeks only: + * VOLATILE cv > 0.60 -> ratio is noise; suppressRatio=true + * RECOVERY prior week < 50% of median -> returning to band, not a new break + * and current >= 70% of median + * NEW current > 115% of median -> stable baseline then a clean step + * and cv < 0.25 + * ONGOING late-third mean > 115% early-third -> already climbing; not new + * IMPROVING current < 80% of median + * STABLE otherwise + * + * The cv < 0.25 guard on NEW is what stops a jittery series from being called a step + * change. A series must have been genuinely boring before a jump counts as news. + */ +const fs = require('fs'); + +const args = process.argv.slice(2); +const file = args.find(a => !a.startsWith('--')); +const floor = +((args.find(a => a.startsWith('--floor=')) || '').split('=')[1] || 5000); +const topN = +((args.find(a => a.startsWith('--top=')) || '').split('=')[1] || 8); +const jsonOut = (args.find(a => a.startsWith('--json=')) || '').split('=')[1]; +const summary = args.includes('--summary'); +const famSepRaw = (args.find(a => a.startsWith('--family-sep=')) || '').split('=')[1]; +const famSep = famSepRaw === undefined ? '_' : famSepRaw; +const familiesEnabled = famSep !== 'none' && famSep !== ''; + +if (!file) { + console.error('Usage: node classify-novelty.js <bucket-trends-sidecar.json> [--floor=N] [--family-sep=_|none] [--top=N] [--json=path] [--summary]'); + process.exit(1); +} + +const d = JSON.parse(fs.readFileSync(file, 'utf8')); +if (!d.buckets) { + console.error('Input does not look like a bucket-trends.js --json sidecar (no .buckets).'); + process.exit(1); +} + +// ---- stats helpers ------------------------------------------------------- +const mean = a => a.reduce((s, v) => s + v, 0) / a.length; +const median = a => { + const s = [...a].sort((x, y) => x - y); + const n = s.length; + return n % 2 ? s[(n - 1) / 2] : (s[n / 2 - 1] + s[n / 2]) / 2; +}; +const stdev = a => { + const m = mean(a); + return Math.sqrt(mean(a.map(v => (v - m) ** 2))); +}; +const pct = v => (v >= 0 ? '+' : '') + (v * 100).toFixed(1) + '%'; + +/** + * `series` follows DISPLAY weeks, which include the in-progress partial week when + * bucket-trends.js ran with --include-partial-end. Classifying on a partial week would + * read as a fake collapse, so trim to the classify-week count. + */ +const completeLen = (d.classifyWeeks || d.weeks || []).length; + +function classify(code, series) { + const comp = series.slice(0, completeLen); + if (comp.length < 4) return null; // too little history to say anything honest + const cur = comp[comp.length - 1]; + const hist = comp.slice(0, -1); + const prev = hist[hist.length - 1]; + const med = median(hist); + if (med <= 0) return null; + + const m = mean(hist); + const cv = m > 0 ? stdev(hist) / m : 99; + const ratio = cur / med; + + const third = Math.max(1, Math.floor(hist.length / 3)); + const early = mean(hist.slice(0, third)); + const late = mean(hist.slice(-third)); + const climb = early > 0 ? late / early : 1; + + // --- Is it STILL climbing, or did it climb and then plateau? --------------- + // `climb` looks at the whole history, so a code that stepped up 6 weeks ago and has been + // flat ever since still reads as "climbing" forever. That is exactly what made the report + // re-triage the same known issues every week. Compare the most recent block against the + // block before it to separate "getting worse now" from "bad, but stable". + const win = Math.min(3, Math.floor(comp.length / 2)); + const recentBlock = mean(comp.slice(-win)); + const priorBlock = mean(comp.slice(-2 * win, -win)); + const recentRatio = priorBlock > 0 ? recentBlock / priorBlock : 1; + + // --- How long has this been at its current level? -------------------------- + // Reference is the EARLY-window baseline (median of the first third), not the current + // value: "within 20% of current" is meaningless for a low-variance series, where a code + // that stepped up this week still looks like it has been here for the whole window. + // Derived purely from the series already in hand -- NO persisted state, so two engineers + // on two machines get identical answers and there is nothing to commit or sync. + const baseline = median(comp.slice(0, third)) || med; + let weeksElevated, sustainedFullWindow; + if (cur < baseline * 1.15) { + // Never left its own early band inside the observable window. Honest ceiling: + // "at this level for as long as we can see" -- a standing condition, not this week's news. + weeksElevated = comp.length; + sustainedFullWindow = true; + } else { + weeksElevated = 0; + for (let i = comp.length - 1; i >= 0; i--) { + if (comp[i] >= baseline * 1.15) weeksElevated++; + else break; + } + sustainedFullWindow = weeksElevated >= comp.length; + } + + let label; + if (cv > 0.60) label = 'VOLATILE'; + else if (prev < med * 0.5 && ratio >= 0.7) label = 'RECOVERY'; + else if (ratio > 1.15 && cv < 0.25) label = 'NEW'; + else if (climb > 1.15) { + // ACCELERATING must mean "getting worse RIGHT NOW", and by a margin worth a card. + // Two observed false positives this gate exists to stop: + // authorization_pending -- down 37% WoW, 16% BELOW its median, but block means drifted up. + // IntuneAppProtectionPolicyRequiredException -- flat (cv 0.08), only +4.9% vs its own + // median, down 3.7% WoW, yet a slow multi-week ramp made it outrank the real finding. + // So require all three: meaningfully elevated, still climbing, and not currently falling. + // ratio > 1.10 is the magnitude bar -- within 10% of its own median is not a story. + const stillRising = ratio > 1.10 && recentRatio > 1.10 && cur >= prev * 0.95; + label = stillRising ? 'ACCELERATING' : 'ONGOING'; + } + else if (ratio < 0.8) label = 'IMPROVING'; + else label = 'STABLE'; + + return { + code, label, + current: cur, + prior: prev, + median: Math.round(med), + cv: +cv.toFixed(2), + vsMedian: +(ratio - 1).toFixed(3), + wowWeek: prev > 0 ? +((cur - prev) / prev).toFixed(3) : null, + weeksElevated, + sustainedFullWindow, + recentRatio: +recentRatio.toFixed(2), + // A percentage off a depressed or wildly swinging base is not reportable as a headline. + suppressRatio: label === 'VOLATILE' || label === 'RECOVERY', + series: comp, + }; +} + +// ---- classify every key across all trend buckets -------------------------- +const all = []; +let belowFloor = 0; +for (const grp of Object.keys(d.buckets)) { + for (const m of d.buckets[grp]) { + const r = classify(m.code, m.series || []); + if (!r) continue; + if (r.current < floor) { belowFloor++; continue; } + r.trendBucket = grp; + all.push(r); + } +} + +// ---- family clustering ---------------------------------------------------- +// Only cluster when members genuinely move together: >=2 members sharing a prefix AND +// agreeing on direction. Grouping divergent codes would invent a story that isn't there. +const families = []; +if (familiesEnabled) { + const byPrefix = new Map(); + for (const r of all) { + const parts = String(r.code).split(famSep); + if (parts.length < 2) continue; + const p = parts[0]; + if (!byPrefix.has(p)) byPrefix.set(p, []); + byPrefix.get(p).push(r); + } + for (const [prefix, members] of byPrefix) { + if (members.length < 2) continue; + const labels = new Set(members.map(x => x.label)); + if (labels.size !== 1) continue; // must agree on direction + const label = [...labels][0]; + if (label === 'STABLE') continue; // a family of nothing is not news + const len = Math.min(...members.map(x => x.series.length)); + const summed = Array.from({ length: len }, (_, i) => + members.reduce((s, x) => s + x.series[i], 0)); + const cur = summed[len - 1], prev = summed[len - 2]; + families.push({ + family: prefix + famSep + '*', + label, + members: members.map(x => x.code), + current: cur, + prior: prev, + wowWeek: prev > 0 ? +((cur - prev) / prev).toFixed(3) : null, + series: summed, + }); + } +} + +// ---- output --------------------------------------------------------------- +// ORDER is also the report's priority order. Only NEW and ACCELERATING earn a chart and a +// card; everything below the ATTENTION line is reference material that must collapse. +const ORDER = ['NEW', 'ACCELERATING', 'ONGOING', 'VOLATILE', 'RECOVERY', 'IMPROVING', 'STABLE']; +const ATTENTION = ['NEW', 'ACCELERATING']; +const HEAD = { + NEW: 'NEW THIS WEEK (stable baseline -> clean step change; lead with these)', + ACCELERATING: 'ACCELERATING (elevated AND still climbing; promote -- this is getting worse)', + ONGOING: 'ONGOING (elevated but PLATEAUED; known/steady -- collapse, do not re-triage)', + VOLATILE: 'VOLATILE (high variance; % is noise -- report absolute level, not ratio)', + RECOVERY: 'RECOVERY (returning to normal band after a dip; not a regression)', + IMPROVING: 'IMPROVING (below historical band)', + STABLE: 'STABLE (within band)', +}; + +const counts = ORDER.map(l => `${l}=${all.filter(r => r.label === l).length}`).join(' '); +console.log(`\nNovelty classification (floor=${floor.toLocaleString()}, complete weeks=${completeLen}): ${counts}`); +if (belowFloor) console.log(`(${belowFloor} key(s) skipped below floor)`); + +const attention = all + .filter(r => ATTENTION.includes(r.label)) + .sort((a, b) => b.current - a.current); +console.log(`ATTENTION set (NEW + ACCELERATING) = ${attention.length} of ${all.length} series.` + + (attention.length === 0 ? ' -> QUIET WEEK: report should be short.' : '')); + +for (const label of ORDER) { + const rows = all.filter(r => r.label === label).sort((a, b) => b.current - a.current); + if (!rows.length) continue; + console.log(`\n### ${HEAD[label]}`); + for (const r of rows.slice(0, topN)) { + const ratioTxt = r.suppressRatio ? '(ratio suppressed)' : `WoW ${pct(r.wowWeek)}`; + const age = r.sustainedFullWindow ? `>=${r.weeksElevated}w` : `${r.weeksElevated}w`; + console.log(` ${r.code.padEnd(44)} cur=${r.current.toLocaleString().padStart(11)} vs median ${pct(r.vsMedian).padStart(8)} cv=${String(r.cv).padStart(4)} elev=${age.padStart(4)} ${ratioTxt}`); + if (!summary) console.log(` ${r.series.map(v => v.toLocaleString()).join(' ')}`); + } + if (rows.length > topN) console.log(` ... and ${rows.length - topN} more`); +} + +if (families.length) { + console.log(`\n### FAMILIES (related keys moving together -- report as ONE finding, one root cause)`); + for (const f of families.sort((a, b) => b.current - a.current)) { + console.log(` ${f.family.padEnd(44)} ${f.label} cur=${f.current.toLocaleString()} WoW ${pct(f.wowWeek)} [${f.members.join(', ')}]`); + if (!summary) console.log(` ${f.series.map(v => v.toLocaleString()).join(' ')}`); + } +} + +if (jsonOut) { + fs.writeFileSync(jsonOut, JSON.stringify({ + floor, + completeWeeks: completeLen, + weeks: (d.classifyWeeks || d.weeks || []).slice(0, completeLen), + metric: d.metric, + key: d.key, + counts: Object.fromEntries(ORDER.map(l => [l, all.filter(r => r.label === l).length])), + // The report's attention section is exactly this list -- nothing else earns a card or a + // chart. Emitting it here (rather than leaving each playbook to re-derive it) is what + // keeps "what needs attention" from drifting back into a volume-ranked dump. + attentionLabels: ATTENTION, + attention: attention.map(r => r.code), + quietWeek: attention.length === 0, + items: all, + families, + }, null, 2)); + console.log(`\nWrote novelty sidecar -> ${jsonOut}`); +} diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/find-suspect-prs.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/find-suspect-prs.ps1 index 2a6369de..07012671 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/find-suspect-prs.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/find-suspect-prs.ps1 @@ -26,16 +26,26 @@ Inclusive end date. Defaults to today. .PARAMETER RepoRoot - Root folder containing `broker/` and `common/` subfolders. Defaults to the + Root folder containing the repo subfolders. Defaults to the git top-level of the current working directory (so running from any clone of android-complete just works). If you keep your repos elsewhere, pass - -RepoRoot explicitly. The script will FAIL loudly if neither broker/ nor - common/ exists under the resolved root — silent no-match would hide + -RepoRoot explicitly. The script will FAIL loudly if none of the requested + repos exist under the resolved root — silent no-match would hide attribution candidates. +.PARAMETER Repos + Which repos to scan. Valid values: broker, common, authenticator. + Defaults to broker,common (the Broker report's attribution set). + Pass -Repos authenticator for the Authenticator report — its PRs are + ADO-hosted, so the emitted links use the ADO /pullrequest/ form, not + github.com/pull/. + .EXAMPLE .\find-suspect-prs.ps1 -Symbol ExceptionAdapter -Since 2026-04-01 +.EXAMPLE + .\find-suspect-prs.ps1 -Symbol getAllEntries -Repos authenticator -Since 2026-07-01 + .EXAMPLE .\find-suspect-prs.ps1 -Symbol clientExceptionFromException -Since 2026-04-01 -Until 2026-05-09 @@ -49,7 +59,8 @@ param( [string]$GrepRegex, [string]$Since = (Get-Date).AddDays(-28).ToString('yyyy-MM-dd'), [string]$Until = (Get-Date).ToString('yyyy-MM-dd'), - [string]$RepoRoot + [string]$RepoRoot, + [string[]]$Repos = @('broker','common') ) # Resolve repo root: explicit -RepoRoot wins; otherwise discover via `git rev-parse --show-toplevel` @@ -66,30 +77,47 @@ if (-not $RepoRoot) { if (-not $GrepRegex) { $GrepRegex = [regex]::Escape($Symbol) } -$repos = @( - @{ Name='broker'; Path=(Join-Path $RepoRoot 'broker'); UrlBase='https://msft.ghe.com/security/ad-accounts-for-android/pull/' } - @{ Name='common'; Path=(Join-Path $RepoRoot 'common'); UrlBase='https://github.com/AzureAD/microsoft-authentication-library-common-for-android/pull/' } -) +# Known repos this script can scan. Authenticator is ADO-hosted, so its PR URLs use the +# ADO /pullrequest/ form -- NOT a github.com /pull/ URL. validate-report.ps1 fails an +# Authenticator report that contains github.com PR links, so getting this wrong is caught +# downstream, but emit it correctly here in the first place. +$repoCatalog = [ordered]@{ + broker = @{ Dir='broker'; UrlBase='https://msft.ghe.com/security/ad-accounts-for-android/pull/' } + common = @{ Dir='common'; UrlBase='https://github.com/AzureAD/microsoft-authentication-library-common-for-android/pull/' } + authenticator = @{ Dir='authenticator'; UrlBase='https://msazure.visualstudio.com/One/_git/AD-MFA-phonefactor-phoneApp-android/pullrequest/' } +} + +# Accept both -Repos broker,common and -Repos 'broker,common' +$selected = @($Repos | ForEach-Object { $_ -split ',' } | ForEach-Object { $_.Trim().ToLower() } | Where-Object { $_ }) +$unknown = @($selected | Where-Object { -not $repoCatalog.Contains($_) }) +if ($unknown.Count -gt 0) { + Write-Error "Unknown -Repos value(s): $($unknown -join ', '). Valid: $($repoCatalog.Keys -join ', ')." + exit 2 +} + +$repoTargets = @($selected | ForEach-Object { + @{ Name=$_; Path=(Join-Path $RepoRoot $repoCatalog[$_].Dir); UrlBase=$repoCatalog[$_].UrlBase } +}) # FAIL LOUDLY if neither subrepo exists under the resolved root. A silent # "No PRs match..." after a path miss is indistinguishable from a real no-match # result and can silently hide attribution candidates. -$availableRepos = @($repos | Where-Object { Test-Path $_.Path }) +$availableRepos = @($repoTargets | Where-Object { Test-Path $_.Path }) if ($availableRepos.Count -eq 0) { + $expected = ($repoTargets | ForEach-Object { " $RepoRoot\$($_.Name)\" }) -join "`n" Write-Error @" -Neither broker/ nor common/ found under -RepoRoot $RepoRoot. +None of the requested repo(s) [$($selected -join ', ')] were found under -RepoRoot $RepoRoot. Expected layout: - $RepoRoot\broker\ (clone of security/ad-accounts-for-android) - $RepoRoot\common\ (clone of AzureAD/microsoft-authentication-library-common-for-android) +$expected -Pass -RepoRoot pointing at the parent of those two clones. The android-complete -mono-repo at the repo root works because broker/ and common/ are submodules there. +Pass -RepoRoot pointing at the parent of those clones. The android-complete +mono-repo at the repo root works because they are submodules there. "@ exit 2 } -if ($availableRepos.Count -lt $repos.Count) { - $missing = $repos | Where-Object { -not (Test-Path $_.Path) } | ForEach-Object { $_.Name } +if ($availableRepos.Count -lt $repoTargets.Count) { + $missing = $repoTargets | Where-Object { -not (Test-Path $_.Path) } | ForEach-Object { $_.Name } Write-Warning "Skipping $($missing -join ', ') — not found under $RepoRoot. Results will be incomplete." } @@ -111,9 +139,22 @@ foreach ($r in $availableRepos) { $sha = $parts[0] if ($seen.ContainsKey($sha)) { continue } $seen[$sha] = $true - # Try to pull the PR number out of the subject (#NNN at end of MS PR convention) + # Pull the PR id out of the subject. Three conventions must all work, and + # ORDER MATTERS -- these subjects routinely carry an Azure Boards work-item + # id that looks just like a PR id: + # broker/common: "Add telemetry, Fixes AB#3604499 (#3141)" -> want 3141, NOT 3604499 + # authenticator: "Merged PR 16115696: SyncCoordinator ..." -> 8 digits, no '#' + # github merge: "Merge pull request #179 from AzureAD/..." + # A naive '#(\d{2,9})' matches AB#3604499 first and cites the wrong link; + # the original '#(\d{2,5})' only avoided that by accident (AB ids are 7 digits). $prNum = $null - if ($parts[3] -match '#(\d{2,5})\b') { $prNum = $Matches[1] } + if ($parts[3] -match '\(#(\d{2,6})\)') { + $prNum = $Matches[1] # GitHub squash-merge "(#3141)" + } elseif ($parts[3] -match '(?:Merged PR|Merge pull request)\s+#?(\d{2,9})\b') { + $prNum = $Matches[1] # ADO "Merged PR N", GitHub merge commit + } elseif ($parts[3] -match '(?<!AB)#(\d{2,5})\b') { + $prNum = $Matches[1] # legacy fallback, AB#-guarded + } $results += [pscustomobject]@{ Repo = $r.Name Date = $parts[1].Substring(0, 10) diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/run-kql.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/run-kql.ps1 index 3686259f..158f7323 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/run-kql.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/run-kql.ps1 @@ -25,11 +25,21 @@ .PARAMETER Out Output JSON file path. +.PARAMETER App + Convenience selector for the cluster/database pair: 'broker' (default) or + 'authapp'. The two reports read completely different clusters AND different + databases; getting one right and the other wrong returns an empty result set + rather than an error, so prefer -App over passing -Cluster/-Database by hand. + Explicit -Cluster / -Database always win if supplied. + + broker -> https://idsharedeus2.kusto.windows.net / ad-accounts-android-otel + authapp -> https://idsharedeus2.eastus2.kusto.windows.net / d496be22d62a46b0a3cf67ea2e736fd8 + .PARAMETER Cluster - Kusto cluster URI (default: idsharedeus2 — the production Android Broker cluster). + Kusto cluster URI. Overrides the -App default. .PARAMETER Database - Database name (default: ad-accounts-android-otel). + Database name. Overrides the -App default. .PARAMETER TimeoutSec HTTP timeout (default 300 s — Kusto itself has a 5-minute server-side query budget). @@ -52,6 +62,10 @@ materialized_view('ErrorStatsMetrics') "@ .\run-kql.ps1 -Query $q -Out 60d-codes.json +.EXAMPLE + # Authenticator: scenario outcomes WoW + .\run-kql.ps1 -App authapp -Query (Get-Content ..\queries\authapp\scenario-outcomes-wow.kql -Raw) -Out outcomes.json + .NOTES * Requires `az login` to have been run beforehand and the caller to have read access to the cluster (Android Auth Client SDK security group). @@ -65,12 +79,23 @@ materialized_view('ErrorStatsMetrics') param( [Parameter(Mandatory=$true)][string]$Query, [Parameter(Mandatory=$true)][string]$Out, - [string]$Cluster = 'https://idsharedeus2.kusto.windows.net', - [string]$Database = 'ad-accounts-android-otel', + [ValidateSet('broker','authapp')] + [string]$App = 'broker', + [string]$Cluster, + [string]$Database, [int]$TimeoutSec = 300 ) $ErrorActionPreference = 'Stop' +# Per-app cluster/database defaults. Explicit -Cluster/-Database win. +$endpoints = @{ + broker = @{ Cluster = 'https://idsharedeus2.kusto.windows.net'; Database = 'ad-accounts-android-otel' } + authapp = @{ Cluster = 'https://idsharedeus2.eastus2.kusto.windows.net'; Database = 'd496be22d62a46b0a3cf67ea2e736fd8' } +} +if (-not $Cluster) { $Cluster = $endpoints[$App].Cluster } +if (-not $Database) { $Database = $endpoints[$App].Database } +Write-Verbose "Querying $Cluster / $Database (-App $App)" + # Acquire token via az CLI (works for users + managed identity) $tok = az account get-access-token --resource $Cluster --query accessToken -o tsv 2>$null if (-not $tok) { @@ -101,3 +126,8 @@ $obj = @{ results = @{ items = $items } } # UTF-8 without BOM — keeps emoji/diacritic data clean for downstream consumption [IO.File]::WriteAllText($Out, ($obj | ConvertTo-Json -Depth 12 -Compress), [System.Text.UTF8Encoding]::new($false)) Write-Host ("Saved {0} rows -> {1}" -f ($primary.Rows.Count), $Out) +if ($primary.Rows.Count -eq 0) { + # Zero rows is the signature of a wrong time column, wrong database, or a + # window that lands outside retention -- Kusto returns success either way. + Write-Warning "Query returned 0 rows from $Cluster / $Database. Verify the time column (broker: EventInfo_Time; authapp MVs: EventDate; brokeroperations: PipelineInfo_IngestionTime) and that -App matches the views you referenced." +} diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 index c96cd6ff..d3e1f411 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 @@ -35,26 +35,38 @@ Exits with non-zero status if any HARD check fails (stale tokens, devs/reqs leak, U+FFFD, unbalanced div depth, missing layout-guard CSS). +.PARAMETER App + Which report profile to validate: 'broker' (default) or 'authapp'. Checks + 7, 8, and 9 assert Broker-specific card structures and are skipped for + authapp; authapp instead gets its own checks (13-scenario scoreboard + coverage, PN section presence, ADO-not-GitHub PR links). Shared checks + (1-6, 10, 11) run for both. + .PARAMETER Path - Absolute path to the report file. Defaults to the current week's report under - $env:USERPROFILE\android-oce-reports\. + Absolute path to the report file. Defaults to the most recent report matching + the -App file prefix under $env:USERPROFILE\android-oce-reports\. .EXAMPLE .\validate-report.ps1 + .\validate-report.ps1 -App authapp .\validate-report.ps1 -Path C:\path\to\oncall-wow-report-2026-07-09.html #> [CmdletBinding()] param( - [string]$Path + [string]$Path, + [ValidateSet('broker','authapp')] + [string]$App = 'broker' ) -# Default: pick the most-recent oncall-wow-report-*.html in the user's reports folder +$filePrefix = if ($App -eq 'authapp') { 'authapp' } else { 'oncall' } + +# Default: pick the most-recent <prefix>-wow-report-*.html in the user's reports folder if (-not $Path) { $reportDir = Join-Path $env:USERPROFILE 'android-oce-reports' - $latest = Get-ChildItem $reportDir -Filter 'oncall-wow-report-*.html' -ErrorAction SilentlyContinue | + $latest = Get-ChildItem $reportDir -Filter "$filePrefix-wow-report-*.html" -ErrorAction SilentlyContinue | Sort-Object LastWriteTime -Descending | Select-Object -First 1 if (-not $latest) { - Write-Error "No oncall-wow-report-*.html found in $reportDir. Pass -Path explicitly." + Write-Error "No $filePrefix-wow-report-*.html found in $reportDir. Pass -Path explicitly, or check -App (currently '$App')." exit 2 } $Path = $latest.FullName @@ -74,6 +86,7 @@ function Pass($msg) { Write-Host " [OK] $msg" -ForegroundColor Green } Write-Host "" Write-Host "Validating: $Path" +Write-Host ("App profile: {0}" -f $App) Write-Host ("Size: {0:N0} bytes" -f (Get-Item $Path).Length) Write-Host "" @@ -169,11 +182,25 @@ if ($kpiTiles -ge 4 -and $sparkCount -lt [Math]::Ceiling($kpiTiles / 2)) { Pass "KPI tiles have data-spark coverage ($sparkCount/$kpiTiles)" } if ($totalCharts -lt 15) { - Add-Warn "Only $totalCharts chart elements found. Expected ~30+ (KPI sparks + 60d-trend rows + WoW-table rows). Did you forget to add data-trend attributes to the WoW / trend tables?" + # Threshold deliberately LOW. This check guards against a body rebuilt with the + # data-trend attributes dropped entirely -- it is not a "more charts is better" + # signal. The redesign that moved sparklines into the attention rows and collapsed + # the 60-day catalog cut a real report from ~226 chart elements to a few dozen, and + # that is the intended direction: charts belong next to findings. Check 18 enforces + # the upper bound; this one only catches "the charts vanished". + Add-Warn "Only $totalCharts chart elements found. Expect at least one per KPI tile plus one per visible attention row. Did the body get rebuilt without data-trend attributes?" } else { Pass "Overall chart coverage looks reasonable ($totalCharts elements)" } +# ---- 7-9. Broker-only card-structure checks ---- +# These assert the Broker report's .attr-card anatomy (tri-state traffic sub-block, +# 8-field code-attribution block, dim-row layout guards). The Authenticator report +# uses a 4-field attribution block and a different card shape, so running them +# against an authapp report produces pure noise. AuthApp-specific structural checks +# live in section 12 below. +if ($App -eq 'broker') { + # ---- 7. Traffic-attribution sub-block color diversity (tri-state convention) ---- # Per assets/templates/template-readme.md: each .attr-card's traffic sub-block should be green # (ruled out), yellow (partly contributing), or red (primary driver). If every @@ -290,6 +317,11 @@ if ($hasAttrCard) { } } +} # end broker-only checks 7-9 +else { + Write-Host " [SKIP] Checks 7-9 (Broker attribution-card anatomy) - not applicable to -App authapp" +} + # ---- 10. Fabricated-sparkline heuristic (v8 regression — hand-rolled data-trend arrays) ---- # Past failure mode: when 60d bucketer dropped a sub-floor code, the report author # fabricated a "roughly monotonic" 8-week series inline in the WoW table HTML. @@ -302,25 +334,56 @@ if ($hasAttrCard) { $trendMatches = [regex]::Matches($content, "data-trend=['""]?\[([0-9.,e\s+\-]+)\]") $suspectCount = 0 $suspectFirst = $null +$flatCount = 0 +$flatFirst = $null foreach ($m in $trendMatches) { $arrStr = $m.Groups[1].Value $vals = $arrStr.Split(',') | ForEach-Object { try { [double]$_.Trim() } catch { 0 } } if ($vals.Count -lt 6) { continue } - # Filter 1: trend with all values < 100 is suspicious (real codes don't sit at 30-50 devices/week for 8 weeks) $maxVal = ($vals | Measure-Object -Maximum).Maximum - if ($maxVal -lt 100) { - $suspectCount++ - if (-not $suspectFirst) { $suspectFirst = $arrStr } + $minVal = ($vals | Measure-Object -Minimum).Minimum + + # A perfectly flat series is fabricated in ANY app profile -- real telemetry + # never repeats a value 8 weeks running. + if ($maxVal -eq $minVal) { + $flatCount++ + if (-not $flatFirst) { $flatFirst = $arrStr } continue } + + # The low-peak heuristic is COUNT-shaped and only valid for Broker, whose + # data-trend arrays carry devices/week. The Authenticator template plots + # success-RATE series (values legitimately sit in the 85-100 band), so + # applying it there fires on every healthy funnel. Rate series get a + # range check instead: a percentage outside 0-100 is the real defect. + if ($App -eq 'broker') { + # Filter 1: trend with all values < 100 is suspicious (real codes don't sit at 30-50 devices/week for 8 weeks) + if ($maxVal -lt 100) { + $suspectCount++ + if (-not $suspectFirst) { $suspectFirst = $arrStr } + continue + } + } else { + if ($maxVal -le 100 -and $minVal -ge 0) { continue } # plausible rate series + if ($maxVal -lt 100) { + # not a rate series and too small to be a count series + $suspectCount++ + if (-not $suspectFirst) { $suspectFirst = $arrStr } + continue + } + } # Filter 2: zero-padded series like [0,0,0,0,0,0,0,N] is fine (legitimate NEW); skip # Filter 3: implausibly regular - if every consecutive delta has the same sign AND is < 5% of the value, that's a fake. # Skip this; too easy to false-positive on genuinely monotonic real series like no_tokens_found. } +if ($flatCount -gt 0) { + Add-Warn "$flatCount data-trend array(s) are perfectly flat (every value identical). Real telemetry never does this. First: [$flatFirst]." +} if ($suspectCount -gt 0) { - Add-Warn "$suspectCount data-trend array(s) have peak value < 100 (suspicious — real WoW-table series usually peak >= 100 devices/week). Likely fabricated. First: [$suspectFirst]. Source from assets/queries/wow-table-sparkline-series.kql instead." -} else { - Pass "No suspicious low-peak data-trend arrays detected" + $src = if ($App -eq 'authapp') { 'assets/queries/authapp/scenario-sparkline-series.kql' } else { 'assets/queries/wow-table-sparkline-series.kql' } + Add-Warn "$suspectCount data-trend array(s) have an implausible magnitude (peak < 100 and not a 0-100 rate series). Likely fabricated. First: [$suspectFirst]. Source from $src instead." +} elseif ($flatCount -eq 0) { + Pass "No suspicious data-trend arrays detected" } # ---- 11. Rolling-window header integrity ---- @@ -328,7 +391,7 @@ if ($suspectCount -gt 0) { # <Mon> <D>, YYYY", and those dates must be self-consistent with the filename's # end-date (curEnd = filename date, curStart = curEnd - 7d). $filename = Split-Path $Path -Leaf -if ($filename -match '^oncall-wow-report-(\d{4}-\d{2}-\d{2})\.html$') { +if ($filename -match "^$filePrefix-wow-report-(\d{4}-\d{2}-\d{2})\.html`$") { $fnEnd = [datetime]::ParseExact($Matches[1], 'yyyy-MM-dd', [System.Globalization.CultureInfo]::InvariantCulture) $fnStart = $fnEnd.AddDays(-7) @@ -360,7 +423,255 @@ if ($filename -match '^oncall-wow-report-(\d{4}-\d{2}-\d{2})\.html$') { Add-Warn "Meta-line 'Last 7 days: ... -> ...' pattern not found. Either the report predates the rolling-window rewrite or the header was hand-edited. Re-run bootstrap-report.ps1 to restore the auto-stamped meta line." } } else { - Add-Warn "Filename '$filename' does not match 'oncall-wow-report-YYYY-MM-DD.html'; skipping meta-line date consistency check." + Add-Warn "Filename '$filename' does not match '$filePrefix-wow-report-YYYY-MM-DD.html'; skipping meta-line date consistency check. (Check -App: currently '$App'.)" +} + +# ---- 12. Authenticator-only structural checks ---- +if ($App -eq 'authapp') { + # 12a. All 13 scenario funnels must appear in the scoreboard. A scenario that + # silently vanishes from the table is indistinguishable from one that was + # never queried -- which is exactly the failure this check exists to catch. + $scenarios = @( + 'Passkey WebAuthN Registration', + 'Passkey InApp Registration', + 'Passkey WebAuthN Authentication', + 'Entra MFA Registration (QR)', + 'Entra MFA Registration (No-QR)', + 'Entra MFA PN+CFA', + 'Entra PSI Registration', + 'Entra PSI PN Registration', + 'Entra PSI PN+CFA', + 'MSA NGC Registration', + 'MSA SA Registration', + 'MSA NGC PN+CFA', + 'MSA SA PN+CFA' + ) + # Compare on a whitespace/markup-stripped, case-folded projection so a scenario + # split across tags or renamed in case still counts as present. + $flat = ([regex]::Replace($content, '<[^>]+>', ' ')) -replace '\s+', ' ' + $missing = @($scenarios | Where-Object { + $needle = ($_ -replace '\s+', ' ') + $flat.IndexOf($needle, [StringComparison]::OrdinalIgnoreCase) -lt 0 + }) + if ($missing.Count -gt 0) { + Add-Fail "Scenario scoreboard is missing $($missing.Count) of 13 scenario(s): $($missing -join ', '). Every scenario gets a row every week -- including flat and low-volume ones. If a name differs from the canonical label, align the report to it rather than dropping the row." + } else { + Pass "All 13 scenario funnels present in the report" + } + + # 12b. Push-notification section must exist (its own required section). + if ($flat -match '(?i)push[- ]notification' -or $flat -match '(?i)\breacted\b') { + Pass "Push-notification section present" + } else { + Add-Fail "No push-notification / 'reacted' content found. The PN completion + Approved/Denied/Error split is a required section." + } + + # 12c. Unknown/abandonment section must exist -- it has no Broker analogue and + # is the single most commonly dropped section when cloning the Broker layout. + if ($flat -match '(?i)\bunknown\b') { + Pass "Unknown/abandonment content present" + } else { + Add-Fail "No 'Unknown' content found. Unknown = max(0, Initiated - (Succeeded + Failed)) is a first-class metric for this report and requires its own section." + } + + # 12d. PR links must be Azure DevOps, never GitHub. A github.com PR link here + # is the fingerprint of attribution pattern-matched from the Broker playbook. + $ghPr = [regex]::Matches($content, 'https?://github\.com/[^"''<\s]*/pull/\d+') + if ($ghPr.Count -gt 0) { + $sample = (($ghPr | ForEach-Object { $_.Value } | Select-Object -Unique -First 3) -join ' | ') + Add-Fail "$($ghPr.Count) GitHub PR link(s) found. Authenticator lives in Azure DevOps -- PR links must be https://msazure.visualstudio.com/One/_git/AD-MFA-phonefactor-phoneApp-android/pullrequest/<id>. Found: $sample" + } else { + Pass "No GitHub PR links (Authenticator PRs are ADO-hosted)" + } + + # 12e. Broker-convention leakage: HLL/TDigest/materialized_view() do not exist + # in the Authenticator database, so their presence means a Broker query was + # copied in and the numbers below it cannot be trusted. + $brokerisms = @('dcount_hll', 'hll_merge', 'percentile_tdigest', 'tdigest_merge', "materialized_view\(") + $found = @($brokerisms | Where-Object { $content -match $_ }) + if ($found.Count -gt 0) { + Add-Fail "Broker-only Kusto constructs present in the report: $($found -join ', '). These do not exist in the Authenticator database (d496be22d62a46b0a3cf67ea2e736fd8) -- a Broker query was copied in. Use sum(<Scenario>DCount) and bare view names. See assets/docs/authapp-kusto-cheatsheet.md." + } else { + Pass "No Broker-only Kusto constructs leaked into the report" + } +} + +# ---- 13. Section 2 boilerplate uniformity (HARD FAIL) ---- +# The v9 report shipped with the SAME sentence on all 10 red-callout rows +# ("Current-window movement needs owner triage; deep dive below has originator and +# dimensions."), plus 10 identical slow-burn bodies and 8 identical win bodies. +# The reader complaint that produced this check was, verbatim: "it's actually hard +# to understand what's actually changed from last week ... no idea what's existing +# regression vs new this week". A body that would read identically on any other row +# carries zero information -- it is worse than no body, because it occupies the slot +# where the specific finding should have been. +$attStart = $content.IndexOf('id="attention"') +$attEnd = $content.IndexOf('id="trend60d"') +if ($attEnd -lt 0) { $attEnd = $content.IndexOf('id="trend"') } +if ($attStart -ge 0 -and $attEnd -gt $attStart) { + $attSec = $content.Substring($attStart, $attEnd - $attStart) + + # NOTE: 'Singleline' is load-bearing. Without it `.` does not cross newlines, so this + # only matched when the generating agent happened to emit .item-body on ONE line. The + # template pretty-prints these blocks across several lines, so a copied-from-template + # row would silently skip the check -- a check that no-ops while printing nothing is + # worse than no check. Same applies to the .item-head regex below. + $bodies = [regex]::Matches($attSec, '<div class="item-body">(.*?)</div>', 'Singleline') | + ForEach-Object { + # Strip inline markup + collapse whitespace so two bodies that differ + # only by a <code> wrapper still count as duplicates. + (([regex]::Replace($_.Groups[1].Value, '<[^>]+>', ' ')) -replace '\s+', ' ').Trim() + } | Where-Object { $_.Length -gt 0 } + + if ($bodies.Count -eq 0) { + Add-Warn "No .item-body rows found in Section 2 -- cannot check for boilerplate. If the attention section uses different markup, this check needs updating." + } else { + $dupes = $bodies | Group-Object | Where-Object { $_.Count -ge 3 } | Sort-Object Count -Descending + if ($dupes) { + foreach ($d in $dupes) { + $snip = $d.Name.Substring(0, [Math]::Min(80, $d.Name.Length)) + Add-Fail "Section 2 boilerplate: $($d.Count) attention rows share the identical body `"$snip...`". Every row must say what changed, from what to what, and why it is or isn't alarming. If there is nothing row-specific to say, the row does not belong in Section 2." + } + } else { + Pass "Section 2 row bodies are row-specific (no sentence repeated 3+ times)" + } + } + + # ---- 14. Top attention row should not be a flat mover (WARN) ---- + # Ranking the attention list by device count is how a code with a +0.1% WoW delta + # ended up at position #1 while the genuinely new ipc_* family sat at #6/#9/#10. + # Section 2 is ordered by NOVELTY (classify-novelty.js), not volume -- so a + # near-zero delta in the lead slot means the ordering was never applied. + # + # The delta label has been observed spelled "Δ WoW", "Δ WoW" and "Delta WoW" + # across runs. Match all of them: a check that silently no-ops on a spelling + # variant is worse than no check, because it reports nothing and looks healthy. + $dLabel = '(?:Δ|Δ|\u0394|Delta)\s*WoW' + $heads = [regex]::Matches($attSec, '<div class="item-head">(.*?)</div>', 'Singleline') + if ($heads.Count -eq 0) { + Add-Warn "No .item-head rows found in Section 2 -- lead-row and volatile-chip checks skipped. If the attention markup changed, these checks need updating." + } else { + $firstHead = $heads[0].Groups[1].Value + $nameM = [regex]::Match($firstHead, '<span class="item-name">\s*([^<]+)') + $nm = if ($nameM.Success) { $nameM.Groups[1].Value.Trim() } else { '(unnamed)' } + if ($App -eq 'authapp') { + # Authenticator Section 3 rows are success/abandonment RATES, so their movement + # is expressed in percentage POINTS, not a WoW ratio. There is no Δ WoW chip to + # read and a 5% flatness threshold has no meaning against pp deltas. + Write-Host " [SKIP] Check 14 (lead-row flatness) - Section 3 uses percentage-point deltas, not Delta WoW" + } else { + $dw = [regex]::Match($firstHead, "$dLabel</span><span class=`"m-value`">([+\-]?[\d.]+)%") + if (-not $dw.Success) { + Add-Warn "Could not parse a WoW delta chip on the top attention row '$nm' -- lead-row flatness check did not run. Expected an m-label matching 'Delta WoW'." + } elseif ([Math]::Abs([double]$dw.Groups[1].Value) -lt 5) { + Add-Warn "Top attention row '$nm' has a WoW delta of $($dw.Groups[1].Value)% -- essentially flat. Section 2 must lead with what is NEW (classify-novelty.js label NEW), not with the highest-volume row. If genuinely nothing is new this week, say so explicitly instead of promoting a flat mover." + } else { + Pass "Top attention row '$nm' has a material WoW delta ($($dw.Groups[1].Value)%)" + } + } + + # ---- 15. VOLATILE / RECOVERY rows must not headline a WoW percentage ---- + # These carry suppressRatio:true because their ratio is measured off an + # anomalous prior week. 429 shipped as "+401.8%" while sitting 94.5% BELOW + # its own 60-day median -- tagging the row VOLATILE and caveating in the body + # is NOT sufficient, because the chip row is the loudest element and the eye + # reads it first. Replace the Δ WoW chip with a "vs 60d median" chip. + $offenders = @() + foreach ($h in $heads) { + $hv = $h.Groups[1].Value + if ($hv -notmatch '>(VOLATILE|RECOVERY)<') { continue } + $m = [regex]::Match($hv, "$dLabel</span><span class=`"m-value`">([+\-]?[\d.]+)%") + if ($m.Success -and [Math]::Abs([double]$m.Groups[1].Value) -ge 25) { + $n2 = [regex]::Match($hv, '<span class="item-name">\s*([^<]+)') + $who = if ($n2.Success) { $n2.Groups[1].Value.Trim() } else { '(unnamed)' } + $offenders += "$who ($($m.Groups[1].Value)%)" + } + } + if ($offenders.Count -gt 0) { + Add-Fail "VOLATILE/RECOVERY rows still headline a WoW percentage: $($offenders -join ', '). Their ratio is an artifact of a depressed prior week and the chip is the first thing a reader sees -- a caveat in the body does not undo it. Replace the 'Δ WoW' chip with a 'vs 60d median' chip showing the absolute level's position in the band." + } else { + Pass "No VOLATILE/RECOVERY row headlines a WoW percentage" + } + } + + # ---- 16. Every VISIBLE attention row must carry its own sparkline (HARD FAIL) ---- + # The 2026-07-31 report shipped 13 attention rows with ZERO charts, while the 60-day + # catalog below it carried 38. The reader's complaint was exactly that inversion: + # "I don't see any graphs for the things that need attention this week". + # + # This is not decoration. "+25.7% WoW" is unreadable on its own -- the reader cannot + # tell a flat-for-seven-weeks series that just stepped up (a real regression) from one + # that has been bouncing all along (noise wearing a big percentage). The 9-week shape + # IS the novelty argument, so it has to sit in the row that makes the claim. + # + # Rows inside a collapsed <details> fold are reference material and are exempt. + $attVisible = [regex]::Replace($attSec, '<details\b.*?</details>', '', 'Singleline') + $visHeads = [regex]::Matches($attVisible, '<div class="item-head">(.*?)</div>', 'Singleline') + if ($visHeads.Count -eq 0) { + Add-Warn "No visible .item-head rows in Section 2 -- sparkline-coverage check skipped." + } else { + $noSpark = @() + foreach ($h in $visHeads) { + $hv = $h.Groups[1].Value + if ($hv -match 'item-spark' -or $hv -match 'data-trend') { continue } + $n = [regex]::Match($hv, '<span class="item-name">\s*([^<]+)') + $noSpark += $(if ($n.Success) { $n.Groups[1].Value.Trim() } else { '(unnamed)' }) + } + if ($noSpark.Count -gt 0) { + Add-Fail "$($noSpark.Count) of $($visHeads.Count) visible attention row(s) have no inline sparkline: $($noSpark -join ', '). Every attention row must carry a .item-spark with data-trend holding its 9-week series -- the shape is what distinguishes a step change from ordinary variance. Charts belong WITH the finding, not in a separate browsable section." + } else { + Pass "All $($visHeads.Count) visible attention row(s) carry an inline sparkline" + } + + # ---- 17. Attention section must stay short (WARN) ---- + # classify-novelty.js emits an ATTENTION set (NEW + ACCELERATING). On real broker + # data that is 5 series out of 53. If the rendered section is much larger than + # that, the classifier's set was ignored and volume ranking crept back in -- + # which is what produced 13 rows for what turned out to be one incident. + if ($visHeads.Count -gt 8) { + Add-Warn "Section 2 has $($visHeads.Count) visible attention rows. classify-novelty.js's ATTENTION set (NEW + ACCELERATING) is typically 3-6 series; anything much larger means plateaued or flat codes were promoted. Move ONGOING/STABLE rows into the collapsed fold -- a long attention list trains the reader to skim the one section that must not be skimmed." + } else { + Pass "Section 2 attention list is short ($($visHeads.Count) visible row(s))" + } + } +} else { + Add-Warn "Could not locate the attention section -- skipping boilerplate + lead-row checks." +} + +# ---- 18. The 60-day section must not be a browsable chart catalog (HARD FAIL) ---- +# The 60-day pass exists to catch slow burns that week-over-week structurally cannot see +# (it found `Failed to parse JWT` going 7 -> 3,461 devices). That is its whole job. +# +# What it must NOT be is a scrollable list of every classified code with a chart each. +# Measured on the 2026-07-31 report: 29 rows / 38 charts in this section, of which 27 were +# duplicated verbatim into the error-code and error-type tables further down -- ~93% +# redundant. Combined with the error tables the report rendered 100 sparklines for a single +# incident, and the four series that actually changed were indistinguishable from the rest. +# +# Rule: only findings get charts in the main flow, and a finding is either NEW/ACCELERATING +# (already charted in Section 2) or a promoted slow burn. Everything else goes in a +# collapsed fold WITHOUT charts. Charts inside <details> are exempt -- the reader opted in. +# +# APP-AWARE CAP. Broker's universe is 40-50 error codes, so charting "everything" is the +# catalog failure this check exists to stop -> cap 6. Authenticator's universe is a FIXED +# ~13 scenarios and that table IS the scoreboard, not a browsable overflow -- charting all +# of them is correct there. Its cap is set just above the scenario count so the check still +# fires if someone starts fanning per-error-reason charts into the section. +$t60Cap = if ($App -eq 'authapp') { 16 } else { 6 } +$t60Start = $content.IndexOf('id="trend60d"') +if ($t60Start -ge 0) { + $t60End = $content.IndexOf('<h2', $t60Start + 10) + if ($t60End -lt 0) { $t60End = $content.Length } + $t60 = $content.Substring($t60Start, $t60End - $t60Start) + $t60Visible = [regex]::Replace($t60, '<details\b.*?</details>', '', 'Singleline') + $t60Charts = [regex]::Matches($t60Visible, 'data-trend=|<svg\b').Count + if ($t60Charts -gt $t60Cap) { + Add-Fail "The 60-day section renders $t60Charts chart(s) outside any collapsed fold (cap $t60Cap for -App $App). It is a slow-burn DETECTOR, not a catalog -- chart only the entries it flags that are NOT already in Section 2, and move the full classification into a <details> fold with no charts. A previous run shipped 38 charts here against 0 in the attention section, which is precisely backwards." + } else { + Pass "60-day section is a detector, not a catalog ($t60Charts visible chart(s), cap $t60Cap)" + } +} else { + Add-Warn "Could not locate the 60-day section -- skipping the chart-catalog check." } Write-Host "" diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/authapp-report-template.html b/.github/skills/oncall-weekly-telemetry-report/assets/templates/authapp-report-template.html new file mode 100644 index 00000000..f4b6918e --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/authapp-report-template.html @@ -0,0 +1,915 @@ +<!DOCTYPE html> +<!-- TEMPLATE - Authenticator app report structural reference. This file is a realistic populated + prior-window report kept as the canonical layout for the Authenticator half of the weekly + Android OCE report. Bootstrap stamps the rolling window into the title and meta block; + authors then edit the KPI values, scenario rows, attribution prose, and PR citations in place. --> +<html lang="en"> +<head> +<meta charset="UTF-8"> +<title>Authenticator Android · On-Call Weekly Report · Jul 23 → Jul 30, 2026 + + + +
+ +
+
+

Microsoft Authenticator (Android) · On-Call Report

+
+ Last 7 days: Thu Jul 23 → Thu Jul 30, 2026  vs  Thu Jul 16 → Thu Jul 23  ·  + 60-day trend: Sun May 31 → Wed Jul 29, 2026 (last 60 days; final bar in progress)  ·  + Source: Authenticator scenario materialized views  ·  + Generated 2026-07-30 +
+
+ v1 · Live data +
+ +
+
↔️ Companion report: Android Broker →
+

Use the same reporting window when cross-checking Broker API regressions or push-notification completion changes.

+
+ + + + +

📊 Top-line health — Authenticator scenario denominator

+
+
Telemetry-active devices
22.8 M
+1.9% WoW
+
Total scenario initiates
55.4 M
+3.2% WoW
+
Overall scenario success rate
97.64%
−0.42 pts WoW
+
Overall unknown / abandonment rate
1.18%
+0.29 pts WoW
+
Broker-API success rate
99.12%
−0.31 pts WoW
+
Crash rate per 1k devices
0.42
+0.05 WoW
+
+ + +

Scenario scoreboard — one row per Authenticator scenario family

+
+ + + + + + + + + + + + + + + + +
ScenarioInitiatedSuccess rateΔ success rate (pts)Failure rateUnknown rateΔ unknown (pts)Devices8-week sparklineStatus
Passkey WebAuthN Registration48,32091.8%+0.64.7%3.5%-0.338,940
healthy
Passkey InApp Registration22,41088.2%-0.46.9%4.9%+0.219,120
watch
Passkey WebAuthN Authentication612,90094.1%-1.43.2%2.7%+0.8402,100
regressed
Entra MFA Registration (QR)1,245,00097.6%+0.31.2%1.2%-0.2902,400
healthy
Entra MFA Registration (No-QR)336,20095.0%-0.12.8%2.2%0.0248,900
flat
Entra MFA PN+CFA42,800,00098.4%-0.90.7%0.9%+0.518,700,000
regressed
Entra PSI Registration188,50093.6%+1.24.4%2.0%-0.6131,800
win
Entra PSI PN Registration74,90092.1%+0.45.1%2.8%-0.258,600
healthy
Entra PSI PN+CFA4,920,00096.8%-0.61.4%1.8%+0.42,860,000
watch
MSA NGC Registration91,30089.5%+0.87.2%3.3%-0.472,100
healthy
MSA SA Registration19,80086.4%+0.28.4%5.2%-0.116,400
flat
MSA NGC PN+CFA3,760,00097.2%+0.51.1%1.7%-0.41,940,000
win
MSA SA PN+CFA1,185,00095.4%-0.22.2%2.4%+0.1764,000
flat
+ + + + +

🚨 Needs attention

+ + + +
+
🔴 New this week — clean step change off a flat baseline
+
+
+
+ EXAMPLE_scenario_name + + 9 wk + success ΔEXAMPLE −2.1 pts + initiatedEXAMPLE 42.8 M + NEWhigh volume +
+
EXAMPLE — flat at 96.2-96.4% for seven weeks, then 94.1% this week. State the shape, the size, and the first dimension that explains it. A body that would read identically on another row is a validator failure.
+
Owner: EXAMPLEAttribution card →
+
+ +
+
+ +
+
🟠 Getting worse — already degraded and still sliding
+
+
+
+ EXAMPLE_accelerating_scenario + + 9 wk + success ΔEXAMPLE −0.8 pts + vs 60d medianEXAMPLE −2.4 pts + ACCELERATINGelevated 5w +
+
EXAMPLE — this is the "known issue is deteriorating" bucket. It is only here because it is STILL sliding; a scenario that is degraded but level belongs in the fold. Say how much steeper this week is than the prior slope.
+
Owner: EXAMPLEAttribution card →
+
+ +
+
+ +
+ Ongoing / known — EXAMPLE_N scenarios still below their baseline, none accelerating +
+ +
+
+
+ EXAMPLE_ongoing_scenario + successEXAMPLE 96.8% + ONGOINGelevated 6w +
+
EXAMPLE — down ~0.6 pts since May and flat since; tracked under EXAMPLE_bug. No change this week.
+
+
+
+
+ +
+
🟢 Wins — recoveries and quality gains
+
+
+
+ EXAMPLE_recovered_scenario + + 9 wk + success ΔEXAMPLE +1.2 pts + devicesEXAMPLE 132 K + recovered +
+
EXAMPLE — a win is a shape claim too, so it carries a sparkline. Name the fix or PR that landed, and say whether the gain has held for more than one week.
+
Owner: EXAMPLEAttribution card →
+
+
+
+ + + + +

📈 60-day per-scenario trend — weekly buckets; final bar in progress

+
+ + + + + + + + + + + + + + + + +
ScenarioFirst weekLast weekFirst → last deltaWeekly sparklineClassification
Passkey WebAuthN Registration89.7%91.8%+2.1 pts
flat
Passkey InApp Registration87.9%88.2%+0.3 pts
flat
Passkey WebAuthN Authentication96.4%94.1%-2.3 pts
regression
Entra MFA Registration (QR)97.1%97.6%+0.5 pts
flat
Entra MFA Registration (No-QR)95.1%95.0%-0.1 pts
flat
Entra MFA PN+CFA99.1%98.4%-0.7 pts
spike
Entra PSI Registration91.9%93.6%+1.7 pts
improvement
Entra PSI PN Registration91.2%92.1%+0.9 pts
flat
Entra PSI PN+CFA97.4%96.8%-0.6 pts
slow regression
MSA NGC Registration88.1%89.5%+1.4 pts
flat
MSA SA Registration85.8%86.4%+0.6 pts
flat
MSA NGC PN+CFA96.2%97.2%+1.0 pts
improvement
MSA SA PN+CFA95.3%95.4%+0.1 pts
flat
+ + +

🔎 Error attribution cards — regressed Authenticator scenarios

+
Authenticator scenario attribution uses exactly three dimensions: AppVersion, OsLevel, and DeviceInfoMake. Broker-only calling-app, active-broker, account-type, and shared-device-mode dimensions are intentionally not used here.
+
+
+
+
+
Entra MFA PN+CFA
+
Success 99.0% → 98.4% WoW (−0.6 pts); unknown 0.4% → 0.9% (+0.5 pts); 42.8 M initiates.
+
+
WoW regression60d risingpush reacted split
+
+
+

Top error reasons

+
+ + + + +
ReasonCurrent errorsPrev errorsΔ%DevicesNEW
NotificationReactionTimeout318,400172,900+84%221,000seen
CfaNumberMismatch74,20051,900+43%62,100seen
PushChannelRegistrationExpired31,6006,900+358%29,500NEW
+
+
AppVersion
6.2507.512367.8%
+
6.2507.480119.4%
+
6.2506.42108.1%
+
other4.7%
+
OsLevel
Android 1441.7%
+
Android 1532.6%
+
Android 1318.5%
+
other7.2%
+
DeviceInfoMake
Samsung54.9%
+
Xiaomi16.4%
+
Google12.1%
+
Oppo7.7%
+
+
Verdict — High-volume regression with a version-plus-OEM concentration. The regression is mostly in the latest production version and over-indexes on Samsung devices, with the dominant symptom being no terminal reaction after the notification is delivered. Cross-check the companion Broker report before assigning ownership because this family depends on Broker callback timing.
+
+
Authenticator attribution
+ + + + + + + +
Likely causePush reaction callback not closing the MFA session before the scenario window expires.
ConcentrationAppVersion 6.2507.5123 carries 67.8% of current errors; Samsung carries 54.9%.
Suspect PRsPR 1827349 push reaction debounce cleanup (medium).
Next step (named owner)Aditi Rao to compare callback latency against Broker API completion in the companion report and validate Samsung notification-channel state.
+
+
+
+
+
+
+
Passkey WebAuthN Authentication
+
Success 95.5% → 94.1% WoW (−1.4 pts); unknown 1.9% → 2.7% (+0.8 pts); 612.9 K initiates.
+
+
WoW regressionpasskeyAndroid 15 skew
+
+
+

Top error reasons

+
+ + + + +
ReasonCurrent errorsPrev errorsΔ%DevicesNEW
CredentialProviderSilentCancel12,8406,320+103%10,900seen
WebAuthnChallengeMismatch4,7101,880+151%3,960NEW
AuthenticatorAttachmentMissing2,1801,940+12%1,870seen
+
+
AppVersion
6.2507.512358.2%
+
6.2507.480123.0%
+
6.2506.421012.6%
+
other6.2%
+
OsLevel
Android 1561.5%
+
Android 1426.0%
+
Android 138.8%
+
other3.7%
+
DeviceInfoMake
Google38.9%
+
Samsung34.5%
+
OnePlus9.7%
+
other16.9%
+
+
Verdict — Passkey authentication regression is concentrated on Android 15. The AppVersion split is broad enough that OS behavior is the stronger signal. The new challenge-mismatch reason points to the passkey handoff path rather than network or service failure.
+
+
Authenticator attribution
+ + + + + + + +
Likely causeCredential Manager response handling changed under Android 15 and leaves some sessions without a terminal result.
ConcentrationOsLevel Android 15 carries 61.5% of current errors against roughly 39% scenario share.
Suspect PRsPR 1826012 passkey assertion parcel normalization (low).
Next step (named owner)Miguel Santos to sample Android 15 traces and verify whether Credential Manager returns cancellation before the app records failure.
+
+
+
+
+
+
+
Entra PSI PN+CFA
+
Success 97.4% → 96.8% over 60 days (−0.6 pts); unknown 1.4% → 1.8% (+0.4 pts); 4.92 M initiates.
+
+
slow-burnPSIno single version
+
+
+

Top error reasons

+
+ + + + +
ReasonCurrent errorsPrev errorsΔ%DevicesNEW
PsiSessionExpiredBeforeCfa41,90027,100+55%36,400seen
PhoneSignInNonceMissing18,30015,600+17%14,900seen
PushApproveStateLost7,8001,200+550%7,100NEW
+
+
AppVersion
6.2507.512345.5%
+
6.2507.480132.4%
+
6.2506.421015.1%
+
other7.0%
+
OsLevel
Android 1437.1%
+
Android 1535.8%
+
Android 1319.5%
+
other7.6%
+
DeviceInfoMake
Samsung33.8%
+
Google20.6%
+
Xiaomi18.9%
+
other26.7%
+
+
Verdict — Broad slow-burn with no single dominant dimension. This looks like a session-lifetime edge rather than a rollout-specific code regression. Keep it on watch, but avoid paging unless the unknown rate crosses 2.5% or the error rate grows another 0.5 points.
+
+
Authenticator attribution
+ + + + + + + +
Likely causePSI session expiry racing with push approval state persistence.
ConcentrationNo dimension exceeds 46%; issue is broad across app versions, OS levels, and device makes.
Suspect PRsPR 1824180 PSI state cache compaction (low).
Next step (named owner)Priya Nair to extend PSI session-expiry query by tenant ring and check whether the new cache compaction is enabled.
+
+
+
+
+ + +

Unknown / abandonment analysis

+
Definition: Unknown = Initiated − (Succeeded + Failed), meaning a session started but produced no terminal result inside the reporting window. This is an Authenticator-specific abandonment signal with no Broker equivalent.
+
+ + + + + + + + +
ScenarioUnknown rateΔ ptsDevices affected8-week sparkline
Passkey WebAuthN Authentication2.7%+0.871,200
Passkey InApp Registration4.9%+0.21,080
Entra MFA PN+CFA0.9%+0.5386,000
Entra PSI PN+CFA1.8%+0.488,400
MSA SA Registration5.2%-0.1820
+ + +

Push-notification reacted split

+
ApprovedDeniedError
+
+ + + + + + + +
FamilyNotifications initiatedReactedCompletion rateApproved %Denied %Error %WoW deltas
Entra MFA PN+CFA
42,800,00041,920,00097.9%91.2%6.1%2.7%completion −0.8 pts · error +0.5 pts
Entra PSI PN+CFA
4,920,0004,780,00097.2%88.6%7.4%4.0%completion −0.4 pts · error +0.3 pts
MSA NGC PN+CFA
3,760,0003,690,00098.1%93.5%4.1%2.4%completion +0.6 pts · denied −0.3 pts
MSA SA PN+CFA
1,185,0001,126,00095.0%86.0%9.8%4.2%completion −0.1 pts · flat
+ + +

Broker API responsiveness

+
⚠️ Cross-check required

Any regression in this section must be cross-checked against the companion Broker report for the same window before it is attributed to Authenticator.

+
+ + + + + + + + +
BrokerApiNameInitiatedCompletedFailedSuccess ratep50 msp95 msp99 msDevicesWoW deltas
GetAccounts8,420,0008,374,00046,00099.45%10441811806,880,000success −0.18 pts · p95 +36 ms
AcquireTokenSilent5,310,0005,241,00069,00098.70%18891024104,260,000success −0.42 pts · p99 +210 ms
BrokerDeviceMode2,980,0002,976,0004,00099.87%34821902,410,000flat
GetCurrentAccount1,640,0001,637,0003,00099.82%421303101,220,000flat
RemoveAccount118,000116,9001,10099.07%965101600104,000success +0.20 pts
+ + +

Crash & stability (App Center)

+
ℹ️ App Center token note

This section is skipped when the App Center token is unavailable; keep the visible note so readers know crash telemetry was not collected for the run.

+ +
+
Crashes per 1,000 active devices
0.42
+0.05 WoW
+
Crash-free active devices
99.958%
flat WoW
+
+
+ + + + + + +
ClusterCountAffected devicesFirst seenApp version
PushApprovalActivity.onResume NullReference4,8203,9402026-07-246.2507.5123
CredentialProviderBridge.parseResponse IllegalState2,1401,8602026-07-226.2507.4801
PsiSessionStore.commit TimeoutCancellation1,3801,0202026-07-256.2507.5123
+ + +

Version adoption + PR attribution

+
+ + + + + + + + +
AppVersionDevicesShare %Δ share pts
6.2507.512312,840,00056.3%+18.4
6.2507.48015,960,00026.1%-9.8
6.2506.42102,380,00010.4%-5.1
6.2505.3902910,0004.0%-2.1
Older730,0003.2%-1.4
+
+
medium
PR 1827349 Debounce push reaction callbacks after CFA approval
Merged 2026-07-21 · first broadly visible in 6.2507.5123
Touches the path implicated by the Entra MFA PN+CFA unknown-rate regression.
+
low
PR 1826012 Normalize passkey assertion parcel before Credential Manager handoff
Merged 2026-07-18 · aligns with Android 15 passkey signature
Code path matches, but OS-level concentration is stronger than app-version concentration.
+
none
No direct PR MSA NGC PN+CFA win appears traffic-quality driven
No code change required
Approval mix improved across all live versions with no rollout-aligned step change.
+
+ + +

Appendix

+
+ Query provenance +
+

Cluster: https://idsharedeus2.eastus2.kusto.windows.net · Database: d496be22d62a46b0a3cf67ea2e736fd8

+

Resolved windows: current 2026-07-23T00:00:00Z → 2026-07-30T00:00:00Z; baseline 2026-07-16T00:00:00Z → 2026-07-23T00:00:00Z; 60-day 2026-05-31T00:00:00Z → 2026-07-30T00:00:00Z.

+
    +
  • assets/queries/authapp/scenario-outcomes-wow.kql
    • +
  • assets/queries/authapp/scenario-sparkline-series.kql
    • +
  • assets/queries/authapp/scenario-60d-trend.kql
    • +
  • assets/queries/authapp/scenario-errors-wow.kql
    • +
  • assets/queries/authapp/scenario-errors-by-dim.kql
    • +
  • assets/queries/authapp/pn-completion-wow.kql
    • +
  • assets/queries/authapp/broker-api-responsiveness-wow.kql
    • +
  • assets/queries/authapp/version-share-wow.kql
    • +
+
+
+ +
+ + + + + diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/index-template.html b/.github/skills/oncall-weekly-telemetry-report/assets/templates/index-template.html new file mode 100644 index 00000000..4078bf2d --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/index-template.html @@ -0,0 +1,89 @@ + + + + + +<!--INDEX:TITLE--> + + + +
+ +

Android Auth · Weekly On-Call Digest

+
+
Generated  ·  Live data
+ +
+ This page is an index, not an analysis. Every number below is lifted verbatim + from the linked reports — open a report to see the attribution, dimension slicing, and + suspect PRs behind it. Cross-app findings are written into both reports, not here. +
+ + + +
+ Built by assets/scripts/build-index.ps1 from the two reports linked above. + Reports live outside the repo under ~/android-oce-reports/ and are never committed. +
+ +
+ + diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html b/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html index cf14c921..f3b563bc 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html @@ -389,6 +389,52 @@ margin-left: auto; color: #0550ae; text-decoration: none; font-weight: 600; } .item-foot .arrow-link:hover { text-decoration: underline; } + + /* ---- Novelty presentation -------------------------------------------------- + The sparkline IS the argument. A reader cannot judge "+25% WoW" without + seeing whether the series was flat for seven weeks or bouncing all along, + so every attention row carries its own 9-week shape inline. Charts live + WITH the finding, not in a separate browsable catalog. */ + .item-spark { + display: inline-flex; align-items: center; line-height: 0; + padding: 2px 6px; border-radius: 6px; + background: #fff; border: 1px solid #eaeef2; + } + .item-spark svg { display: block; } + .spark-cap { + font-size: 10px; color: #8c959f; font-weight: 500; + letter-spacing: 0.2px; white-space: nowrap; + } + .tag-new { background: #ffeef0; color: #cf222e; border: 1px solid #ffc1c0; } + .tag-accel { background: #fff1e5; color: #bc4c00; border: 1px solid #ffd8b5; } + .tag-ongoing { background: #f0f3f6; color: #656d76; } + + /* ---- Collapsed reference material ------------------------------------------ + Anything that is elevated-but-plateaued, flat, or already-known is reference, + not this week's news. It stays reachable (an OCE may need to confirm a level) + but must not occupy vertical space or imply "triage me". */ + details.fold { + margin-top: 10px; border: 1px solid #eaeef2; + border-radius: 6px; background: #fbfcfd; + } + details.fold > summary { + cursor: pointer; padding: 8px 12px; font-size: 12.5px; + font-weight: 600; color: #1f2328; list-style: none; user-select: none; + } + details.fold > summary::-webkit-details-marker { display: none; } + details.fold > summary::before { content: "\25B8 "; color: #656d76; } + details.fold[open] > summary::before { content: "\25BE "; } + details.fold > summary .count { font-weight: 400; color: #656d76; } + details.fold > .fold-body { padding: 0 12px 12px; } + details.fold > .fold-body > .item-list { margin-top: 0; } + + /* ---- Quiet week ------------------------------------------------------------- + A week with nothing new must LOOK like a week with nothing new. */ + .quiet-banner { + border: 1px solid #1a7f37; background: #dafbe1; color: #116329; + border-radius: 6px; padding: 12px 14px; margin: 12px 0; font-size: 13px; + } + .quiet-banner strong { color: #1a7f37; } @@ -475,6 +521,19 @@

📊 Top-line health — auth-only denominat

🚨 Things that need attention this week

+ +
ℹ️ Denominator caveat — read this first

The headline BrokerAdoptionStats device count dropped −18.6% WoW (1.52 B → 1.24 B), but this is not a real fleet shrink. The drop is fully explained by three low-value spans deflating as the 16.0.1 rollout completes:

@@ -487,54 +546,82 @@

🚨 Things that need attention this week

-
🔴 WoW regressions (last 7 days vs prior 7) — sorted by current-week devices, descending
-

Tags: NEW first appeared this week or last; 60d↑ also rising on the 60-day window; broker / eSTS / Android / env = originator. Built from the standard WoW table union with wow-movers.kql so small-but-recent spikes appear alongside the high-volume movers.

+
🔴 NEW this week — flat baseline, then a clean step change. Triage these first.
+

Ordered by novelty, not volume — a flat-but-huge code is not news. Labels come from classify-novelty.js, which scores every series against its own 7-week baseline. The sparkline is the argument: a flat line that steps up at the right edge is a real regression; a line that was already bouncing is noise wearing a big percentage. broker / eSTS / Android / env = originator.

- +
EXAMPLE_error_code - devicesEXAMPLE 65 K - Δ WoWEXAMPLE +6.1% - on 16.0.1EXAMPLE 73% + + 9 wk + devicesEXAMPLE 52.1 K + Δ WoWEXAMPLE +25.7% + vs own medianEXAMPLE +19.4% + NEW broker - 60d↑ +52%
-
EXAMPLE one-line narrative: throw site common/SomeClass.someMethod:NN, dominant message, and the verdict. Keep this short — the deep dive is in the attribution card below.
+
EXAMPLE one-line narrative: flat at ~41 K for seven weeks (cv 0.03), stepped to 52 K this week. Throw site common/SomeClass.someMethod:NN. Keep this short — the deep dive is in the attribution card below.
Owner: EXAMPLE teamAttribution card →
+
-
🟡 Slow-burn 60-day regressions — rising on 60d window but flat WoW; codes that also moved WoW are in the red callout above with a 60d↑ tag
+
🟠 Getting worse — already elevated and still climbing. Not new, but the trend has not turned.
- EXAMPLE_slow_burn_code - devicesEXAMPLE 4.5 M - Δ 60dEXAMPLE +56% - Δ requests 60dEXAMPLE +40% - on 16.0.1EXAMPLE 78% + EXAMPLE_accelerating_code + + 9 wk + devicesEXAMPLE 86.8 K + Δ WoWEXAMPLE +28.5% + vs own medianEXAMPLE +108.7% - broker + ACCELERATING + 3 wk elevated
-
EXAMPLE: WoW only +X%. Tracks 16.0.1 rollout share; one-line hypothesis or owner pointer.
+
EXAMPLE: climbing for three consecutive weeks and up again this week. State what changed since it was last looked at — if nothing has, say so and name the owner.
-

See the 60-day trend section for the full ranked list.

+ +
+ Ongoing / known — elevated but plateaued (EXAMPLE 4 codes, EXAMPLE 2.1 M devices — no change in status this week) +
+
+
+
+ EXAMPLE_ongoing_code + + devicesEXAMPLE 1.79 M + Δ WoWEXAMPLE −3.2% + ONGOING · 3 wk +
+
+
+
+
+
🟢 Real wins this week
@@ -542,6 +629,8 @@

🚨 Things that need attention this week

EXAMPLE_recovered_code + + 9 wk devicesEXAMPLE 834 K Δ WoWEXAMPLE −86% Δ requestsEXAMPLE −78% @@ -559,59 +648,89 @@

🚨 Things that need attention this week

-

📈 60-Day Trend Analysis — literal last 60 days ending today; final bar = current partial week

+

📈 60-day cross-check — slow-burn detector, not a browsing list

- Methodology: Pulled all error codes from the ErrorStats view over the last 9 weeks. Dropped the partial start week (Mar 1). Kept all codes whose peak weekly device count ≥ 10 K. Bucketed each 8-week series by delta = (last − first) / first: - regression if delta > +15% and trajectory is monotonic-ish; ephemeral spike if peak ≥ 3× mean of surrounding weeks; improvement if delta < −15%; flat otherwise. Every code in the regression list gets a spike-attribution card below. + Why this section is short. The 60-day pass exists to catch what a week-over-week + comparison structurally cannot: a code that climbs 10 % every week for two months and never + once trips a WoW threshold. It found Failed to parse JWT going 7 → 3,461 devices + that way. That is its entire job. +

+ It is not a catalog to browse. Rendering all ~30 classified codes with a chart each + produced ~100 sparklines for what was, on inspection, a single incident — and it buried the four + series that actually changed. So: anything the 60-day pass flags that is also in the + attention section above is not repeated here, and anything flat is collapsed. +

+ Method: weekly device counts over the last 9 weeks; drop the partial start week; + keep codes with peak weekly devices ≥ 10 K; classify by delta = (last − first) / first + — regression > +15 % and monotonic-ish, ephemeral spike if peak ≥ 3× surrounding + mean, improvement < −15 %, else flat. Delta math uses complete weeks only.
+
-
⚠️ True 60-day regressions — 5 codes
- - - - - - - - - -
Error codeWk 1 devicesWk 8 devicesΔ over 8w60d sparklineTrajectory
no_tokens_found13.9 M23.7 M+70.6%monotonic up
unauthorized_client2.72 M3.37 M+23.6%monotonic up
Code:-631.8 K86.4 K+171.5%step-up at week 6
unknown_crypto_error59.3 K78.4 K+32.4%U-shaped, climbing
null_pointer_error48.5 K70.7 K+45.9%monotonic up
+
🐌 Slow burns — rising over 60 days, invisible week-over-week
+
+
+
+ EXAMPLE_slow_burn_code + + 9 wk + devicesEXAMPLE 3,461 + Δ WoWEXAMPLE +8.4% + Δ 60dEXAMPLE +49,000% + SLOW BURN +
+
EXAMPLE: never tripped a WoW threshold — grew ~8-12 % every week for eight weeks. This is the case the 60-day pass exists for. Name the owner and the suspected change.
+
+
-
-
⚡ Ephemeral 60-day spikes (peaked then recovered)
- - - - - - - -
Error codeBaselinePeakNow60d sparkline
timed_out_execution17.9 M142.9 M (week Apr 12)53.4 M
unknown_authority~1 K34.1 M (week Apr 12)1.45 M
429 (eSTS rate-limit)~10218 K (week Mar 22)2.5 K
-

Both unknown_authority (common #3082 ABBA deadlock fix) and timed_out_execution (broker #141 flight gating) are recovering. Recommendation: add Aria guardrail at >1M devices/week for unknown_authority to detect any future excursion early.

-
+ +
+ Full 60-day classification (EXAMPLE 29 codes — 5 regression, 3 ephemeral spike, 8 improvement, 13 flat) +
-
-
✅ True 60-day improvements
- - - - - - - - - -
Error codeWk 1Wk 8ΔSparkline
timed_out36.1 M5.1 M−85.9%
invalid_scope1.92 M0.36 M−81.3%
timed_out_thread_pool_saturated1.64 M0.62 M−62.1%
illegal_argument_exception0.21 M0.19 M−7.5% (peak −62%)
null_object, device_network_not_available, access_denied, ONLY_SUPPORTS_ACCOUNT_MANAGER_ERROR_CODE, invalid_keyall −17% to −78% over 8 wks (see appendix)
-

Note: the timed_out drop and timed_out_execution climb are partly the same event — broker #141 reclassifies legacy timed_out into the more specific timed_out_execution. The reclassification is net-neutral but the new code is louder; treat the timed_out "win" with caution.

-
+

Regressions — codes already covered above are marked (in attention).

+ + + + + + +
Error codeWk 1 devicesWk 8 devicesΔ over 8wTrajectory
EXAMPLE_no_tokens_found (in attention)13.9 M23.7 M+70.6%monotonic up
EXAMPLE_unknown_crypto_error59.3 K78.4 K+32.4%U-shaped, climbing
-
-
Flat on 60d (within ±10%)
-

io_error, no_account_found, invalid_grant, interaction_required, device_network_not_available_doze_mode, authorization_pending, expired_token, User cancelled, auth_cancelled_by_sdk, invalid_resource, invalid_request, device_registration_needed, Code:-1, Code:-2, Code:-8, operation_interrupted, ipc_return_null_cursor, device_needs_to_be_managed, Redirect url scheme not SSL protected, ipc_operation_not_supported_on_server_side, invalid_client, ipc_connection_error, unknown_error.

-
+

Ephemeral spikes — peaked then recovered. Not actionable unless the tail is still elevated.

+ + + + + + +
Error codeBaselinePeakNow
EXAMPLE_timed_out_execution17.9 M142.9 M (wk Apr 12)53.4 M
EXAMPLE_429 (eSTS rate-limit)~10218 K (wk Mar 22)2.5 K
+ +

Improvements

+ + + + + + +
Error codeWk 1Wk 8Δ
EXAMPLE_timed_out36.1 M5.1 M−85.9%
EXAMPLE_invalid_scope1.92 M0.36 M−81.3%
+

Caveat to carry: a code that drops because a change reclassified it into a new code is not a win. State the pairing when you see one (e.g. legacy timed_out falling as timed_out_execution rises).

+ +

Flat on 60d (within ±10%)

+

EXAMPLE_io_error, EXAMPLE_no_account_found, EXAMPLE_invalid_grant, EXAMPLE_interaction_required — list the rest inline, no table.

+ +
+

🔎 Spike Attribution — one card per regression

diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md b/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md index 9c0fd35b..7da4d553 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md @@ -129,12 +129,31 @@ Required spark/trend coverage in every report: | Where | Attribute | Length | Color (see palette below) | |---|---|---|---| | Every KPI tile in `.kpi-grid` (Top-line health) | `
` inside the tile | 8–9 weekly values | blue/green/dark-blue per metric semantic | -| **Every** row in the 60-day trend tables — true regressions, **ephemeral spikes**, and **true improvements** (all three callout tables) | `` in the trajectory cell | ~9 weekly values (incl. the current partial week as the final point) | red regression / amber spike / green improvement / grey flat | +| **Every visible row in the Section 2 attention list** — including the wins | `` right after `.item-name` | 9 weekly values | red worsening / orange accelerating / green improving / amber volatile | +| Broker 60-day section: **only the promoted slow burns** (rising on 60d, not already in Section 2 — often zero) | `` in the trajectory cell | ~9 weekly values (incl. the current partial week as the final point) | red regression / amber spike / green improvement | +| Authenticator 60-day per-scenario table — **all ~13 scenarios** (it is the scoreboard, not an overflow list) | `
` in the sparkline cell | 8–9 weekly values | red regression / amber spike / green improvement / grey flat | | Every row in the error-codes WoW table and error-types WoW table | `` in the 60d-trend column | 8 complete weekly values (no partial week — see `wow-table-sparkline-series.kql`) | same palette | +> **📌 The two WoW reference tables keep a sparkline on every row — deliberate exemption, decided +> explicitly. Do not strip them when reducing noise.** They are **lookup tables**: the reader arrives +> with a code in mind and scans for it, so the sparkline is glanceable context in a cell their eye is +> already on. The noise failure was the *60-day trend catalog* — a section read top-to-bottom whose +> rows ~93% duplicated these tables — not the tables themselves. + +> **⚠️ Charts go WITH the finding, not in a browsable section.** The Broker 60-day section used to +> chart every classified code — 38 charts, ~93% duplicating rows in the error tables below, while the +> attention section above had **zero**. The full 60-day classification now lives in a `
` fold +> **without** a chart column; only promoted slow burns are charted in the main flow. +> `validate-report.ps1` check 16 hard-fails a visible attention row missing its sparkline, and check 18 +> hard-fails a 60-day section that renders more than 6 (broker) / 16 (authapp) charts outside a fold. +> Charts *inside* a `
` fold are exempt from both — the reader opted in. **Checks 16/17/18 are +> scoped to Section 2 and the 60-day section only** — they intentionally say nothing about the WoW +> reference tables above. + **Past failure modes:** - v7 first pass: the body rebuild emitted *zero* `data-spark` / `data-trend` (validator now hard-fails this). -- v7 second pass: only the *true regressions* table got sparklines; the **ephemeral spikes** and **true improvements** tables were left text-only. All three tables in the 60-day trend section need the trajectory column with a sparkline — the validator's overall-coverage warn (≥15) catches this approximately, but the rule of thumb is: **if a row reports an 8-week delta, it gets a sparkline.** +- v7 second pass: only the *true regressions* table got sparklines; the ephemeral-spike and improvement rows were left text-only. Rule of thumb at the time: *"if a row reports an 8-week delta, it gets a sparkline."* +- v9: that rule was over-applied and became the opposite problem — **100 sparklines across the report, none of them next to a finding.** The rule is now **"if a row is a finding, it gets a sparkline"**; volume of charts is not coverage. ## Traffic-shape callout styling