From ad87f1bd57d0ffd511f64d7af129efdbd2f09a0c Mon Sep 17 00:00:00 2001 From: Shahzaib Date: Wed, 19 Aug 2026 18:14:40 -0700 Subject: [PATCH] OCE report: add novelty noise gate and redesign the Broker report around it The weekly Broker report had become a browsing exercise rather than a triage tool. The 60-day section rendered 38 charts, ~93% of which duplicated rows in the error tables below it, while the "needs attention this week" section -- the part an on-call engineer actually reads first -- carried 13 volume-ranked rows and zero charts. A flat-but-huge code led the list; the genuinely new ipc_* family sat at positions #6/#9/#10. Root cause: bucket-trends.js reports what MOVED, but nothing decided whether a movement was NEWS. Ranking by device count is not a proxy for novelty. This change adds classify-novelty.js, which labels every series against its own 7-week baseline (NEW / ACCELERATING / ONGOING / VOLATILE / RECOVERY / IMPROVING / STABLE) and emits an `attention` set = NEW + ACCELERATING. That set, plus at most 2 wins, is all that renders visibly with charts; everything still-elevated collapses into a fold with its weeksElevated count. The ACCELERATING/ONGOING split is the whole fix: only "still getting worse" earns a second look. The 60-day section becomes a slow-burn DETECTOR -- it charts only what it promotes (rows rising on 60d and absent from the attention section, typically 0-3, often zero) and folds the full classification with no chart column. Sections 6/7 keep a per-row sparkline as a deliberate exemption: they are lookup tables, not a browsing section. Also guards two measured false positives: a WoW % off an anomalous prior week (429 headlined at +397.8% while sitting 94.5% BELOW its own 60-day median), and a slow drift in block means labelling a flat, WoW-negative series ACCELERATING. validate-report.ps1 gains checks 13-18 to enforce all of this (row-body specificity, no flat top row, no suppressed-ratio chip, mandatory .item-spark, <=8 visible rows, <=6 charts in the 60-day section). Check 12 is left reserved. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../oncall-weekly-telemetry-report/SKILL.md | 265 +++++++++++++- .../assets/scripts/classify-novelty.js | 332 ++++++++++++++++++ .../assets/scripts/validate-report.ps1 | 210 ++++++++++- .../assets/templates/report-template.html | 247 +++++++++---- .../assets/templates/template-readme.md | 22 +- 5 files changed, 998 insertions(+), 78 deletions(-) create mode 100644 .github/skills/oncall-weekly-telemetry-report/assets/scripts/classify-novelty.js diff --git a/.github/skills/oncall-weekly-telemetry-report/SKILL.md b/.github/skills/oncall-weekly-telemetry-report/SKILL.md index 3c661051..37821349 100644 --- a/.github/skills/oncall-weekly-telemetry-report/SKILL.md +++ b/.github/skills/oncall-weekly-telemetry-report/SKILL.md @@ -22,6 +22,7 @@ Reusable helpers in [`assets/`](assets/): | [`queries/`](assets/queries/) | Canonical KQL templates, one file per query — see [`queries/README.md`](assets/queries/README.md). Highlights: [`attr-union-by-dim.kql`](assets/queries/attr-union-by-dim.kql) (NEW — all 7 dims in one round-trip), [`error-message-and-location.kql`](assets/queries/error-message-and-location.kql) (now accepts BOTH `` and `` in one call) | | [`templates/`](assets/templates/) | Copy-paste HTML snippets (`spike-card.html`, `traffic-attr-card.html`, `sparkline-footer.html`) | | [`bucket-trends.js`](assets/scripts/bucket-trends.js) | Bucket all error codes into 60-day regression / spike / improvement / flat. Run with `--metric=devs` AND `--metric=reqs`. Pass `--end=YYYY-MM-DD` (the Sunday that OPENS the current in-progress week, exclusive — i.e. `startofweek(today)`, printed by bootstrap as "Trend delta cutoff") to exclude the partial week from the delta math, plus **`--include-partial-end`** to still chart it as the final bar. **`--summary` suppresses the verbose header; `--json=` emits a structured sidecar for programmatic consumption.** | +| [`classify-novelty.js`](assets/scripts/classify-novelty.js) | Reads a `bucket-trends.js --json=` sidecar and labels each key **NEW / ACCELERATING / ONGOING / VOLATILE / RECOVERY / IMPROVING / STABLE** against its own 7-week baseline, plus family clustering. **Mandatory** — it is what stops the attention section from being a volume-ranked list where a flat-but-huge code outranks a real step change, and it is also the **noise gate**: its `attention` set (`NEW` + `ACCELERATING`), plus at most 2 wins, is all that renders visibly with charts — everything else collapses into a fold. | | [`agg.js`](assets/scripts/agg.js) | Per-error per-dim top-N rollup with WoW deltas. Workhorse for filling spike-attribution dim blocks. | | [`summarize-attribution.js`](assets/scripts/summarize-attribution.js) | Roll up 7-dim attribution slices for spike-attribution cards. Supports BOTH `--union ` (preferred for 2-week WoW; pairs with `attr-union-by-dim.kql`) AND legacy `--label= file.json` per-dim mode. **Auto-detects the array-form schema produced by `assets/scripts/run-kql.ps1` — no schema-transformer step needed.** | | [`find-suspect-prs.ps1`](assets/scripts/find-suspect-prs.ps1) | Parallel `git log -S` + `--grep` across broker/ + common/ for a class/method symbol, with PR numbers + URLs. Run *only after* the Originator pre-check has identified a specific throw-site class — the unscoped 4-week PR window is small enough (<30 PRs) to scan with plain `git log` first. | @@ -69,22 +70,127 @@ Reusable helpers in [`assets/`](assets/): 1. **Top-line health KPIs** — total requests, total devices, silent-auth reliability %, interactive reliability %, p95 latency on the hot spans. WoW delta on each. Inline SVG sparklines. 2. **Things that need attention this week** — callouts: - **Denominator caveat** — explain any large total-spans device-count shift caused by span-emission changes (e.g. `goAsync()` refactors). Always state which denominator the report uses (auth-only: `SilentAuthStats` ∪ `InteractiveAuthStats`). - - **🔴 WoW regressions (last 7 days)** — *one* callout listing every code/type that moved sharply WoW, **sorted by current-week device count descending**. Built from the union of (a) the standard WoW table and (b) [`assets/queries/wow-movers.kql`](assets/queries/wow-movers.kql) so small-but-recent spikes appear in the same list as the high-volume ones. Each row uses the `.item` flat-row pattern (see `assets/templates/template-readme.md` § "Section 2 callouts"): name + inline metric chips + tags pushed right + one-line body + optional foot with `Attribution card →` link. **Section 2 rows are at-a-glance only** — do not duplicate the dim slicing / PR analysis / detailed verdict here; that belongs in the Section 4 spike-attribution card. Each row carries tags: `NEW` (first appeared this week or last), `60d↑` (also rising on 60d), and an originator chip (`broker` / `eSTS` / `Android` / `env`). Reader's eye prioritizes naturally by row order and tag combination — broker-tagged rows at the top demand the most attention. - - **Slow-burn 60-day regressions** — codes/types climbing on the 60d window that are flat WoW. Anything that *also* moved WoW belongs in the red callout above (with `60d↑`), not here. Link to the 60-Day Trend section. - - **Real wins this week**, with PR links. + - **🔴 Regressions — grouped by NOVELTY, never by volume.** Built from [`classify-novelty.js`](assets/scripts/classify-novelty.js) (Step 3e), unioned with [`assets/queries/wow-movers.kql`](assets/queries/wow-movers.kql) so small-but-recent spikes land in the same grouping. + + > **The section has a hard budget: ≤ 8 visible rows, counting the wins.** The classifier's + > `attention` set (`NEW` + `ACCELERATING`) is typically 3–6 series out of 40–50 — that is the + > whole point — and you may add **at most 2** "Real wins" rows on top. Everything else goes into + > a collapsed fold. A previous report shipped **13 visible rows and zero charts** while the + > 60-day section below it carried **38 charts**; a reader could not tell which of the 13 was this + > week's story. `validate-report.ps1` check 17 warns above 8 rows and it counts **every** visible + > `.item` row in Section 2, wins included — so budget accordingly. + > + > **This budget caps VISIBILITY, never COVERAGE.** Step 5's rule that every regressed code/type + > must get an attribution card still holds in full — the surplus cards go into a collapsed fold, + > they are never dropped. See the boxed note under Step 5 for the exact resolution. If you ever + > find yourself deleting a mandated card to hit 8 rows, you have misread both rules. + + Emit these sub-groups **in this order**, omitting any that are empty: + + 1. **🆕 New this week** — label `NEW`: a genuinely boring baseline (cv < 0.25) that took a clean + step up. **These lead the section.** Typically 0–3 items. If there are none, say *"nothing new + this week"* explicitly — do **not** promote an `ACCELERATING` or `ONGOING` item to fill the slot. + 2. **🟠 Getting worse** — label `ACCELERATING`: already elevated, but **still climbing** (rising + over the window, ≥ 10% above its own median, and not falling WoW). This is the "known issue is + deteriorating" bucket and it is the *only* multi-week category that stays visible. + + > **When the classifier and the headline WoW disagree, keep the row here and show both numbers.** + > The classifier's "not falling" gate runs on **complete Sun–Sat calendar weeks**; the report's + > headline `Δ WoW` runs on the **rolling 7-day** window. These are different bases and they + > legitimately disagree — a code can be `ACCELERATING` on calendar weeks while showing a small + > rolling-window decline. That is *not* a reason to demote it, rename the group, or hedge the + > heading. Keep the group heading exactly **"Getting worse"**, and resolve it *in the row body*: + > *"Up 18% over the last three complete weeks; the rolling 7-day window shows −4% as the ramp + > flattens. Still ~30% above its own 60-day median — watch, don't close."* The sparkline settles + > it visually, which is why the row has one. Do **not** invent a "needs verification" group. + 3. **🔵 Ongoing / known** — label `ONGOING`: elevated but flat. **These go inside a collapsed + `
`**, summarised by one line ("N codes remain elevated, none accelerating"). + They are still in the report — a reader can open the fold — but they no longer compete with the + finding. Give each the number of weeks it has been elevated (`weeksElevated` from the classifier) + so a reader can see it is old news at a glance. + 4. **🔁 Volatile** — label `VOLATILE` (`suppressRatio: true`): high-variance series where a WoW + percentage is an artifact of a depressed baseline. **Delete the `Δ WoW` chip from the row head + and put a `vs 60d median` chip in its place** — a caveat in the body does not undo a `+401.8%` + sitting in the chip row, because the chip is what the eye reads first: + ```html + + Delta WoW+401.8% + + vs 60d median-94.5% + ``` + `validate-report.ps1` check 15 **hard-fails** any `VOLATILE`/`RECOVERY` row that still carries a WoW chip ≥ 25%. + 5. **↩️ Recovery** — label `RECOVERY`: returning to its normal band after a suppressed week. Explicitly *not* a regression. + + **⚠️ Every visible row carries its own 9-week sparkline.** The shape is what separates a step + change from ordinary variance, and it must sit *next to the claim* — not in a separate browsable + section further down. Use the `.item-spark` pattern from the template: + ```html + ipc_return_null_cursor + + 9 wk + ``` + Colour by direction: `#cf222e` worsening, `#1a7f37` improving, `#9a6700` volatile. The series is + the same `comparable` array `classify-novelty.js` already read from the trend sidecar — you do not + run another query for it. `validate-report.ps1` check 16 **hard-fails** any visible attention row + without one. Rows inside the collapsed fold are exempt. + + **Novelty chips.** Tag each row with its label so the grouping survives skimming: + `NEW`, `tag-accel` for `ACCELERATING`, `tag-ongoing` for + `ONGOING`, plus `elevated Nw` where the classifier reports `weeksElevated > 1`. + + **Families outrank individuals.** If the classifier emits a `families` entry, render it as ONE row + naming the family and its members — related codes moving together are one root cause, not N + findings. Within each sub-group, order by the classifier's `ORDER`/novelty ranking, **not** by + current-week device volume. + + **Quiet weeks are a valid, good outcome.** If the classifier reports `quietWeek: true` (empty + attention set), lead the section with the quiet-week banner from the template and keep the fold + closed. Do **not** manufacture a headline finding — resist the pull to promote the largest flat + code. A short report that says "nothing new" is more trustworthy than a long one that pads. + + Each row uses the `.item` flat-row pattern (see `assets/templates/template-readme.md` § "Section 2 callouts"): name + sparkline + inline metric chips + tags pushed right + one-line body + optional foot with `Attribution card →` link. **Section 2 rows are at-a-glance only** — no dim slicing or PR analysis here; that belongs in the Section 4 card. Tags: `60d↑` (also rising on 60d) plus an originator chip (`broker` / `eSTS` / `Android` / `env`). + + > **Every row's one-line body must say something specific to that row** — what changed, from what to what, and why it is or isn't alarming. A sentence that would read identically on any other row (*"movement needs owner triage; deep dive below"*) is worthless and the validator will fail the report for it. If you have nothing specific to say, the row does not belong in Section 2. + - **Real wins this week**, with PR links. These carry sparklines too — a recovery is a shape claim. + **Cap at 2 rows, and they count against Section 2's ≤ 8 visible-row budget.** A win is worth + showing; a list of wins is padding. - **Traffic shape** — flat / surge / collapse summary. -3. **📈 60-Day Trend Analysis** — built from the `ErrorStatsMetrics` materialized view over the **literal last 60 days ending today** (final bar = current in-progress week). **Run the bucketing pipeline FOUR times — the cross-product of `{error_code, error_type} × {devices, requests}`** — and union the regression sets. An entry (code OR type) is flagged if it regresses on either metric. Deltas are computed on complete weeks only; the partial current week is charted but excluded from classification. +3. **📈 60-day cross-check** — a **slow-burn detector, not a browsing list**. Built from the `ErrorStatsMetrics` materialized view over the **literal last 60 days ending today** (final bar = current in-progress week). **Run the bucketing pipeline FOUR times — the cross-product of `{error_code, error_type} × {devices, requests}`** — and union the regression sets. An entry (code OR type) is flagged if it regresses on either metric. Deltas are computed on complete weeks only; the partial current week is charted but excluded from classification. - **% of devices** affected (`devicesHit / authActiveDevices`) — catches errors hitting more users. - **% of requests** affected (`errRequests / authTotalRequests`) — catches per-device retry storms (fewer users, more traffic per user). The previous report would have missed `kdfv2_key_derivation_error` (262 → 5,374 requests on ~57 devices) without this dim. - Categories: True 60d regression / Ephemeral 60d spike (peak-then-recover) / True 60d improvement / Flat. Every rising entry — whether `error_code` or `error_type` — gets the same Spike Attribution + Code Attribution treatment (Step 4 / Step 5). + > **⚠️ This section exists to catch what a 7-day window structurally cannot see: something that has + > crept up ~5%/week for eight weeks and never triggers a WoW alarm.** That is its *only* job. + > + > **Chart only the codes it promotes** — series flagged as rising on 60d that are **not already in + > Section 2**. In a typical week that is **0–3 rows**, and an empty result is the normal, healthy + > outcome; say "no slow burns this week" and move on. Everything else — the full classification of + > all 40–50 series — goes into a collapsed `
` **with no chart column at all**. + > + > Rationale: a previous report rendered 38 charts here, ~93% of which duplicated rows already in the + > error tables below, while the attention section above had none. Reviewing 38 long-elevated series + > every week is exactly the noise that trains an on-call engineer to skim. `validate-report.ps1` + > check 18 **hard-fails** if this section renders more than 6 charts outside a fold. + + Categories: True 60d regression / Ephemeral 60d spike (peak-then-recover) / True 60d improvement / Flat. Every **promoted** rising entry — whether `error_code` or `error_type` — gets the same Spike Attribution + Code Attribution treatment (Step 4 / Step 5); entries already covered in Section 2 are not re-analysed here, just cross-referenced. Always apply `MergeUiRequiredExceptions(error_type)` before bucketing on type; otherwise the 6+ string variants of `UiRequiredException` will each be tracked separately and skew the buckets. 4. **🔎 Spike Attribution** — one card per WoW regression AND per 60-day regression, **for both `error_code` and `error_type` regressions**. Each card slices on **all 7 dimensions** (broker version, span, active broker pkg, calling app, account type AAD/MSA, shared-device mode, client SKU). Each card ends with a **deep Code Attribution block** (see Step 4 for the required fields) and a Traffic Attribution verdict. 5. **🚚 Traffic Attribution** — top-level section listing every error whose spike is fully or partly explained by traffic volume from a specific calling app, rather than a code regression. If none qualify this week, render the section with an explicit "None this week" note. 6. **Error codes — WoW with stable denominator** — full table with `Δ requests %` and `Δ devices %` columns and the 60d sparkline. 7. **Error types — WoW with stable denominator** — full table, **same columns and rigor as the error-codes table** (`Δ requests %`, `Δ devices %`, 60d sparkline, status pill). Any regressing type also gets a spike-attribution card in Section 4. For composite types (e.g. `ClientException` is the umbrella for many sub-codes), include a **decomposition card** that breaks the WoW Δ down into the top 3 contributing sub-codes — so a `ClientException` −5 pp drop is explicitly attributed to e.g. `−8.5 pp timed_out_execution` + `−3.4 pp unknown_authority` + `−0.15 pp illegal_argument_exception`. + +> **📌 Sections 6 and 7 keep a sparkline on EVERY row — this is a deliberate exemption from the +> "charts follow findings" rule, decided explicitly. Do not strip them as part of noise reduction.** +> These are **lookup tables**, not a browsing section: the reader arrives with a code in mind, scans +> the `Error code` column for it, and the 60-day sparkline is glanceable context in a cell their eye +> is already on. It costs no extra attention. The noise problem the redesign fixed was the *60-day +> trend catalog* — a section you had to read top-to-bottom, ~93% of whose rows duplicated these very +> tables. Checks 16/17/18 deliberately scope to Section 2 and the 60-day section only; the +> `$totalCharts` count that these ~62 charts dominate is now a floor-only guard ("the charts didn't +> vanish"), never a ceiling. 8. **📊 Traffic analysis** — total requests/devices (WoW + 60d), top calling apps, top spans, **requests-per-device ratio** per error and overall (a rising ratio = retry storm; a falling ratio = caching gain), sampling-rate change indicator. 9. **Latency** — p50/p95/p99 by hot span. 10. **Broker version adoption** — week-over-week version share. @@ -248,10 +354,107 @@ To catch these, **always** run [`assets/queries/wow-movers.kql`](assets/queries/ // floor: cDev>=500 OR cReq>=5000 move: |Δd|>=25% OR |Δr|>=50% OR new-this-window ``` -Run it **twice — once for `error_code`, once for `error_type`**. **Merge its output rows into the same 🔴 WoW regressions callout as the standard WoW table** (sorted by current-week device count descending). Tag rows that came in via this pass with `NEW` if they were absent or near-zero in the prior week. Do *not* render this as a separate "emerging" callout — the size split is implementation detail; readers prioritize naturally by absolute device count + originator chip. +Run it **twice — once for `error_code`, once for `error_type`**. **Merge its output rows into the same regression callout as the standard WoW table**, then let Step 3e's novelty classification decide their grouping and order. The size split is implementation detail; what a reader needs first is *"is this new?"*, not *"is this big?"*. Do **not** sort the merged list by device count — that is exactly how a flat-but-huge code ends up above a real step change. For each WoW mover (regardless of size), you still owe the full Code Attribution treatment (Step 4). The dim-slicing pass (Step 5) is allowed to be deferred for sub-1K-device spikes if the throw-site + dominant message already pin the originator unambiguously — but say so explicitly in the card ("dims not yet sliced — file the bug first; pull dims if it persists"). +### Step 3e — Classify novelty (what is actually NEW vs already-known) + +**This step is mandatory and it is what makes Section 2 readable.** `bucket-trends.js` tells you *what moved*; it cannot tell you *whether the movement is news*. Without this pass the attention section degenerates into a volume-ranked list where a flat-but-huge code leads and the real step change sits at position #9. + +Run it over each sidecar `bucket-trends.js` wrote (`--json=`): + +```powershell +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\classify-novelty.js --summary --json= +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\classify-novelty.js --summary --json= +``` + +Each key is classified against **its own history**, using complete weeks only (first match wins): + +| Label | Rule | What it means for the report | +|---|---|---| +| `VOLATILE` | `cv > 0.60` | Series swings wildly. **Suppress the ratio** — a WoW % here is meaningless. | +| `RECOVERY` | prior week `< median × 0.5` **and** now back near median | Bounce-back off a suppressed week, not a regression. **Suppress the ratio.** | +| `NEW` | `ratio > 1.15` **and** `cv < 0.25` | Boring baseline, clean step up. **This is the news.** Visible + charted. | +| `ACCELERATING` | rising over the window (`climb > 1.15`) **and** `ratio > 1.10` **and** `recentRatio > 1.10` **and** not falling WoW | Known issue that is **still deteriorating**. Visible + charted. | +| `ONGOING` | rising over the window, but level or easing now | Elevated and flat. **Collapse into the fold** with its `weeksElevated` count. | +| `IMPROVING` | `ratio < 0.8` | | +| `STABLE` | otherwise | | + +**⚠️ The `ACCELERATING` / `ONGOING` split is the whole noise fix.** Both are multi-week elevated +series, and lumping them together is what produced a 13-row attention list. `ACCELERATING` answers +"is this getting *worse*?" — the only reason a known issue deserves the on-call engineer's eye a +second time. Everything else that is merely still-elevated goes in the fold. + +**The classifier decides what gets a chart.** `attention = NEW ∪ ACCELERATING` — that set, and only +that set, is rendered visibly with sparklines. The sidecar exposes it directly: + +```jsonc +{ "attention": ["ipc_return_null_cursor", "access_denied"], // render these, with charts + "attentionLabels": { "ipc_return_null_cursor": "NEW", "access_denied": "ACCELERATING" }, + "quietWeek": false, // true => nothing to headline + "counts": { "NEW": 3, "ACCELERATING": 1, "ONGOING": 9, "STABLE": 35, "VOLATILE": 3, "IMPROVING": 2 } } +``` + +On the 2026-07-30 fixture that is **5 attention rows out of 53 series**. If your attention section is +much longer than the `attention` array, you promoted rows the classifier did not. + +**`weeksElevated` is derived, never persisted.** It counts consecutive recent weeks above the +*early-window baseline* (`median` of the first third), so it is identical on any machine and needs no +state file. Its known limit: with 7–9 weeks of history you cannot distinguish "elevated for 7 weeks" +from "normal at a high level" — the classifier sets `sustainedFullWindow: true` for those, and the +correct phrasing is *"elevated for the entire visible window"*, not a hard week count. + +**Two guards that exist because they were violated in real runs:** +- **`ratio > 1.10` on `ACCELERATING`** — `IntuneAppProtectionPolicyRequiredException` (cv 0.08, flat, + **down 3.7% WoW**, only 4.9% above its own median) was labelled `ACCELERATING` by a slow drift in + block means and led the whole types list. A code within 10% of its own median is not this week's + story regardless of slope. +- **`cur >= prev * 0.95`** — a series that is *falling* this week cannot be "getting worse", even if + the multi-week trend is up. + +**Why `cv < 0.25` gates `NEW`:** a series must have been genuinely boring before a jump counts as news. Without that guard a jittery code that happens to be up this week gets promoted over a real step change. + +**⚠️ The trap this exists to kill — a big WoW % off an anomalous baseline is not a regression.** Real 2026-07-30 data: + +``` +429 300,664 299,965 892,839 974,980 11,512 32,530 2,724 → 16,531 cv=1.07 +temporarily_unavailable 30,257 29,962 37,263 6,168 41,971 141 71 → 36,153 cv=0.80 +``` + +`429` was reported as **+397.8% WoW** — but it is **94.5% *below* its own 60-day median**; the ratio is measured off a 2,724 floor after a collapse from ~975K. `temporarily_unavailable` was reported as **+400.7%** — it merely returned to its normal ~36K band after two suppressed weeks. Both were headlined. Neither is a regression. **A WoW percentage is meaningless whenever the *prior* week was itself anomalous.** + +Meanwhile the week's actual story classified as `NEW` and was buried at report positions #6/#9/#10: + +``` +ipc_return_null_cursor 43,093 43,950 43,552 44,571 43,759 42,117 41,473 → 52,129 cv=0.02 +ipc_operation_not_supported_on_server 19,503 19,957 20,074 21,457 21,466 20,849 20,575 → 24,050 cv=0.03 +ipc_connection_error 8,317 8,685 8,517 8,634 8,104 8,410 8,593 → 10,133 cv=0.02 +IPC FAMILY TOTAL 70,641 → 86,312 (+22.2%) +``` + +Three codes, each flat for seven straight weeks, all stepping up in the *same* week — one root cause in the IPC layer, reported as **one** finding. (`BrokerCommunicationException`, `NEW` on the type axis at +21.8%, is the same incident seen through the type dimension — say so rather than filing it twice.) + +**Families.** The classifier clusters keys sharing a prefix before `_` when ≥2 members share the same label. Report a family as ONE row. Error *types* are CamelCase and produce no families under `_` — that is correct, not a bug. + +**⚠️ Two different WoW bases exist — do not conflate them.** The report headline ΔWoW is a **rolling 7-day** window (`[CUR_START, CUR_END)` vs the 7 days before). The classifier's `WoW` is **calendar Sun–Sat weeks**. They legitimately disagree — `authorization_pending` read **+3.5%** rolling and **−37.1%** weekly on the same data. Use novelty as *history and context* ("flat for seven weeks, first step this week"), **never** as a competing delta number, or the report will appear to contradict its own tables. + +> **The division of labour, stated plainly so you do not have to derive it:** +> +> | Use the **rolling 7-day** numbers for… | Use the **calendar-week** classifier for… | +> |---|---| +> | Every KPI tile, table cell, and Δ% chip | Which rows are promoted (`attention` set) | +> | Any number a reader can see | Which label a row carries (NEW / ACCELERATING / …) | +> | The sentence "X rose N% this week" | The sentence "…and it has been climbing for six weeks" | +> +> **Rule: every *number* in the report comes from the rolling window; the classifier contributes +> *selection and narrative*, never a figure.** The one place the two meet is a row that is +> `ACCELERATING` on calendar weeks while the rolling delta is flat or negative — keep it in +> "Getting worse", keep the heading verbatim, and resolve it in the row body by stating both +> numbers and letting the sparkline settle it. Do not invent a hedged sub-group for these. + +--- + ### Step 4 — Code attribution (deep PR correlation) > ⚠️ **HARD RULE — Originator pre-check.** Before claiming `Originator: Broker` on any card, you MUST run [`assets/queries/error-message-and-location.kql`](assets/queries/error-message-and-location.kql) for that error code (or type) and read **(a) the throw-site stack and (b) the top 3 `error_message` strings**. Most broker error codes flow through `common/ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult, clientExceptionFromException}` — which intentionally bridge eSTS responses into broker exceptions. **If the throw site is in any of those three methods AND the error_message starts with `AADSTS`, the originator is eSTS, not broker.** See the AADSTS reference table in [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md). Cards that skip this step must be marked low-confidence, not high. @@ -318,6 +521,23 @@ For errors with no broker code in the stack (Android system errors like `Code:-1 **Coverage rule: every `error_code` AND every `error_type` that lands in either the WoW regression list OR the 60-day regression list MUST get a spike-attribution card.** No silent skips. +> **⚠️ Coverage and the ≤ 8 visible-row budget are NOT in conflict — they govern different things.** +> This is the most-reported ambiguity in the playbook, so read it carefully: +> - The **≤ 8 budget (§2) limits what is VISIBLE at the top level.** It is about what the on-call +> engineer is asked to read first. +> - The **coverage rule here limits what may be OMITTED.** It is about what must exist somewhere in +> the document, so a regression can never silently vanish. +> +> **Resolution: cards beyond the budget go into a collapsed fold, they do not get dropped.** Render +> the `attention` set (plus ≤ 2 wins) as visible cards, and put every remaining mandated card in a +> `
` fold titled *"Full attribution coverage (N more codes/types)"*. Coverage is satisfied +> by the card **existing and being reachable**, not by it being expanded on load. A run with 12 +> mandated cards and 7 visible rows is correct and expected — that is the design working, not a +> budget violation. +> +> Never resolve this the other way: do **not** expand Section 2 past 8 rows to fit the cards, and do +> **not** skip a mandated card to protect the budget. + **`ErrorStatsMetrics` already carries `account_type` and `is_shared_device`** (use the `MergeAccountType` / `MergeIsSharedDevice` helpers to normalize) — so you do **not** need a fallback to raw `android_spans` for these dims. Earlier versions of this skill claimed otherwise; that was wrong. The only dim that requires `android_spans` is `DeviceInfo_OsVersion` (OEM/version slicing). Slice on **all 7 dimensions** for each spike. **Preferred for 2-week WoW attribution: one union query that covers all 7 dims for all regressions in a single round-trip** — see [`assets/queries/attr-union-by-dim.kql`](assets/queries/attr-union-by-dim.kql). Typical payload for 8 codes × 2 weeks × 7 dims is ~800 KB, well under the MCP limit. Pipe the result into `summarize-attribution.js --union ` (which prints per-dim top-N share + Δ devices + Δ requests for every code). Fall back to the per-dim form ([`attr-codes-by-dim.kql`](assets/queries/attr-codes-by-dim.kql)) only when (a) you need a wider time window, or (b) the union response exceeds payload size. @@ -487,6 +707,15 @@ The validator hard-fails on: 7. **Code-attribution depth** — each `.attr-card`'s "Code attribution" block must contain an `Originator` row (proxy for the full 8-field structure: Originator / Top throw site / Wrapper / Caller hot-spots / Underlying cause / Top error_messages / Likely PRs / Next step). Catches the v7-third-pass regression where cards shipped with a `pr-list`-only stub. 8. **Attribution-card layout guards (v8)** — the CSS must define `.attr-card { margin-bottom: 16px }` AND `.dim-row` overflow rules (`text-overflow: ellipsis` + `min-width: 0`). Catches the "cards touching" and "text bleeding out of dim boxes" regressions from a stale `` block. 9. **Fabricated-sparkline heuristic (v8)** — warns when a `data-trend` array's peak value is < 100 (almost certainly hand-rolled rather than sourced from real data). See [`assets/queries/wow-table-sparkline-series.kql`](assets/queries/wow-table-sparkline-series.kql) for the canonical KQL that pulls real 8-week series for every code in the WoW tables. Its `` / `` tokens are the last **8 complete** Sun-Sat weeks (`` = `startofweek(today)`, exclusive) — deliberately distinct from the trend-chart's `` / `` (literal last 60 days ending today). Per-row sparklines stay on complete weeks so a partial final point doesn't create a misleading dip in every WoW row. +10. **Rolling-window header integrity (check 11)** — the meta-line "Last 7 days" dates must agree with the filename's end-date, so a stale stub can never be published as fresh. +11. **Noise-gate checks 13–18** — these enforce Step 3e's classification end-to-end and are the reason + Section 2 stays readable: + - **13** — Section 2 boilerplate uniformity: each attention row needs its own specific one-line body. + - **14** — the top attention row must not be flat; Section 2 leads with what is `NEW`, never with the highest-volume row. + - **15** — `VOLATILE` / `RECOVERY` rows must not headline a `Δ WoW` percentage (`suppressRatio`). + - **16** — every visible attention row carries an inline 9-week `.item-spark`. + - **17** — Section 2 visible-row budget: ≤ 8 rows including wins. + - **18** — the 60-day section is a detector, not a catalog: at most 6 charts outside a collapsed fold. Then: - **Run the visual smoke test (recommended)** — catches rendered-layout bugs that pure HTML/CSS validation can't see: @@ -517,7 +746,29 @@ Then: - **Never carry a numeric telemetry value forward between runs.** Every KPI, table cell, delta %, device/request count, sparkline point, and verdict number must be re-pulled from Kusto for *this* run — never copied from a previous report, from a checkpoint/summary, from notes, or from memory. Telemetry shifts between runs and stale numbers read as fabricated. Near-miss precedent: a `no_tokens_found` count was about to be carried as ~23.7M when the actual current-window value was ~4.86M — a ~5× error that only the re-pull caught. If a number isn't backed by a query result file in this run's `_data//`, it does not go in the report. - **Never hardcode the "Generated" date.** It is the *run* date in **UTC**, auto-stamped by `bootstrap-report.ps1` (which uses `(Get-Date).ToUniversalTime()`). If you rebuild the body programmatically, derive it live with a **UTC-date** formatter (`new Date().toISOString().slice(0,10)` in Node, `[datetime]::UtcNow.ToString('yyyy-MM-dd')` in PowerShell) — never paste a literal, and stay on UTC so the assembler can never stamp a different day than `bootstrap-report.ps1`. The v8 "Generated 2026-06-15 on a 2026-06-18 file" bug came from a hardcoded string in the assembler. (Reporting-week / baseline / 60d window dates are author-set and verified against the user's intended Sunday bucket — see template-readme "Date fields".) - **Originator pre-check is mandatory.** A card cannot claim `Originator: Broker` without first running [`assets/queries/error-message-and-location.kql`](assets/queries/error-message-and-location.kql) and reading the throw site + top 3 `error_message` strings. If the throw site is in `common/ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult}` AND the message starts with `AADSTS`, the originator is **eSTS, not broker** — see the AADSTS reference in [`assets/docs/kusto-cheatsheet.md`](assets/docs/kusto-cheatsheet.md). -- **WoW-movers pass is mandatory.** The 60d bucketer's `--peak-floor` silently drops sub-10K-device codes, so [`assets/queries/wow-movers.kql`](assets/queries/wow-movers.kql) MUST be run as a separate pass for both `error_code` and `error_type` (per Step 3d). Its output is **merged into the single 🔴 WoW regressions callout**, sorted by current-week device count descending, with rows tagged `NEW` / `60d↑` / originator chip. Do not render a separate "emerging" callout. Skipping the pass is how the Apr 26 `Failed to parse JWT` spike (7 → 3,461 devs over 7 weeks) hid for two reports running. +- **WoW-movers pass is mandatory.** The 60d bucketer's `--peak-floor` silently drops sub-10K-device codes, so [`assets/queries/wow-movers.kql`](assets/queries/wow-movers.kql) MUST be run as a separate pass for both `error_code` and `error_type` (per Step 3d). Its output is **merged into the single regression callout** and then grouped by Step 3e's novelty labels. Do not render a separate "emerging" callout. Skipping the pass is how the Apr 26 `Failed to parse JWT` spike (7 → 3,461 devs over 7 weeks) hid for two reports running. +- **Novelty classification is mandatory, and Section 2 is ordered by it — never by volume.** Run [`classify-novelty.js`](assets/scripts/classify-novelty.js) (Step 3e) and lead with `NEW`. Ranking the attention list by device count is a known, reported defect: it put `IntuneAppProtectionPolicyRequiredException` (ΔWoW **+0.1%**, classifier says `ONGOING` and *falling*) at #1 while the genuinely new `ipc_*` family sat at #6/#9/#10. If the `NEW` bucket is empty, write "nothing new this week" — do not backfill it with `ONGOING` items. +- **Section 2's visible rows are the classifier's `attention` set plus at most 2 wins — nothing else.** + `NEW` + `ACCELERATING` visible with sparklines; `ONGOING` inside a collapsed `
`. + Budget: **≤ 8 visible rows total, wins included** (validator check 17 warns above it and counts wins). + The failure mode this replaces is measured, not hypothetical: 13 visible rows, 0 charts, and the + 60-day section below carrying 38. +- **Every visible attention row carries a 9-week `.item-spark`.** Validator check 16 hard-fails + otherwise. The series comes from the trend sidecar you already loaded — no extra query. Charts belong + beside the claim they support; a separate browsable chart section is the noise, not the signal. +- **The 60-day section is a detector, not a catalog.** Chart only the slow burns it *promotes* (rising + on 60d and absent from Section 2) — typically 0–3, often zero. The full classification goes in a fold + with no charts. Validator check 18 hard-fails above 6 visible charts there. +- **A quiet week is a valid outcome — publish it as one.** If `quietWeek: true`, say so plainly and + keep the report short. Padding the attention list with the biggest flat code to look thorough is the + exact behaviour that trains readers to skim. +- **A `VOLATILE`/`RECOVERY` row must not carry a `Δ WoW` chip at all — swap it for `vs 60d median`.** These carry `suppressRatio: true` because their WoW % is an artifact of a depressed prior week, not a regression. Tagging the row `VOLATILE` and caveating in the body is **not sufficient**: a naive run shipped `429` tagged `VOLATILE` with the body reading *"large percentage move but classified volatile"* — and still rendered `+401.8%` in `metric up` styling, which is the first thing a reader sees. `429` at "+401.8%" while sitting 94.5% *below* its own 60-day median is the canonical failure. `validate-report.ps1` check 15 hard-fails this. +- **A family is one finding, not N — and that includes the type axis.** When related codes move together (classifier `families`), emit **one** row whose `item-name` is the family and whose body names the members. Reconcile across axes too: if a `NEW` type is the umbrella for a `NEW` code family, that is still **one** row. A naive run emitted four rows — `BrokerCommunicationException`, `ipc_return_null_cursor`, `ipc_operation_not_supported_on_server_side`, `ipc_connection_error` — for a single IPC incident, which re-creates the wall-of-codes problem this section exists to fix. Correct shape: + ```html + ipc_* / BrokerCommunicationExceptionfamily + ``` + with the body reading *"Three IPC codes stepped up together off a flat 7-week baseline (…null_cursor 41.5K→52.1K, …not_supported 20.6K→24.1K, …connection_error 8.6K→10.1K; family +22.2%). `BrokerCommunicationException` is the same incident seen on the type axis."* One row, one owner, one attribution card. +- **No boilerplate in Section 2.** Every row's one-line body must be specific to that row — what changed, from what to what, why it is or isn't alarming. Reusing one generic sentence across rows (*"Current-window movement needs owner triage; deep dive below has originator and dimensions."*) makes the section unreadable and `validate-report.ps1` fails the report for it. - **Section 2 callouts are at-a-glance, Section 4 is the deep dive.** WoW / Slow-burn / Wins items in Section 2 use the `.item` flat-row pattern (no nested cards, no per-item left bars — the parent `.callout` border is the only severity affordance). Each row is a single line of metric chips + a one-line body + an `Attribution card →` link to the corresponding `.attr-card` in Section 4. Do NOT duplicate the dim slicing, PR analysis, or detailed verdict between the two sections — Section 4 is where that lives. See [`assets/templates/template-readme.md`](assets/templates/template-readme.md) for the CSS class reference and the example `.item` markup. - **Never use bash/PowerShell regex to bulk-edit balanced HTML.** This skill has burned twice on regex strip scripts that ate matched-pair `` closes, producing inception-style nested-callout bugs that take a depth-tracking script to find. If you need a structural change to the HTML, make a targeted, single-occurrence string replacement (with explicit before/after context) or rewrite the affected block end-to-end. Never run a `-replace` across the whole file expecting it to leave balance intact. - **Denominator caveat must cite evidence, not hand-wave.** If you flag a large all-spans device-count shift, run [`assets/queries/broker-version-share-wow.kql`](assets/queries/broker-version-share-wow.kql) (single WoW snapshot) or [`assets/queries/broker-version-share.kql`](assets/queries/broker-version-share.kql) (time-series) and name the version cohort the shift moved with. Do not write "recurring telemetry-shape artifact" without backing data; if you don't have it, drop the callout. diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/classify-novelty.js b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/classify-novelty.js new file mode 100644 index 00000000..a0753b58 --- /dev/null +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/classify-novelty.js @@ -0,0 +1,332 @@ +#!/usr/bin/env node +/** + * classify-novelty.js -- Answer "is this NEW this week, or has it been broken for weeks?" + * + * WHY THIS EXISTS + * --------------- + * bucket-trends.js answers "which direction has this moved over 60 days?" That is a + * *direction* question. It cannot answer the question an on-call engineer actually asks + * when triaging: "which of these 20 moving error codes started THIS week, and which have + * I already been staring at for a month?" + * + * Without that split, the attention section degenerates into a volume-ranked list where a + * flat-but-huge error code outranks a genuine step change, and every card carries the same + * "needs owner triage" boilerplate. That is precisely the failure this script fixes. + * + * IT ALSO SUPPRESSES RATIO ARTIFACTS + * ---------------------------------- + * A week-over-week percentage is meaningless when the PRIOR week was itself anomalous. + * Real example from the 2026-07-30 broker run: + * + * 429 300,664 299,965 892,839 974,980 11,512 32,530 2,724 -> 16,531 + * temporarily_unavailable 30,257 29,962 37,263 6,168 41,971 141 71 -> 36,153 + * + * Both reported as ~+400% WoW and led the regression list. Both are noise: `429` is + * actually 94% BELOW its own 60-day median (it collapsed from ~975K in June), and + * `temporarily_unavailable` merely returned to its normal ~36K band after two suppressed + * weeks. Meanwhile the genuinely new regression that week -- the ipc_* family, flat for + * seven straight weeks (cv 0.02-0.03) then stepping up 22% together -- sat at positions + * #6, #9 and #10. + * + * A series with high variance gets `suppressRatio: true`. The playbook must NOT headline a + * percentage for those; report the absolute level and its position within the historical + * band instead. + * + * FAMILY CLUSTERING + * ----------------- + * Related codes that move together are one root cause, not N findings. When >=2 keys + * sharing a prefix all move the same direction in the same week, they are emitted as a + * `families` entry with a summed series. One story, one card. + * + * IT ALSO DECIDES WHAT DESERVES A CHART (noise control) + * ---------------------------------------------------- + * "Elevated" and "getting worse" are different questions, and conflating them is what made + * the report re-triage the same known issues every week. `ONGOING` is therefore split: + * + * ACCELERATING elevated AND still climbing right now -> promote, chart it, card it + * ONGOING elevated but PLATEAUED -> collapse to a counted line + * + * Only NEW + ACCELERATING form the `attention` set. Everything else is reference material. + * On the 2026-07-31 broker data this is 5 series out of 53, against 13 volume-ranked + * attention rows before -- and the 5 are the real ones. + * + * Two false positives the ACCELERATING gate exists to stop, both observed on real data: + * authorization_pending down 37% WoW, 16% BELOW its own median + * IntuneAppProtectionPolicyRequiredException flat (cv 0.08), +4.9% vs median, down 3.7% WoW + * Both had a rising block-mean, which alone is not evidence of anything. Hence the gate + * requires magnitude (ratio > 1.10), slope (recentRatio > 1.10) AND not-currently-falling. + * + * `weeksElevated` answers "how long has this been like this?" WITHOUT any persisted state -- + * it is derived from the same 9-week series, so two engineers on two machines get identical + * answers and there is nothing to commit, sync, or go stale. It counts consecutive recent + * weeks above the EARLY-window baseline (median of the first third). A series that never + * left that baseline band reports the full window with `sustainedFullWindow: true`, i.e. + * "at this level for as long as we can see" -- a standing condition, not this week's news. + * The reference is the baseline, NOT the current value: "within 20% of current" is + * meaningless for a flat series and made a code that stepped up this week claim it had been + * elevated for seven. + * + * Input: the --json sidecar written by bucket-trends.js. + * + * Usage: + * node classify-novelty.js + * [--floor=N] # ignore keys whose current complete week is below N (default 5000) + * [--family-sep=_] # token separator for family detection ('none' disables) + * [--top=N] # rows printed per bucket (default 8) + * [--json=] # structured sidecar for programmatic use + * [--summary] # counts + rows only, no series arrays + * + * Classification (evaluated in order, first match wins), computed on COMPLETE weeks only: + * VOLATILE cv > 0.60 -> ratio is noise; suppressRatio=true + * RECOVERY prior week < 50% of median -> returning to band, not a new break + * and current >= 70% of median + * NEW current > 115% of median -> stable baseline then a clean step + * and cv < 0.25 + * ONGOING late-third mean > 115% early-third -> already climbing; not new + * IMPROVING current < 80% of median + * STABLE otherwise + * + * The cv < 0.25 guard on NEW is what stops a jittery series from being called a step + * change. A series must have been genuinely boring before a jump counts as news. + */ +const fs = require('fs'); + +const args = process.argv.slice(2); +const file = args.find(a => !a.startsWith('--')); +const floor = +((args.find(a => a.startsWith('--floor=')) || '').split('=')[1] || 5000); +const topN = +((args.find(a => a.startsWith('--top=')) || '').split('=')[1] || 8); +const jsonOut = (args.find(a => a.startsWith('--json=')) || '').split('=')[1]; +const summary = args.includes('--summary'); +const famSepRaw = (args.find(a => a.startsWith('--family-sep=')) || '').split('=')[1]; +const famSep = famSepRaw === undefined ? '_' : famSepRaw; +const familiesEnabled = famSep !== 'none' && famSep !== ''; + +if (!file) { + console.error('Usage: node classify-novelty.js [--floor=N] [--family-sep=_|none] [--top=N] [--json=path] [--summary]'); + process.exit(1); +} + +const d = JSON.parse(fs.readFileSync(file, 'utf8')); +if (!d.buckets) { + console.error('Input does not look like a bucket-trends.js --json sidecar (no .buckets).'); + process.exit(1); +} + +// ---- stats helpers ------------------------------------------------------- +const mean = a => a.reduce((s, v) => s + v, 0) / a.length; +const median = a => { + const s = [...a].sort((x, y) => x - y); + const n = s.length; + return n % 2 ? s[(n - 1) / 2] : (s[n / 2 - 1] + s[n / 2]) / 2; +}; +const stdev = a => { + const m = mean(a); + return Math.sqrt(mean(a.map(v => (v - m) ** 2))); +}; +const pct = v => (v >= 0 ? '+' : '') + (v * 100).toFixed(1) + '%'; + +/** + * `series` follows DISPLAY weeks, which include the in-progress partial week when + * bucket-trends.js ran with --include-partial-end. Classifying on a partial week would + * read as a fake collapse, so trim to the classify-week count. + */ +const completeLen = (d.classifyWeeks || d.weeks || []).length; + +function classify(code, series) { + const comp = series.slice(0, completeLen); + if (comp.length < 4) return null; // too little history to say anything honest + const cur = comp[comp.length - 1]; + const hist = comp.slice(0, -1); + const prev = hist[hist.length - 1]; + const med = median(hist); + if (med <= 0) return null; + + const m = mean(hist); + const cv = m > 0 ? stdev(hist) / m : 99; + const ratio = cur / med; + + const third = Math.max(1, Math.floor(hist.length / 3)); + const early = mean(hist.slice(0, third)); + const late = mean(hist.slice(-third)); + const climb = early > 0 ? late / early : 1; + + // --- Is it STILL climbing, or did it climb and then plateau? --------------- + // `climb` looks at the whole history, so a code that stepped up 6 weeks ago and has been + // flat ever since still reads as "climbing" forever. That is exactly what made the report + // re-triage the same known issues every week. Compare the most recent block against the + // block before it to separate "getting worse now" from "bad, but stable". + const win = Math.min(3, Math.floor(comp.length / 2)); + const recentBlock = mean(comp.slice(-win)); + const priorBlock = mean(comp.slice(-2 * win, -win)); + const recentRatio = priorBlock > 0 ? recentBlock / priorBlock : 1; + + // --- How long has this been at its current level? -------------------------- + // Reference is the EARLY-window baseline (median of the first third), not the current + // value: "within 20% of current" is meaningless for a low-variance series, where a code + // that stepped up this week still looks like it has been here for the whole window. + // Derived purely from the series already in hand -- NO persisted state, so two engineers + // on two machines get identical answers and there is nothing to commit or sync. + const baseline = median(comp.slice(0, third)) || med; + let weeksElevated, sustainedFullWindow; + if (cur < baseline * 1.15) { + // Never left its own early band inside the observable window. Honest ceiling: + // "at this level for as long as we can see" -- a standing condition, not this week's news. + weeksElevated = comp.length; + sustainedFullWindow = true; + } else { + weeksElevated = 0; + for (let i = comp.length - 1; i >= 0; i--) { + if (comp[i] >= baseline * 1.15) weeksElevated++; + else break; + } + sustainedFullWindow = weeksElevated >= comp.length; + } + + let label; + if (cv > 0.60) label = 'VOLATILE'; + else if (prev < med * 0.5 && ratio >= 0.7) label = 'RECOVERY'; + else if (ratio > 1.15 && cv < 0.25) label = 'NEW'; + else if (climb > 1.15) { + // ACCELERATING must mean "getting worse RIGHT NOW", and by a margin worth a card. + // Two observed false positives this gate exists to stop: + // authorization_pending -- down 37% WoW, 16% BELOW its median, but block means drifted up. + // IntuneAppProtectionPolicyRequiredException -- flat (cv 0.08), only +4.9% vs its own + // median, down 3.7% WoW, yet a slow multi-week ramp made it outrank the real finding. + // So require all three: meaningfully elevated, still climbing, and not currently falling. + // ratio > 1.10 is the magnitude bar -- within 10% of its own median is not a story. + const stillRising = ratio > 1.10 && recentRatio > 1.10 && cur >= prev * 0.95; + label = stillRising ? 'ACCELERATING' : 'ONGOING'; + } + else if (ratio < 0.8) label = 'IMPROVING'; + else label = 'STABLE'; + + return { + code, label, + current: cur, + prior: prev, + median: Math.round(med), + cv: +cv.toFixed(2), + vsMedian: +(ratio - 1).toFixed(3), + wowWeek: prev > 0 ? +((cur - prev) / prev).toFixed(3) : null, + weeksElevated, + sustainedFullWindow, + recentRatio: +recentRatio.toFixed(2), + // A percentage off a depressed or wildly swinging base is not reportable as a headline. + suppressRatio: label === 'VOLATILE' || label === 'RECOVERY', + series: comp, + }; +} + +// ---- classify every key across all trend buckets -------------------------- +const all = []; +let belowFloor = 0; +for (const grp of Object.keys(d.buckets)) { + for (const m of d.buckets[grp]) { + const r = classify(m.code, m.series || []); + if (!r) continue; + if (r.current < floor) { belowFloor++; continue; } + r.trendBucket = grp; + all.push(r); + } +} + +// ---- family clustering ---------------------------------------------------- +// Only cluster when members genuinely move together: >=2 members sharing a prefix AND +// agreeing on direction. Grouping divergent codes would invent a story that isn't there. +const families = []; +if (familiesEnabled) { + const byPrefix = new Map(); + for (const r of all) { + const parts = String(r.code).split(famSep); + if (parts.length < 2) continue; + const p = parts[0]; + if (!byPrefix.has(p)) byPrefix.set(p, []); + byPrefix.get(p).push(r); + } + for (const [prefix, members] of byPrefix) { + if (members.length < 2) continue; + const labels = new Set(members.map(x => x.label)); + if (labels.size !== 1) continue; // must agree on direction + const label = [...labels][0]; + if (label === 'STABLE') continue; // a family of nothing is not news + const len = Math.min(...members.map(x => x.series.length)); + const summed = Array.from({ length: len }, (_, i) => + members.reduce((s, x) => s + x.series[i], 0)); + const cur = summed[len - 1], prev = summed[len - 2]; + families.push({ + family: prefix + famSep + '*', + label, + members: members.map(x => x.code), + current: cur, + prior: prev, + wowWeek: prev > 0 ? +((cur - prev) / prev).toFixed(3) : null, + series: summed, + }); + } +} + +// ---- output --------------------------------------------------------------- +// ORDER is also the report's priority order. Only NEW and ACCELERATING earn a chart and a +// card; everything below the ATTENTION line is reference material that must collapse. +const ORDER = ['NEW', 'ACCELERATING', 'ONGOING', 'VOLATILE', 'RECOVERY', 'IMPROVING', 'STABLE']; +const ATTENTION = ['NEW', 'ACCELERATING']; +const HEAD = { + NEW: 'NEW THIS WEEK (stable baseline -> clean step change; lead with these)', + ACCELERATING: 'ACCELERATING (elevated AND still climbing; promote -- this is getting worse)', + ONGOING: 'ONGOING (elevated but PLATEAUED; known/steady -- collapse, do not re-triage)', + VOLATILE: 'VOLATILE (high variance; % is noise -- report absolute level, not ratio)', + RECOVERY: 'RECOVERY (returning to normal band after a dip; not a regression)', + IMPROVING: 'IMPROVING (below historical band)', + STABLE: 'STABLE (within band)', +}; + +const counts = ORDER.map(l => `${l}=${all.filter(r => r.label === l).length}`).join(' '); +console.log(`\nNovelty classification (floor=${floor.toLocaleString()}, complete weeks=${completeLen}): ${counts}`); +if (belowFloor) console.log(`(${belowFloor} key(s) skipped below floor)`); + +const attention = all + .filter(r => ATTENTION.includes(r.label)) + .sort((a, b) => b.current - a.current); +console.log(`ATTENTION set (NEW + ACCELERATING) = ${attention.length} of ${all.length} series.` + + (attention.length === 0 ? ' -> QUIET WEEK: report should be short.' : '')); + +for (const label of ORDER) { + const rows = all.filter(r => r.label === label).sort((a, b) => b.current - a.current); + if (!rows.length) continue; + console.log(`\n### ${HEAD[label]}`); + for (const r of rows.slice(0, topN)) { + const ratioTxt = r.suppressRatio ? '(ratio suppressed)' : `WoW ${pct(r.wowWeek)}`; + const age = r.sustainedFullWindow ? `>=${r.weeksElevated}w` : `${r.weeksElevated}w`; + console.log(` ${r.code.padEnd(44)} cur=${r.current.toLocaleString().padStart(11)} vs median ${pct(r.vsMedian).padStart(8)} cv=${String(r.cv).padStart(4)} elev=${age.padStart(4)} ${ratioTxt}`); + if (!summary) console.log(` ${r.series.map(v => v.toLocaleString()).join(' ')}`); + } + if (rows.length > topN) console.log(` ... and ${rows.length - topN} more`); +} + +if (families.length) { + console.log(`\n### FAMILIES (related keys moving together -- report as ONE finding, one root cause)`); + for (const f of families.sort((a, b) => b.current - a.current)) { + console.log(` ${f.family.padEnd(44)} ${f.label} cur=${f.current.toLocaleString()} WoW ${pct(f.wowWeek)} [${f.members.join(', ')}]`); + if (!summary) console.log(` ${f.series.map(v => v.toLocaleString()).join(' ')}`); + } +} + +if (jsonOut) { + fs.writeFileSync(jsonOut, JSON.stringify({ + floor, + completeWeeks: completeLen, + weeks: (d.classifyWeeks || d.weeks || []).slice(0, completeLen), + metric: d.metric, + key: d.key, + counts: Object.fromEntries(ORDER.map(l => [l, all.filter(r => r.label === l).length])), + // The report's attention section is exactly this list -- nothing else earns a card or a + // chart. Emitting it here (rather than leaving each playbook to re-derive it) is what + // keeps "what needs attention" from drifting back into a volume-ranked dump. + attentionLabels: ATTENTION, + attention: attention.map(r => r.code), + quietWeek: attention.length === 0, + items: all, + families, + }, null, 2)); + console.log(`\nWrote novelty sidecar -> ${jsonOut}`); +} diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 index c96cd6ff..5dac0f1d 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 @@ -31,6 +31,17 @@ (curEnd = end-date, curStart = end-date - 7d). Prevents a stale template stub from being published as if it were fresh, and catches any hand-edit that broke the auto-stamp. + (12 is reserved.) + 13. Section 2 boilerplate uniformity — every attention row needs its own + one-line body, not one sentence copy-pasted across rows. + 14. The top attention row must not be flat — Section 2 leads with what is + NEW (classify-novelty.js), never with the highest-volume row. + 15. VOLATILE / RECOVERY rows must not headline a WoW percentage — their + ratio is measured off an anomalous prior week and is meaningless. + 16. Every visible attention row carries an inline .item-spark sparkline. + 17. Section 2 visible-row budget (<= 8 rows including wins). + 18. The 60-day section is a slow-burn DETECTOR, not a browsable chart + catalog — at most 6 charts outside a collapsed fold. Exits with non-zero status if any HARD check fails (stale tokens, devs/reqs leak, U+FFFD, unbalanced div depth, missing layout-guard CSS). @@ -169,7 +180,13 @@ if ($kpiTiles -ge 4 -and $sparkCount -lt [Math]::Ceiling($kpiTiles / 2)) { Pass "KPI tiles have data-spark coverage ($sparkCount/$kpiTiles)" } if ($totalCharts -lt 15) { - Add-Warn "Only $totalCharts chart elements found. Expected ~30+ (KPI sparks + 60d-trend rows + WoW-table rows). Did you forget to add data-trend attributes to the WoW / trend tables?" + # Threshold deliberately LOW. This check guards against a body rebuilt with the + # data-trend attributes dropped entirely -- it is not a "more charts is better" + # signal. The redesign that moved sparklines into the attention rows and collapsed + # the 60-day catalog cut a real report from ~226 chart elements to a few dozen, and + # that is the intended direction: charts belong next to findings. Check 18 enforces + # the upper bound; this one only catches "the charts vanished". + Add-Warn "Only $totalCharts chart elements found. Expect at least one per KPI tile plus one per visible attention row. Did the body get rebuilt without data-trend attributes?" } else { Pass "Overall chart coverage looks reasonable ($totalCharts elements)" } @@ -302,12 +319,24 @@ if ($hasAttrCard) { $trendMatches = [regex]::Matches($content, "data-trend=['""]?\[([0-9.,e\s+\-]+)\]") $suspectCount = 0 $suspectFirst = $null +$flatCount = 0 +$flatFirst = $null foreach ($m in $trendMatches) { $arrStr = $m.Groups[1].Value $vals = $arrStr.Split(',') | ForEach-Object { try { [double]$_.Trim() } catch { 0 } } if ($vals.Count -lt 6) { continue } - # Filter 1: trend with all values < 100 is suspicious (real codes don't sit at 30-50 devices/week for 8 weeks) $maxVal = ($vals | Measure-Object -Maximum).Maximum + $minVal = ($vals | Measure-Object -Minimum).Minimum + + # A perfectly flat series is fabricated in ANY app profile -- real telemetry + # never repeats a value 8 weeks running. + if ($maxVal -eq $minVal) { + $flatCount++ + if (-not $flatFirst) { $flatFirst = $arrStr } + continue + } + + # Filter 1: trend with all values < 100 is suspicious (real codes don't sit at 30-50 devices/week for 8 weeks) if ($maxVal -lt 100) { $suspectCount++ if (-not $suspectFirst) { $suspectFirst = $arrStr } @@ -317,10 +346,13 @@ foreach ($m in $trendMatches) { # Filter 3: implausibly regular - if every consecutive delta has the same sign AND is < 5% of the value, that's a fake. # Skip this; too easy to false-positive on genuinely monotonic real series like no_tokens_found. } +if ($flatCount -gt 0) { + Add-Warn "$flatCount data-trend array(s) are perfectly flat (every value identical). Real telemetry never does this. First: [$flatFirst]." +} if ($suspectCount -gt 0) { - Add-Warn "$suspectCount data-trend array(s) have peak value < 100 (suspicious — real WoW-table series usually peak >= 100 devices/week). Likely fabricated. First: [$suspectFirst]. Source from assets/queries/wow-table-sparkline-series.kql instead." -} else { - Pass "No suspicious low-peak data-trend arrays detected" + Add-Warn "$suspectCount data-trend array(s) have an implausible magnitude (peak < 100). Likely fabricated. First: [$suspectFirst]. Source from assets/queries/wow-table-sparkline-series.kql instead." +} elseif ($flatCount -eq 0) { + Pass "No suspicious data-trend arrays detected" } # ---- 11. Rolling-window header integrity ---- @@ -363,6 +395,174 @@ if ($filename -match '^oncall-wow-report-(\d{4}-\d{2}-\d{2})\.html$') { Add-Warn "Filename '$filename' does not match 'oncall-wow-report-YYYY-MM-DD.html'; skipping meta-line date consistency check." } +# ---- 13. Section 2 boilerplate uniformity (HARD FAIL) ---- +# The v9 report shipped with the SAME sentence on all 10 red-callout rows +# ("Current-window movement needs owner triage; deep dive below has originator and +# dimensions."), plus 10 identical slow-burn bodies and 8 identical win bodies. +# The reader complaint that produced this check was, verbatim: "it's actually hard +# to understand what's actually changed from last week ... no idea what's existing +# regression vs new this week". A body that would read identically on any other row +# carries zero information -- it is worse than no body, because it occupies the slot +# where the specific finding should have been. +$attStart = $content.IndexOf('id="attention"') +$attEnd = $content.IndexOf('id="trend60d"') +if ($attEnd -lt 0) { $attEnd = $content.IndexOf('id="trend"') } +if ($attStart -ge 0 -and $attEnd -gt $attStart) { + $attSec = $content.Substring($attStart, $attEnd - $attStart) + + # NOTE: 'Singleline' is load-bearing. Without it `.` does not cross newlines, so this + # only matched when the generating agent happened to emit .item-body on ONE line. The + # template pretty-prints these blocks across several lines, so a copied-from-template + # row would silently skip the check -- a check that no-ops while printing nothing is + # worse than no check. Same applies to the .item-head regex below. + $bodies = [regex]::Matches($attSec, '
(.*?)
', 'Singleline') | + ForEach-Object { + # Strip inline markup + collapse whitespace so two bodies that differ + # only by a wrapper still count as duplicates. + (([regex]::Replace($_.Groups[1].Value, '<[^>]+>', ' ')) -replace '\s+', ' ').Trim() + } | Where-Object { $_.Length -gt 0 } + + if ($bodies.Count -eq 0) { + Add-Warn "No .item-body rows found in Section 2 -- cannot check for boilerplate. If the attention section uses different markup, this check needs updating." + } else { + $dupes = $bodies | Group-Object | Where-Object { $_.Count -ge 3 } | Sort-Object Count -Descending + if ($dupes) { + foreach ($d in $dupes) { + $snip = $d.Name.Substring(0, [Math]::Min(80, $d.Name.Length)) + Add-Fail "Section 2 boilerplate: $($d.Count) attention rows share the identical body `"$snip...`". Every row must say what changed, from what to what, and why it is or isn't alarming. If there is nothing row-specific to say, the row does not belong in Section 2." + } + } else { + Pass "Section 2 row bodies are row-specific (no sentence repeated 3+ times)" + } + } + + # ---- 14. Top attention row should not be a flat mover (WARN) ---- + # Ranking the attention list by device count is how a code with a +0.1% WoW delta + # ended up at position #1 while the genuinely new ipc_* family sat at #6/#9/#10. + # Section 2 is ordered by NOVELTY (classify-novelty.js), not volume -- so a + # near-zero delta in the lead slot means the ordering was never applied. + # + # The delta label has been observed spelled "Δ WoW", "Δ WoW" and "Delta WoW" + # across runs. Match all of them: a check that silently no-ops on a spelling + # variant is worse than no check, because it reports nothing and looks healthy. + $dLabel = '(?:Δ|Δ|\u0394|Delta)\s*WoW' + $heads = [regex]::Matches($attSec, '
(.*?)
', 'Singleline') + if ($heads.Count -eq 0) { + Add-Warn "No .item-head rows found in Section 2 -- lead-row and volatile-chip checks skipped. If the attention markup changed, these checks need updating." + } else { + $firstHead = $heads[0].Groups[1].Value + $nameM = [regex]::Match($firstHead, '\s*([^<]+)') + $nm = if ($nameM.Success) { $nameM.Groups[1].Value.Trim() } else { '(unnamed)' } + $dw = [regex]::Match($firstHead, "$dLabel([+\-]?[\d.]+)%") + if (-not $dw.Success) { + Add-Warn "Could not parse a WoW delta chip on the top attention row '$nm' -- lead-row flatness check did not run. Expected an m-label matching 'Delta WoW'." + } elseif ([Math]::Abs([double]$dw.Groups[1].Value) -lt 5) { + Add-Warn "Top attention row '$nm' has a WoW delta of $($dw.Groups[1].Value)% -- essentially flat. Section 2 must lead with what is NEW (classify-novelty.js label NEW), not with the highest-volume row. If genuinely nothing is new this week, say so explicitly instead of promoting a flat mover." + } else { + Pass "Top attention row '$nm' has a material WoW delta ($($dw.Groups[1].Value)%)" + } + + # ---- 15. VOLATILE / RECOVERY rows must not headline a WoW percentage ---- + # These carry suppressRatio:true because their ratio is measured off an + # anomalous prior week. 429 shipped as "+401.8%" while sitting 94.5% BELOW + # its own 60-day median -- tagging the row VOLATILE and caveating in the body + # is NOT sufficient, because the chip row is the loudest element and the eye + # reads it first. Replace the Δ WoW chip with a "vs 60d median" chip. + $offenders = @() + foreach ($h in $heads) { + $hv = $h.Groups[1].Value + if ($hv -notmatch '>(VOLATILE|RECOVERY)<') { continue } + $m = [regex]::Match($hv, "$dLabel([+\-]?[\d.]+)%") + if ($m.Success -and [Math]::Abs([double]$m.Groups[1].Value) -ge 25) { + $n2 = [regex]::Match($hv, '\s*([^<]+)') + $who = if ($n2.Success) { $n2.Groups[1].Value.Trim() } else { '(unnamed)' } + $offenders += "$who ($($m.Groups[1].Value)%)" + } + } + if ($offenders.Count -gt 0) { + Add-Fail "VOLATILE/RECOVERY rows still headline a WoW percentage: $($offenders -join ', '). Their ratio is an artifact of a depressed prior week and the chip is the first thing a reader sees -- a caveat in the body does not undo it. Replace the 'Δ WoW' chip with a 'vs 60d median' chip showing the absolute level's position in the band." + } else { + Pass "No VOLATILE/RECOVERY row headlines a WoW percentage" + } + } + + # ---- 16. Every VISIBLE attention row must carry its own sparkline (HARD FAIL) ---- + # The 2026-07-31 report shipped 13 attention rows with ZERO charts, while the 60-day + # catalog below it carried 38. The reader's complaint was exactly that inversion: + # "I don't see any graphs for the things that need attention this week". + # + # This is not decoration. "+25.7% WoW" is unreadable on its own -- the reader cannot + # tell a flat-for-seven-weeks series that just stepped up (a real regression) from one + # that has been bouncing all along (noise wearing a big percentage). The 9-week shape + # IS the novelty argument, so it has to sit in the row that makes the claim. + # + # Rows inside a collapsed
fold are reference material and are exempt. + $attVisible = [regex]::Replace($attSec, '', '', 'Singleline') + $visHeads = [regex]::Matches($attVisible, '
(.*?)
', 'Singleline') + if ($visHeads.Count -eq 0) { + Add-Warn "No visible .item-head rows in Section 2 -- sparkline-coverage check skipped." + } else { + $noSpark = @() + foreach ($h in $visHeads) { + $hv = $h.Groups[1].Value + if ($hv -match 'item-spark' -or $hv -match 'data-trend') { continue } + $n = [regex]::Match($hv, '\s*([^<]+)') + $noSpark += $(if ($n.Success) { $n.Groups[1].Value.Trim() } else { '(unnamed)' }) + } + if ($noSpark.Count -gt 0) { + Add-Fail "$($noSpark.Count) of $($visHeads.Count) visible attention row(s) have no inline sparkline: $($noSpark -join ', '). Every attention row must carry a .item-spark with data-trend holding its 9-week series -- the shape is what distinguishes a step change from ordinary variance. Charts belong WITH the finding, not in a separate browsable section." + } else { + Pass "All $($visHeads.Count) visible attention row(s) carry an inline sparkline" + } + + # ---- 17. Attention section must stay short (WARN) ---- + # classify-novelty.js emits an ATTENTION set (NEW + ACCELERATING). On real broker + # data that is 5 series out of 53. If the rendered section is much larger than + # that, the classifier's set was ignored and volume ranking crept back in -- + # which is what produced 13 rows for what turned out to be one incident. + if ($visHeads.Count -gt 8) { + Add-Warn "Section 2 has $($visHeads.Count) visible attention rows. classify-novelty.js's ATTENTION set (NEW + ACCELERATING) is typically 3-6 series; anything much larger means plateaued or flat codes were promoted. Move ONGOING/STABLE rows into the collapsed fold -- a long attention list trains the reader to skim the one section that must not be skimmed." + } else { + Pass "Section 2 attention list is short ($($visHeads.Count) visible row(s))" + } + } +} else { + Add-Warn "Could not locate the attention section -- skipping boilerplate + lead-row checks." +} + +# ---- 18. The 60-day section must not be a browsable chart catalog (HARD FAIL) ---- +# The 60-day pass exists to catch slow burns that week-over-week structurally cannot see +# (it found `Failed to parse JWT` going 7 -> 3,461 devices). That is its whole job. +# +# What it must NOT be is a scrollable list of every classified code with a chart each. +# Measured on the 2026-07-31 report: 29 rows / 38 charts in this section, of which 27 were +# duplicated verbatim into the error-code and error-type tables further down -- ~93% +# redundant. Combined with the error tables the report rendered 100 sparklines for a single +# incident, and the four series that actually changed were indistinguishable from the rest. +# +# Rule: only findings get charts in the main flow, and a finding is either NEW/ACCELERATING +# (already charted in Section 2) or a promoted slow burn. Everything else goes in a +# collapsed fold WITHOUT charts. Charts inside
are exempt -- the reader opted in. +# +# Broker's universe is 40-50 error codes, so charting "everything" is exactly the catalog +# failure this check exists to stop. +$t60Cap = 6 +$t60Start = $content.IndexOf('id="trend60d"') +if ($t60Start -ge 0) { + $t60End = $content.IndexOf('', '', 'Singleline') + $t60Charts = [regex]::Matches($t60Visible, 'data-trend=| fold with no charts. A previous run shipped 38 charts here against 0 in the attention section, which is precisely backwards." + } else { + Pass "60-day section is a detector, not a catalog ($t60Charts visible chart(s), cap $t60Cap)" + } +} else { + Add-Warn "Could not locate the 60-day section -- skipping the chart-catalog check." +} + Write-Host "" if ($failures.Count -eq 0) { Write-Host "All hard checks passed." -ForegroundColor Green diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html b/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html index cf14c921..f3b563bc 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html @@ -389,6 +389,52 @@ margin-left: auto; color: #0550ae; text-decoration: none; font-weight: 600; } .item-foot .arrow-link:hover { text-decoration: underline; } + + /* ---- Novelty presentation -------------------------------------------------- + The sparkline IS the argument. A reader cannot judge "+25% WoW" without + seeing whether the series was flat for seven weeks or bouncing all along, + so every attention row carries its own 9-week shape inline. Charts live + WITH the finding, not in a separate browsable catalog. */ + .item-spark { + display: inline-flex; align-items: center; line-height: 0; + padding: 2px 6px; border-radius: 6px; + background: #fff; border: 1px solid #eaeef2; + } + .item-spark svg { display: block; } + .spark-cap { + font-size: 10px; color: #8c959f; font-weight: 500; + letter-spacing: 0.2px; white-space: nowrap; + } + .tag-new { background: #ffeef0; color: #cf222e; border: 1px solid #ffc1c0; } + .tag-accel { background: #fff1e5; color: #bc4c00; border: 1px solid #ffd8b5; } + .tag-ongoing { background: #f0f3f6; color: #656d76; } + + /* ---- Collapsed reference material ------------------------------------------ + Anything that is elevated-but-plateaued, flat, or already-known is reference, + not this week's news. It stays reachable (an OCE may need to confirm a level) + but must not occupy vertical space or imply "triage me". */ + details.fold { + margin-top: 10px; border: 1px solid #eaeef2; + border-radius: 6px; background: #fbfcfd; + } + details.fold > summary { + cursor: pointer; padding: 8px 12px; font-size: 12.5px; + font-weight: 600; color: #1f2328; list-style: none; user-select: none; + } + details.fold > summary::-webkit-details-marker { display: none; } + details.fold > summary::before { content: "\25B8 "; color: #656d76; } + details.fold[open] > summary::before { content: "\25BE "; } + details.fold > summary .count { font-weight: 400; color: #656d76; } + details.fold > .fold-body { padding: 0 12px 12px; } + details.fold > .fold-body > .item-list { margin-top: 0; } + + /* ---- Quiet week ------------------------------------------------------------- + A week with nothing new must LOOK like a week with nothing new. */ + .quiet-banner { + border: 1px solid #1a7f37; background: #dafbe1; color: #116329; + border-radius: 6px; padding: 12px 14px; margin: 12px 0; font-size: 13px; + } + .quiet-banner strong { color: #1a7f37; } @@ -475,6 +521,19 @@

📊 Top-line health — auth-only denominat

🚨 Things that need attention this week

+ +
ℹ️ Denominator caveat — read this first

The headline BrokerAdoptionStats device count dropped −18.6% WoW (1.52 B → 1.24 B), but this is not a real fleet shrink. The drop is fully explained by three low-value spans deflating as the 16.0.1 rollout completes:

@@ -487,54 +546,82 @@

🚨 Things that need attention this week

-
🔴 WoW regressions (last 7 days vs prior 7) — sorted by current-week devices, descending
-

Tags: NEW first appeared this week or last; 60d↑ also rising on the 60-day window; broker / eSTS / Android / env = originator. Built from the standard WoW table union with wow-movers.kql so small-but-recent spikes appear alongside the high-volume movers.

+
🔴 NEW this week — flat baseline, then a clean step change. Triage these first.
+

Ordered by novelty, not volume — a flat-but-huge code is not news. Labels come from classify-novelty.js, which scores every series against its own 7-week baseline. The sparkline is the argument: a flat line that steps up at the right edge is a real regression; a line that was already bouncing is noise wearing a big percentage. broker / eSTS / Android / env = originator.

- +
EXAMPLE_error_code - devicesEXAMPLE 65 K - Δ WoWEXAMPLE +6.1% - on 16.0.1EXAMPLE 73% + + 9 wk + devicesEXAMPLE 52.1 K + Δ WoWEXAMPLE +25.7% + vs own medianEXAMPLE +19.4% + NEW broker - 60d↑ +52%
-
EXAMPLE one-line narrative: throw site common/SomeClass.someMethod:NN, dominant message, and the verdict. Keep this short — the deep dive is in the attribution card below.
+
EXAMPLE one-line narrative: flat at ~41 K for seven weeks (cv 0.03), stepped to 52 K this week. Throw site common/SomeClass.someMethod:NN. Keep this short — the deep dive is in the attribution card below.
Owner: EXAMPLE teamAttribution card →
+
-
🟡 Slow-burn 60-day regressions — rising on 60d window but flat WoW; codes that also moved WoW are in the red callout above with a 60d↑ tag
+
🟠 Getting worse — already elevated and still climbing. Not new, but the trend has not turned.
- EXAMPLE_slow_burn_code - devicesEXAMPLE 4.5 M - Δ 60dEXAMPLE +56% - Δ requests 60dEXAMPLE +40% - on 16.0.1EXAMPLE 78% + EXAMPLE_accelerating_code + + 9 wk + devicesEXAMPLE 86.8 K + Δ WoWEXAMPLE +28.5% + vs own medianEXAMPLE +108.7% - broker + ACCELERATING + 3 wk elevated
-
EXAMPLE: WoW only +X%. Tracks 16.0.1 rollout share; one-line hypothesis or owner pointer.
+
EXAMPLE: climbing for three consecutive weeks and up again this week. State what changed since it was last looked at — if nothing has, say so and name the owner.
-

See the 60-day trend section for the full ranked list.

+ +
+ Ongoing / known — elevated but plateaued (EXAMPLE 4 codes, EXAMPLE 2.1 M devices — no change in status this week) +
+
+
+
+ EXAMPLE_ongoing_code + + devicesEXAMPLE 1.79 M + Δ WoWEXAMPLE −3.2% + ONGOING · 3 wk +
+
+
+
+
+
🟢 Real wins this week
@@ -542,6 +629,8 @@

🚨 Things that need attention this week

EXAMPLE_recovered_code + + 9 wk devicesEXAMPLE 834 K Δ WoWEXAMPLE −86% Δ requestsEXAMPLE −78% @@ -559,59 +648,89 @@

🚨 Things that need attention this week

-

📈 60-Day Trend Analysis — literal last 60 days ending today; final bar = current partial week

+

📈 60-day cross-check — slow-burn detector, not a browsing list

- Methodology: Pulled all error codes from the ErrorStats view over the last 9 weeks. Dropped the partial start week (Mar 1). Kept all codes whose peak weekly device count ≥ 10 K. Bucketed each 8-week series by delta = (last − first) / first: - regression if delta > +15% and trajectory is monotonic-ish; ephemeral spike if peak ≥ 3× mean of surrounding weeks; improvement if delta < −15%; flat otherwise. Every code in the regression list gets a spike-attribution card below. + Why this section is short. The 60-day pass exists to catch what a week-over-week + comparison structurally cannot: a code that climbs 10 % every week for two months and never + once trips a WoW threshold. It found Failed to parse JWT going 7 → 3,461 devices + that way. That is its entire job. +

+ It is not a catalog to browse. Rendering all ~30 classified codes with a chart each + produced ~100 sparklines for what was, on inspection, a single incident — and it buried the four + series that actually changed. So: anything the 60-day pass flags that is also in the + attention section above is not repeated here, and anything flat is collapsed. +

+ Method: weekly device counts over the last 9 weeks; drop the partial start week; + keep codes with peak weekly devices ≥ 10 K; classify by delta = (last − first) / first + — regression > +15 % and monotonic-ish, ephemeral spike if peak ≥ 3× surrounding + mean, improvement < −15 %, else flat. Delta math uses complete weeks only.
+
-
⚠️ True 60-day regressions — 5 codes
- - - - - - - - - -
Error codeWk 1 devicesWk 8 devicesΔ over 8w60d sparklineTrajectory
no_tokens_found13.9 M23.7 M+70.6%monotonic up
unauthorized_client2.72 M3.37 M+23.6%monotonic up
Code:-631.8 K86.4 K+171.5%step-up at week 6
unknown_crypto_error59.3 K78.4 K+32.4%U-shaped, climbing
null_pointer_error48.5 K70.7 K+45.9%monotonic up
+
🐌 Slow burns — rising over 60 days, invisible week-over-week
+
+
+
+ EXAMPLE_slow_burn_code + + 9 wk + devicesEXAMPLE 3,461 + Δ WoWEXAMPLE +8.4% + Δ 60dEXAMPLE +49,000% + SLOW BURN +
+
EXAMPLE: never tripped a WoW threshold — grew ~8-12 % every week for eight weeks. This is the case the 60-day pass exists for. Name the owner and the suspected change.
+
+
-
-
⚡ Ephemeral 60-day spikes (peaked then recovered)
- - - - - - - -
Error codeBaselinePeakNow60d sparkline
timed_out_execution17.9 M142.9 M (week Apr 12)53.4 M
unknown_authority~1 K34.1 M (week Apr 12)1.45 M
429 (eSTS rate-limit)~10218 K (week Mar 22)2.5 K
-

Both unknown_authority (common #3082 ABBA deadlock fix) and timed_out_execution (broker #141 flight gating) are recovering. Recommendation: add Aria guardrail at >1M devices/week for unknown_authority to detect any future excursion early.

-
+ +
+ Full 60-day classification (EXAMPLE 29 codes — 5 regression, 3 ephemeral spike, 8 improvement, 13 flat) +
-
-
✅ True 60-day improvements
- - - - - - - - - -
Error codeWk 1Wk 8ΔSparkline
timed_out36.1 M5.1 M−85.9%
invalid_scope1.92 M0.36 M−81.3%
timed_out_thread_pool_saturated1.64 M0.62 M−62.1%
illegal_argument_exception0.21 M0.19 M−7.5% (peak −62%)
null_object, device_network_not_available, access_denied, ONLY_SUPPORTS_ACCOUNT_MANAGER_ERROR_CODE, invalid_keyall −17% to −78% over 8 wks (see appendix)
-

Note: the timed_out drop and timed_out_execution climb are partly the same event — broker #141 reclassifies legacy timed_out into the more specific timed_out_execution. The reclassification is net-neutral but the new code is louder; treat the timed_out "win" with caution.

-
+

Regressions — codes already covered above are marked (in attention).

+ + + + + + +
Error codeWk 1 devicesWk 8 devicesΔ over 8wTrajectory
EXAMPLE_no_tokens_found (in attention)13.9 M23.7 M+70.6%monotonic up
EXAMPLE_unknown_crypto_error59.3 K78.4 K+32.4%U-shaped, climbing
-
-
Flat on 60d (within ±10%)
-

io_error, no_account_found, invalid_grant, interaction_required, device_network_not_available_doze_mode, authorization_pending, expired_token, User cancelled, auth_cancelled_by_sdk, invalid_resource, invalid_request, device_registration_needed, Code:-1, Code:-2, Code:-8, operation_interrupted, ipc_return_null_cursor, device_needs_to_be_managed, Redirect url scheme not SSL protected, ipc_operation_not_supported_on_server_side, invalid_client, ipc_connection_error, unknown_error.

-
+

Ephemeral spikes — peaked then recovered. Not actionable unless the tail is still elevated.

+ + + + + + +
Error codeBaselinePeakNow
EXAMPLE_timed_out_execution17.9 M142.9 M (wk Apr 12)53.4 M
EXAMPLE_429 (eSTS rate-limit)~10218 K (wk Mar 22)2.5 K
+ +

Improvements

+ + + + + + +
Error codeWk 1Wk 8Δ
EXAMPLE_timed_out36.1 M5.1 M−85.9%
EXAMPLE_invalid_scope1.92 M0.36 M−81.3%
+

Caveat to carry: a code that drops because a change reclassified it into a new code is not a win. State the pairing when you see one (e.g. legacy timed_out falling as timed_out_execution rises).

+ +

Flat on 60d (within ±10%)

+

EXAMPLE_io_error, EXAMPLE_no_account_found, EXAMPLE_invalid_grant, EXAMPLE_interaction_required — list the rest inline, no table.

+ +
+

🔎 Spike Attribution — one card per regression

diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md b/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md index 9c0fd35b..34fcf89f 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md @@ -129,12 +129,30 @@ Required spark/trend coverage in every report: | Where | Attribute | Length | Color (see palette below) | |---|---|---|---| | Every KPI tile in `.kpi-grid` (Top-line health) | `
` inside the tile | 8–9 weekly values | blue/green/dark-blue per metric semantic | -| **Every** row in the 60-day trend tables — true regressions, **ephemeral spikes**, and **true improvements** (all three callout tables) | `` in the trajectory cell | ~9 weekly values (incl. the current partial week as the final point) | red regression / amber spike / green improvement / grey flat | +| **Every visible row in the Section 2 attention list** — including the wins | `` right after `.item-name` | 9 weekly values | red worsening / orange accelerating / green improving / amber volatile | +| 60-day section: **only the promoted slow burns** (rising on 60d, not already in Section 2 — often zero) | `` in the trajectory cell | ~9 weekly values (incl. the current partial week as the final point) | red regression / amber spike / green improvement | | Every row in the error-codes WoW table and error-types WoW table | `` in the 60d-trend column | 8 complete weekly values (no partial week — see `wow-table-sparkline-series.kql`) | same palette | +> **📌 The two WoW reference tables keep a sparkline on every row — deliberate exemption, decided +> explicitly. Do not strip them when reducing noise.** They are **lookup tables**: the reader arrives +> with a code in mind and scans for it, so the sparkline is glanceable context in a cell their eye is +> already on. The noise failure was the *60-day trend catalog* — a section read top-to-bottom whose +> rows ~93% duplicated these tables — not the tables themselves. + +> **⚠️ Charts go WITH the finding, not in a browsable section.** The 60-day section used to chart +> every classified code — 38 charts, ~93% duplicating rows in the error tables below, while the +> attention section above had **zero**. The full 60-day classification now lives in a `
` fold +> **without** a chart column; only promoted slow burns are charted in the main flow. +> `validate-report.ps1` check 16 hard-fails a visible attention row missing its sparkline, and check 18 +> hard-fails a 60-day section that renders more than 6 charts outside a fold. +> Charts *inside* a `
` fold are exempt from both — the reader opted in. **Checks 16/17/18 are +> scoped to Section 2 and the 60-day section only** — they intentionally say nothing about the WoW +> reference tables above. + **Past failure modes:** - v7 first pass: the body rebuild emitted *zero* `data-spark` / `data-trend` (validator now hard-fails this). -- v7 second pass: only the *true regressions* table got sparklines; the **ephemeral spikes** and **true improvements** tables were left text-only. All three tables in the 60-day trend section need the trajectory column with a sparkline — the validator's overall-coverage warn (≥15) catches this approximately, but the rule of thumb is: **if a row reports an 8-week delta, it gets a sparkline.** +- v7 second pass: only the *true regressions* table got sparklines; the ephemeral-spike and improvement rows were left text-only. Rule of thumb at the time: *"if a row reports an 8-week delta, it gets a sparkline."* +- v9: that rule was over-applied and became the opposite problem — **100 sparklines across the report, none of them next to a finding.** The rule is now **"if a row is a finding, it gets a sparkline"**; volume of charts is not coverage. ## Traffic-shape callout styling