Skip to content

OCE report: add novelty noise gate and redesign the Broker report around it, Fixes AB#3733390 - #460

Open
Shahzaib (shahzaibj) wants to merge 1 commit into
masterfrom
shjameel-microsoft-oce-noise-gate
Open

OCE report: add novelty noise gate and redesign the Broker report around it, Fixes AB#3733390#460
Shahzaib (shahzaibj) wants to merge 1 commit into
masterfrom
shjameel-microsoft-oce-noise-gate

Conversation

@shahzaibj

@shahzaibj Shahzaib (shahzaibj) commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Why

The 60-day trend section of the weekly Broker report had grown into a catalog. Every code with a 60-day regression got a full row and a chart, whether or not anything had changed recently — so each week the on-call engineer was asked to re-triage the same long-standing known regressions. Meanwhile the section that actually matters, "things that need attention this week", had no charts at all.

That's backwards. The report was generating noise where nothing had changed and withholding evidence where something had.

What this does

Adds assets/scripts/classify-novelty.js — a novelty classifier that reads a bucket-trends.js --json= sidecar and labels each key against its own 7-week baseline:

Label Meaning
NEW absent (or negligible) in the baseline, present now
ACCELERATING already elevated, and materially worse this week
ONGOING elevated but flat — a known regression, not news
VOLATILE swings wide enough that this week isn't distinguishable from noise
RECOVERY / IMPROVING moving the right way
STABLE nothing to say

It also clusters related codes into families, so a single upstream failure doesn't consume six attention slots.

The classifier is the noise gate. Its attention set (NEW + ACCELERATING), plus at most 2 wins, is all that renders visibly with charts. Everything else collapses into a fold — still one click away, never deleted, but no longer competing for the reader's attention.

The Broker report template is redesigned around that gate:

  • Attention rows carry a mandatory inline sparkline, so a claim and its evidence sit on the same line.
  • The 60-day section becomes a detector, not a catalog — capped at 6 charts.
  • The attention list is capped at 8 visible rows.

validate-report.ps1 gains hard checks 13–18 so a future run can't quietly regress the gate: row-body specificity, no flat top row, no suppressed-ratio chip, mandatory .item-spark on attention rows, the 8-row cap, and the 6-chart cap.

What this is not

This does not change what the classifier is fed. It grades Sunday-aligned calendar weeks, which is what it did before. That window is misaligned with the rolling 7-day window the report displays — a real bug, but a separate one, fixed in the follow-up PR so it can be reviewed on its own evidence.

Check 12 in validate-report.ps1 is deliberately left as a reserved gap. The prose refers to checks by number, and the Authenticator profile (next PR) fills that slot — keeping the numbering stable across the stack means the follow-up's validator diff is a pure insertion rather than a renumbering.

Verification

validate-report.ps1 -Path oncall-wow-report-2026-08-18.html — all hard checks pass, exit 0, including 13–18:

[OK] All 4 visible attention row(s) carry an inline sparkline
[OK] Section 2 attention list is short (4 visible row(s))
[OK] 60-day section is a detector, not a catalog (0 visible chart(s), cap 6)
[OK] Section 2 row bodies are row-specific
[OK] No VOLATILE/RECOVERY row headlines a WoW percentage

Stack

This is 1 of 3. Each PR is reviewable in isolation:

  1. this PR — novelty noise gate + Broker report redesign → master
  2. OCE report: add Authenticator app telemetry and turn the skill into a router, Fixes AB#3731627 #461 — Authenticator app report + router-ify the skill → this branch
  3. OCE report: align the noise gate to the report's rolling window, Fixes AB#3731628 #462 — align the noise gate to the report's rolling window → PR 2's branch

Splitting this way keeps the Broker-behaviour changes separate from the purely-additive Authenticator support, so neither has to be reviewed through the other.

Fixes AB#3733390

…und it

The weekly Broker report had become a browsing exercise rather than a triage
tool. The 60-day section rendered 38 charts, ~93% of which duplicated rows in
the error tables below it, while the "needs attention this week" section --
the part an on-call engineer actually reads first -- carried 13 volume-ranked
rows and zero charts. A flat-but-huge code led the list; the genuinely new
ipc_* family sat at positions #6/#9/#10.

Root cause: bucket-trends.js reports what MOVED, but nothing decided whether a
movement was NEWS. Ranking by device count is not a proxy for novelty.

This change adds classify-novelty.js, which labels every series against its own
7-week baseline (NEW / ACCELERATING / ONGOING / VOLATILE / RECOVERY / IMPROVING
/ STABLE) and emits an `attention` set = NEW + ACCELERATING. That set, plus at
most 2 wins, is all that renders visibly with charts; everything still-elevated
collapses into a fold with its weeksElevated count. The ACCELERATING/ONGOING
split is the whole fix: only "still getting worse" earns a second look.

The 60-day section becomes a slow-burn DETECTOR -- it charts only what it
promotes (rows rising on 60d and absent from the attention section, typically
0-3, often zero) and folds the full classification with no chart column.
Sections 6/7 keep a per-row sparkline as a deliberate exemption: they are
lookup tables, not a browsing section.

Also guards two measured false positives: a WoW % off an anomalous prior week
(429 headlined at +397.8% while sitting 94.5% BELOW its own 60-day median), and
a slow drift in block means labelling a flat, WoW-negative series ACCELERATING.

validate-report.ps1 gains checks 13-18 to enforce all of this (row-body
specificity, no flat top row, no suppressed-ratio chip, mandatory .item-spark,
<=8 visible rows, <=6 charts in the 60-day section). Check 12 is left reserved.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@shahzaibj
Shahzaib (shahzaibj) requested a review from a team as a code owner August 20, 2026 01:27
@github-actions github-actions Bot changed the title OCE report: add novelty noise gate and redesign the Broker report around it OCE report: add novelty noise gate and redesign the Broker report around it, Fixes AB#3733390 Aug 20, 2026
@github-actions

Copy link
Copy Markdown

✅ Work item link check complete. Description contains link AB#3733390 to an Azure Boards work item.

@github-actions

Copy link
Copy Markdown

✅ Work item link check complete. Description contains link AB#3733390 to an Azure Boards work item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant