Skip to content

OCE report: align the noise gate to the report's rolling window, Fixes AB#3731628 - #462

Open
Shahzaib (shahzaibj) wants to merge 1 commit into
shjameel-microsoft-oce-authapp-v2from
shjameel-microsoft-oce-window-fix
Open

OCE report: align the noise gate to the report's rolling window, Fixes AB#3731628#462
Shahzaib (shahzaibj) wants to merge 1 commit into
shjameel-microsoft-oce-authapp-v2from
shjameel-microsoft-oce-window-fix

Conversation

@shahzaibj

@shahzaibj Shahzaib (shahzaibj) commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

The bug

The novelty classifier graded Sunday-aligned calendar weeks. The report displays a rolling 7-day window. Those two windows drift apart by up to six days — so anything that turned in the last ~6 days, exactly the period an on-call engineer cares about most, was structurally invisible to the noise gate.

Measured on the 2026-08-01 run: the gate's "current" week was 07/19 → 07/26 against a report window of 07/25 → 08/01. One day of overlap.

Code Report showed Classifier saw Verdict printed
authorization_pending +63.2% −37.1% ONGOING — do not re-triage
expired_token +26.7% −51.0% ONGOING — do not re-triage

Both were filed "don't look at this" directly beneath their own rising numbers. On the Authenticator side the same defect surfaced differently: red scoreboard pills (rolling-derived) sitting above an empty "Needs attention" section (calendar-derived) — the exact confusion reported.

The fix

Bucket the 60-day trend and the sparklines with:

bin_at(<TIME>, 7d, datetime(<TREND_END>))

instead of startofweek(<TIME>).

The final bucket then is the report's displayed window, every bucket is a complete 7 days, and classifier WoW == displayed WoW by construction — not by convention, and not something a future change can quietly break.

--include-partial-end and TREND_CLASS_END become obsolete and are removed. bucket-trends.js now warns if --end is omitted, because its partial-end auto-drop heuristic (if (!endArg …)) would otherwise silently discard a genuinely complete final bucket — under rolling alignment a real 70% collapse could be thrown away as "looks partial".

Because 60 isn't a multiple of 7, the oldest bucket is the partial one — the safe end to be partial on — and --start drops it, leaving 8 complete weeks.

This is not "more alerts"

A/B on real data:

  • attention set went 4 → 5 keys
  • both mis-filed codes promoted to ACCELERATING
  • access_denied correctly demoted — it was actually −53.2%, a false positive the calendar window had been surfacing

Alignment adds real signal and removes phantom signal. A controlled A/B over the affected window confirmed the classifier's WoW input matched the displayed WoW on 8 of 8 sampled codes after the change, versus 0 of 8 before.

⚠️ One correction to earlier framing, worth stating plainly for reviewers: the "promoted both to ACCELERATING and demoted access_denied" result is specific to the 2026-08-01 window, and the prose now says so. On other windows the same fix produces different (still correct) label changes. The claim being made here is about input correctness, not about any one code's verdict.

Also: reconciling red pills

Adds validate-report.ps1 check 19. Scoreboard tables colour a row from its own rolling delta; the attention section is populated from the classifier's novelty verdict. Those answer different questions, so a row can legitimately be red in the table and legitimately absent from attention — but a reader who sees that mismatch unexplained concludes the report is broken.

Precedent: Passkey WebAuthN Registration shipped carrying tag-bad (−1.27 pts, worst delta in its table) directly above the words "Quiet week — 0 NEW or ACCELERATING". Both statements were true — the scenario peaks at ~732 bad-outcome devices, below the 1,000-device classification floor, so it is structurally excluded and can never appear in attention however sharply it moves.

Every tag-bad/tag-warn row must now be either promoted into attention or named in a muted .reconcile-note giving the reason, tested in order: (1) below the classification floor, (2) within its own normal band, (3) ONGOING and flat. Check 19 hard-fails an unreconciled pill.

This closes the "red pill above an empty attention section" confusion at the report level, independently of window alignment.

Stack

This is 3 of 3:

  1. OCE report: add novelty noise gate and redesign the Broker report around it, Fixes AB#3733390 #460 — novelty noise gate + Broker report redesign → master
  2. OCE report: add Authenticator app telemetry and turn the skill into a router, Fixes AB#3731627 #461 — Authenticator app report + router → PR 1's branch
  3. this PR — rolling-window alignment + check 19 → PR 2's branch

Fixes AB#3731628

The novelty classifier graded Sunday-aligned CALENDAR weeks while the report
displays a ROLLING 7-day window. The two windows drift apart by up to six days,
so anything that turned in the last ~6 days -- exactly the period an on-call
engineer cares about most -- was structurally invisible to the gate.

Measured on the 2026-08-01 run: the gate's "current" week was 07/19-07/26
against a report window of 07/25-08/01, ONE day of overlap.
authorization_pending read +63.2% in the report and -37.1% to the classifier;
expired_token +26.7% vs -51.0%. Both were filed "ONGOING -- do not re-triage"
directly beneath their own rising numbers. On the Authenticator side the same
defect surfaced as red scoreboard pills (rolling-derived) above an empty
"Needs attention" section (calendar-derived).

Fix: bucket the 60-day trend and the sparklines with
bin_at(<TIME>, 7d, datetime(<TREND_END>)) instead of startofweek(<TIME>). The
final bucket then IS the report's displayed window, every bucket is a complete
7 days, and classifier WoW == displayed WoW by construction.
--include-partial-end and TREND_CLASS_END become obsolete and are removed;
bucket-trends.js now warns if --end is omitted, because its partial-end
auto-drop heuristic would otherwise discard a genuinely complete final bucket.

Because 60 is not a multiple of 7 the OLDEST bucket is the partial one -- the
safe end to be partial on -- and --start drops it, leaving 8 complete weeks.

A/B on real data: attention went 4 -> 5 keys; both mis-filed codes promoted to
ACCELERATING; access_denied correctly DEMOTED (actually -53.2%, a former false
positive). Alignment adds real signal and removes phantom signal -- it is not
merely "more alerts".

Adds validate-report.ps1 check 19: every red/amber scoreboard pill must be
either promoted into the attention section or named in a .reconcile-note
explaining why it is not escalated (below the classification floor / within its
own normal band / ONGOING and flat). This closes the reported "red pill above an
empty attention section" confusion at the report level, independent of window
alignment.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@shahzaibj
Shahzaib (shahzaibj) requested a review from a team as a code owner August 20, 2026 01:28
@github-actions github-actions Bot changed the title OCE report: align the noise gate to the report's rolling window OCE report: align the noise gate to the report's rolling window, Fixes AB#3731628 Aug 20, 2026
@github-actions

Copy link
Copy Markdown

✅ Work item link check complete. Description contains link AB#3731628 to an Azure Boards work item.

@github-actions

Copy link
Copy Markdown

✅ Work item link check complete. Description contains link AB#3731628 to an Azure Boards work item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant