Skip to content

feat: rate bug reports in maintained projects as urgent, not notable - #50

Merged
alchemydc merged 1 commit into
mainfrom
feat/urgent-bug-reports-in-maintained-projects
Aug 6, 2026
Merged

feat: rate bug reports in maintained projects as urgent, not notable#50
alchemydc merged 1 commit into
mainfrom
feat/urgent-bug-reports-in-maintained-projects

Conversation

@alchemydc

Copy link
Copy Markdown
Owner

Why

A Zebra bug report on the community forum — Zebra 6.2.3 Release, post #3: "attempted to add a banned peer addr" logged ~300k times in 24h, 61 MB/day of log bloat — was surfaced by the digest bot but labelled 🟡 notable (medium) instead of 🔴 urgent (high). Defect reports against software we maintain are exactly what the digest exists to escalate.

Three things in the rubric (pushSharedRules()) caused the miss:

  1. No such category existed. high covered security vulns, outages, releases, and open governance votes. An operator-reported functional bug fit none of them, so medium was the model's best available answer. Nothing in the repo told the model which software is ours either — the Discourse adapter is fully generic.
  2. Group-level averaging. importance is one value per topic group and the prompt asked for the group's "overall" importance. This was the release thread with the bug report as one reply among four, so the rating was diluted. (It also explains why a release thread — already high under the old rubric — came out medium.)
  3. A tie-break that fought any new rule: "When in doubt between two levels, choose the lower one."

What changed

Prompt steering plus one config variable. No changes to the Importance enum, ranking, or Slack rendering.

  • MAINTAINED_PROJECTS (new, optional, comma-separated) — keeps project names out of src/ so the Discourse adapter stays reusable. Parsed to string[] following the existing DISCORD_CHANNELS idiom.
  • Rubric (AiSdkProcessor.ts) — high now covers "bug reports, defects, regressions, crashes, or resource-exhaustion problems in software this organization maintains (…)"; medium is scoped so bug reports can't fall through; a new line forbids averaging; the tie-break carves out security issues and our-software bug reports.
  • Wired into daily-digest.yml, .env.example, README.md, the runbook, and the AGENTS.md rubric pointer.

Verification

npm run build clean, npm test 112/112. Also ran the real thread through the live model:

Topic Result
Zebra 6.2.3 Release 🔴 high — was medium
Coinholder Retroactive Grants CFP 🟡 medium
Welcome to the Zcash Forum 🟡 medium
NU7 Token Holder Vote 🔴 high (correct under the pre-existing governance rule)

The Zebra thread now splits the bug report into its own linked conversation, crediting cartesien's report and nsheep's confirmation that a fix lands in v6.3.0. The lift is targeted, not blanket.

Action required after merge

Set the MAINTAINED_PROJECTS repo Variable (e.g. Zebra,zcashd,Zallet). Without it the rule still fires but the rubric names no projects, which is weaker steering.

Reviewer notes

  • MAINTAINED_PROJECTS puts .optional() inside the preprocess. An unset ${{ vars.X }} arrives as "", which toStr maps to undefined; an outer .optional() is skipped and the inner z.string() rejects it. This is the .default() gotcha already documented in AGENTS.md — it crashed loadConfig() during testing. Several pre-existing fields have the outer-.optional() shape and the same latent crash; tracked separately, not touched here.
  • The perennial "Welcome to the Zcash Forum" thread came out medium rather than low. I did not A/B it against the old prompt, so I can't say whether the no-averaging line caused that or it was already medium — worth watching in the next real digest.
  • CI may fail for reasons unrelated to this change; GitHub is having an outage as of opening.

🤖 Generated with Claude Code

A Zebra bug report on the community forum (~300k "attempted to add a
banned peer addr" warnings in 24h, 61 MB/day of log bloat) was surfaced
by the digest but labelled notable (medium) instead of urgent (high).

Three things in the rubric caused the miss:

- No such category existed. `high` covered security vulns, outages,
  releases, and open governance votes; an operator-reported functional
  bug fit none of them, so medium was the model's best answer. Nothing
  told the model which software is ours, either.
- The rubric asked for the group's "overall" importance, so a single
  high-signal bug report inside a busy release thread was averaged down.
- The tie-break ("when in doubt, choose the lower one") actively fought
  any new high rule.

Fix is prompt steering plus one config variable, MAINTAINED_PROJECTS,
so the project names stay out of src/ and the Discourse adapter remains
generic. The `high` bullet now covers bug reports, defects, regressions,
crashes, and resource exhaustion in software this org maintains; `medium`
is scoped so those can't fall through; a new line forbids averaging; and
the tie-break carves out security issues and our-software bug reports.

Verified against the live model: the real Zebra 6.2.3 thread now rates
high and splits the bug report into its own linked conversation, while
control threads (grants CFP, welcome thread) stay medium — a targeted
lift, not a blanket escalation.

Note MAINTAINED_PROJECTS puts `.optional()` INSIDE the preprocess. An
unset repo Variable arrives as "", which toStr maps to undefined; an
outer `.optional()` is skipped and the inner z.string() rejects it.
This is the documented `.default()` gotcha in AGENTS.md and it bit
during testing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alchemydc
alchemydc merged commit d728b74 into main Aug 6, 2026
@alchemydc
alchemydc deleted the feat/urgent-bug-reports-in-maintained-projects branch August 6, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant