feat: rate bug reports in maintained projects as urgent, not notable - #50
Merged
Merged
Conversation
A Zebra bug report on the community forum (~300k "attempted to add a
banned peer addr" warnings in 24h, 61 MB/day of log bloat) was surfaced
by the digest but labelled notable (medium) instead of urgent (high).
Three things in the rubric caused the miss:
- No such category existed. `high` covered security vulns, outages,
releases, and open governance votes; an operator-reported functional
bug fit none of them, so medium was the model's best answer. Nothing
told the model which software is ours, either.
- The rubric asked for the group's "overall" importance, so a single
high-signal bug report inside a busy release thread was averaged down.
- The tie-break ("when in doubt, choose the lower one") actively fought
any new high rule.
Fix is prompt steering plus one config variable, MAINTAINED_PROJECTS,
so the project names stay out of src/ and the Discourse adapter remains
generic. The `high` bullet now covers bug reports, defects, regressions,
crashes, and resource exhaustion in software this org maintains; `medium`
is scoped so those can't fall through; a new line forbids averaging; and
the tie-break carves out security issues and our-software bug reports.
Verified against the live model: the real Zebra 6.2.3 thread now rates
high and splits the bug report into its own linked conversation, while
control threads (grants CFP, welcome thread) stay medium — a targeted
lift, not a blanket escalation.
Note MAINTAINED_PROJECTS puts `.optional()` INSIDE the preprocess. An
unset repo Variable arrives as "", which toStr maps to undefined; an
outer `.optional()` is skipped and the inner z.string() rejects it.
This is the documented `.default()` gotcha in AGENTS.md and it bit
during testing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A Zebra bug report on the community forum — Zebra 6.2.3 Release, post #3:
"attempted to add a banned peer addr"logged ~300k times in 24h, 61 MB/day of log bloat — was surfaced by the digest bot but labelled 🟡 notable (medium) instead of 🔴 urgent (high). Defect reports against software we maintain are exactly what the digest exists to escalate.Three things in the rubric (
pushSharedRules()) caused the miss:highcovered security vulns, outages, releases, and open governance votes. An operator-reported functional bug fit none of them, somediumwas the model's best available answer. Nothing in the repo told the model which software is ours either — the Discourse adapter is fully generic.importanceis one value per topic group and the prompt asked for the group's "overall" importance. This was the release thread with the bug report as one reply among four, so the rating was diluted. (It also explains why a release thread — alreadyhighunder the old rubric — came outmedium.)What changed
Prompt steering plus one config variable. No changes to the
Importanceenum, ranking, or Slack rendering.MAINTAINED_PROJECTS(new, optional, comma-separated) — keeps project names out ofsrc/so the Discourse adapter stays reusable. Parsed tostring[]following the existingDISCORD_CHANNELSidiom.AiSdkProcessor.ts) —highnow covers "bug reports, defects, regressions, crashes, or resource-exhaustion problems in software this organization maintains (…)";mediumis scoped so bug reports can't fall through; a new line forbids averaging; the tie-break carves out security issues and our-software bug reports.daily-digest.yml,.env.example,README.md, the runbook, and theAGENTS.mdrubric pointer.Verification
npm run buildclean,npm test112/112. Also ran the real thread through the live model:mediumThe Zebra thread now splits the bug report into its own linked conversation, crediting cartesien's report and nsheep's confirmation that a fix lands in v6.3.0. The lift is targeted, not blanket.
Action required after merge
Set the
MAINTAINED_PROJECTSrepo Variable (e.g.Zebra,zcashd,Zallet). Without it the rule still fires but the rubric names no projects, which is weaker steering.Reviewer notes
MAINTAINED_PROJECTSputs.optional()inside thepreprocess. An unset${{ vars.X }}arrives as"", whichtoStrmaps toundefined; an outer.optional()is skipped and the innerz.string()rejects it. This is the.default()gotcha already documented inAGENTS.md— it crashedloadConfig()during testing. Several pre-existing fields have the outer-.optional()shape and the same latent crash; tracked separately, not touched here.mediumrather thanlow. I did not A/B it against the old prompt, so I can't say whether the no-averaging line caused that or it was alreadymedium— worth watching in the next real digest.🤖 Generated with Claude Code