feat(adf): real module-attribution for METRICS constraint failures#246
Merged
Conversation
charter adf suggest's loadedButViolated detector previously could only correlate "some downstream command failed nearby" with a loaded module (session/time-window heuristic) -- it had no way to know which module's own constraint actually failed. Scope note / premise correction: earlier work in this area assumed validate/drift/evidence all needed module-attribution. That's wrong -- validate checks git commit trailers and drift checks blessed-stack anti-patterns in source files; neither is module-shaped. Only adf-evidence's METRICS ceiling checks (and serve.ts's getProjectState, left untouched since it already groups output by module) actually check per-module ADF constraints. This closes that gap only. Changes (all additive, no breaking changes): - ConstraintResult gains an optional `module` field; validateConstraints takes an optional third `module` param to stamp it. - BundleResult gains an optional `documents` map (module path -> parsed doc), populated by bundleModules(). evaluateEvidence validates each module's document separately when available, attributing constraints to their module -- falling back to the old merged-document validation when `documents` is absent, so existing BundleResult fixtures/tests are unaffected. - New AdfConstraintEvent telemetry type + recordAdfConstraintEvents(): records one event per failing, module-attributed constraint. Wired into `charter adf evidence`. - adf-suggest's loadedButViolated now reports exactFailures (joined on module + session/time against adf.constraint fail events) alongside the existing correlatedFailures (kept as the fallback signal for non-module-shaped checks and pre-existing telemetry). Verified end-to-end with real CLI invocations (adf bundle -> adf evidence --auto-measure with a breached ceiling -> adf suggest), not just hand-written telemetry fixtures, to confirm the join actually fires on a real command sequence. packages/adf and packages/cli bumped 1.8.0 -> 1.9.0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1. exactFailures was counting joined resolution events, not distinct
constraint failures -- a module resolved 5x in one session with a
single real ceiling breach reported "5 confirmed constraint
failures" instead of 1. Fixed by attributing exactFailures directly
from the number of fail events per module (grouped once), decoupled
from how many times the module happened to be resolved.
2. evaluateEvidence used a bare truthiness check on bundle.documents,
so a present-but-incomplete `documents: {}` would silently validate
zero constraints (hiding real violations) instead of falling back
to merged-document validation. Fixed with a hasCompleteDocuments
guard that confirms every resolved module has a matching document.
3. Constraint aggregation (weightSummary/failCount/warnCount) was
duplicated between validator.ts and evidence.ts's per-module path,
and computeWeightSummary ran twice on the same document in the
merged-document fallback. Extracted computeConstraints() as the
shared per-entry logic; validateConstraints wraps it with the
aggregate fields, evidence.ts uses the lighter helper directly.
4. bundler.ts maintained two parallel structures (a documents array
and a documentsByPath map) built from the same loop. Removed the
array; mergeDocuments now takes Object.values(documentsByPath).
5. adf-suggest.ts's loadedButViolated was built from three separate
Maps plus a Set-union; consolidated into one Map<string,
LoadedModuleStats>. Also added a shared groupBy() helper, used by
both the new failedConstraintEventsByModule grouping and the
pre-existing co-occurrence byModule grouping in buildEmitOps.
Added regression tests for (1) and (2). 497/497 tests pass, full
workspace build/typecheck clean. Verified (1) live with a real CLI
sequence: 5x `adf bundle` + 1 real ceiling breach now reports
occurrences: 5, exactFailures: 1 (previously would have reported 5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Member
Author
|
Addressed all 5 findings from the code review:
Added regression tests for #1 and #2. 497/497 tests pass, full workspace build/typecheck clean. |
stackbilt-admin
added a commit
that referenced
this pull request
Jul 5, 2026
…249) * chore: unify workspace packages at 1.9.0, cut CHANGELOG for release adf/cli were already at 1.9.0 in-repo (#246) with everything else drifted across 1.7.0/1.8.0; the publish gate requires one version == tag. No 1.8.0 was ever published — scaffold-core ships 1.7.0 -> 1.9.0 on npm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(scaffold-core): assert semver shape, not a hardcoded release version The exact version is release-managed (unified workspace gate in release.yml verifies every package == tag); a hardcoded string here breaks on every bump. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Kurt Overmier <kurt@stackbilt.dev> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements real module-attribution for
charter adf suggest'sloadedButViolateddetector, replacing/augmenting its previous session/time-window correlation heuristic with exact evidence for METRICS ceiling failures.Scope correction first: earlier work in this thread assumed
validate/drift/adf-evidenceall needed module-attribution. That premise was wrong —validate.tschecks git commit governance trailers anddrift.tschecks blessed-stack anti-patterns in source files; neither is module-shaped, so there's nothing to attribute there. Onlyadf-evidence.ts's METRICS ceiling checks (andserve.ts'sgetProjectState, left untouched since it already groups its own output by module) actually check per-module ADF constraints. This PR closes that gap only — verified viaadvisorreview before implementation.ConstraintResultgains an optionalmodulefield;validateConstraintstakes an optional thirdmoduleparam.BundleResultgains an optionaldocumentsmap (module path → parsed doc);evaluateEvidencevalidates per-module when available, attributing constraints back to their module, falling back to the old merged-document (unattributed) path whendocumentsis absent — so hand-builtBundleResultfixtures in existing tests are untouched.AdfConstraintEventtelemetry type +recordAdfConstraintEvents(): one event per failing, module-attributed constraint, written bycharter adf evidence.adf suggest'sloadedButViolatednow reportsexactFailures(real evidence, joined by module + session/time) alongside the existingcorrelatedFailures(kept as the fallback for non-module-shaped checks and pre-existing telemetry).packages/adfandpackages/clibumped 1.8.0 → 1.9.0 (additive minor).All changes are additive — no public API removed or renamed, no breaking changes to existing
BundleResult/ConstraintResultconsumers.Test plan
pnpm build/pnpm typecheckclean across the workspacepackages/adf+packages/cli), including new coverage: per-module attribution + merged-doc fallback (evidence.test.ts),documentspopulation (bundler.test.ts), exact-vs-correlated join logic (adf-suggest.test.ts), and theadf.constraintevent write/skip paths (evidence-contracts.test.ts)adf bundle --task ...→adf evidence --auto-measure(breached ceiling) →adf suggestsequence in a scratch repo withCHARTER_SESSION_IDset and confirmedexactFailures: 1on the real offending moduleadf evidencegate passed cleanly during commit (dogfooding the exact code path this PR changes)🤖 Generated with Claude Code