Skip to content

feat(adf): real module-attribution for METRICS constraint failures#246

Merged
stackbilt-admin merged 2 commits into
mainfrom
feat/adf-real-module-attribution
Jul 4, 2026
Merged

feat(adf): real module-attribution for METRICS constraint failures#246
stackbilt-admin merged 2 commits into
mainfrom
feat/adf-real-module-attribution

Conversation

@stackbilt-admin

Copy link
Copy Markdown
Member

Summary

Implements real module-attribution for charter adf suggest's loadedButViolated detector, replacing/augmenting its previous session/time-window correlation heuristic with exact evidence for METRICS ceiling failures.

Scope correction first: earlier work in this thread assumed validate/drift/adf-evidence all needed module-attribution. That premise was wrong — validate.ts checks git commit governance trailers and drift.ts checks blessed-stack anti-patterns in source files; neither is module-shaped, so there's nothing to attribute there. Only adf-evidence.ts's METRICS ceiling checks (and serve.ts's getProjectState, left untouched since it already groups its own output by module) actually check per-module ADF constraints. This PR closes that gap only — verified via advisor review before implementation.

  • ConstraintResult gains an optional module field; validateConstraints takes an optional third module param.
  • BundleResult gains an optional documents map (module path → parsed doc); evaluateEvidence validates per-module when available, attributing constraints back to their module, falling back to the old merged-document (unattributed) path when documents is absent — so hand-built BundleResult fixtures in existing tests are untouched.
  • New AdfConstraintEvent telemetry type + recordAdfConstraintEvents(): one event per failing, module-attributed constraint, written by charter adf evidence.
  • adf suggest's loadedButViolated now reports exactFailures (real evidence, joined by module + session/time) alongside the existing correlatedFailures (kept as the fallback for non-module-shaped checks and pre-existing telemetry).
  • packages/adf and packages/cli bumped 1.8.0 → 1.9.0 (additive minor).

All changes are additive — no public API removed or renamed, no breaking changes to existing BundleResult/ConstraintResult consumers.

Test plan

  • pnpm build / pnpm typecheck clean across the workspace
  • 494/494 tests pass (packages/adf + packages/cli), including new coverage: per-module attribution + merged-doc fallback (evidence.test.ts), documents population (bundler.test.ts), exact-vs-correlated join logic (adf-suggest.test.ts), and the adf.constraint event write/skip paths (evidence-contracts.test.ts)
  • Verified live (not just fixtures): ran a real adf bundle --task ...adf evidence --auto-measure (breached ceiling) → adf suggest sequence in a scratch repo with CHARTER_SESSION_ID set and confirmed exactFailures: 1 on the real offending module
  • This repo's own pre-commit adf evidence gate passed cleanly during commit (dogfooding the exact code path this PR changes)

🤖 Generated with Claude Code

Kurt Overmier and others added 2 commits July 4, 2026 04:26
charter adf suggest's loadedButViolated detector previously could only
correlate "some downstream command failed nearby" with a loaded module
(session/time-window heuristic) -- it had no way to know which module's
own constraint actually failed.

Scope note / premise correction: earlier work in this area assumed
validate/drift/evidence all needed module-attribution. That's wrong --
validate checks git commit trailers and drift checks blessed-stack
anti-patterns in source files; neither is module-shaped. Only
adf-evidence's METRICS ceiling checks (and serve.ts's getProjectState,
left untouched since it already groups output by module) actually check
per-module ADF constraints. This closes that gap only.

Changes (all additive, no breaking changes):
- ConstraintResult gains an optional `module` field; validateConstraints
  takes an optional third `module` param to stamp it.
- BundleResult gains an optional `documents` map (module path -> parsed
  doc), populated by bundleModules(). evaluateEvidence validates each
  module's document separately when available, attributing constraints
  to their module -- falling back to the old merged-document validation
  when `documents` is absent, so existing BundleResult fixtures/tests
  are unaffected.
- New AdfConstraintEvent telemetry type + recordAdfConstraintEvents():
  records one event per failing, module-attributed constraint. Wired
  into `charter adf evidence`.
- adf-suggest's loadedButViolated now reports exactFailures (joined on
  module + session/time against adf.constraint fail events) alongside
  the existing correlatedFailures (kept as the fallback signal for
  non-module-shaped checks and pre-existing telemetry).

Verified end-to-end with real CLI invocations (adf bundle -> adf
evidence --auto-measure with a breached ceiling -> adf suggest), not
just hand-written telemetry fixtures, to confirm the join actually
fires on a real command sequence.

packages/adf and packages/cli bumped 1.8.0 -> 1.9.0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1. exactFailures was counting joined resolution events, not distinct
   constraint failures -- a module resolved 5x in one session with a
   single real ceiling breach reported "5 confirmed constraint
   failures" instead of 1. Fixed by attributing exactFailures directly
   from the number of fail events per module (grouped once), decoupled
   from how many times the module happened to be resolved.

2. evaluateEvidence used a bare truthiness check on bundle.documents,
   so a present-but-incomplete `documents: {}` would silently validate
   zero constraints (hiding real violations) instead of falling back
   to merged-document validation. Fixed with a hasCompleteDocuments
   guard that confirms every resolved module has a matching document.

3. Constraint aggregation (weightSummary/failCount/warnCount) was
   duplicated between validator.ts and evidence.ts's per-module path,
   and computeWeightSummary ran twice on the same document in the
   merged-document fallback. Extracted computeConstraints() as the
   shared per-entry logic; validateConstraints wraps it with the
   aggregate fields, evidence.ts uses the lighter helper directly.

4. bundler.ts maintained two parallel structures (a documents array
   and a documentsByPath map) built from the same loop. Removed the
   array; mergeDocuments now takes Object.values(documentsByPath).

5. adf-suggest.ts's loadedButViolated was built from three separate
   Maps plus a Set-union; consolidated into one Map<string,
   LoadedModuleStats>. Also added a shared groupBy() helper, used by
   both the new failedConstraintEventsByModule grouping and the
   pre-existing co-occurrence byModule grouping in buildEmitOps.

Added regression tests for (1) and (2). 497/497 tests pass, full
workspace build/typecheck clean. Verified (1) live with a real CLI
sequence: 5x `adf bundle` + 1 real ceiling breach now reports
occurrences: 5, exactFailures: 1 (previously would have reported 5).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@stackbilt-admin

Copy link
Copy Markdown
Member Author

Addressed all 5 findings from the code review:

  1. exactFailures count inflation (fixed) — was counting joined resolution events instead of distinct constraint failures, so a module resolved 5x in one session with a single real ceiling breach reported "5 confirmed constraint failures." Now attributes exactFailures directly from the number of fail events per module, independent of resolution count. Verified live: 5x adf bundle + 1 real breach → occurrences: 5, exactFailures: 1.
  2. bundle.documents truthiness gap (fixed) — bare truthiness check treated documents: {} as "per-module data available," which would have silently produced zero constraints instead of falling back. Added a hasCompleteDocuments guard that confirms every resolved module has a matching document before taking the per-module path.
  3. Duplicated aggregation logic (fixed) — extracted computeConstraints() as the shared per-entry validation logic; validateConstraints wraps it with the aggregate fields, evidence.ts's per-module path uses the lighter helper directly instead of discarding a redundant weightSummary/failCount/warnCount per module.
  4. Parallel documents/documentsByPath structures in bundler.ts (fixed) — removed the array; mergeDocuments now takes Object.values(documentsByPath).
  5. Three separate Maps + Set-union in adf-suggest.ts (fixed) — consolidated into one Map<string, LoadedModuleStats>; added a shared groupBy() helper used by both the new constraint-event grouping and the pre-existing co-occurrence grouping.

Added regression tests for #1 and #2. 497/497 tests pass, full workspace build/typecheck clean.

@stackbilt-admin stackbilt-admin merged commit 182f5ec into main Jul 4, 2026
5 checks passed
@stackbilt-admin stackbilt-admin deleted the feat/adf-real-module-attribution branch July 4, 2026 10:27
stackbilt-admin added a commit that referenced this pull request Jul 5, 2026
…249)

* chore: unify workspace packages at 1.9.0, cut CHANGELOG for release

adf/cli were already at 1.9.0 in-repo (#246) with everything else drifted
across 1.7.0/1.8.0; the publish gate requires one version == tag. No 1.8.0
was ever published — scaffold-core ships 1.7.0 -> 1.9.0 on npm.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(scaffold-core): assert semver shape, not a hardcoded release version

The exact version is release-managed (unified workspace gate in release.yml
verifies every package == tag); a hardcoded string here breaks on every bump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Kurt Overmier <kurt@stackbilt.dev>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant