diff --git a/CHANGELOG.md b/CHANGELOG.md index ca64324..bcac63c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -124,6 +124,12 @@ repository still gets a decision, never by following the link; no release carrie - :memo: docs(openspec): amend REQ-EX-S07-03/04 to match the measured implementation - :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S07 lane follow-ups - :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S07 lane +- :memo: docs(walkthrough): pin the walkthrough to measured multi-format CLI output (EX-S09) +- :memo: docs(walkthrough): close review gaps in the EX-S09 pins and repo snapshot +- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane +- :memo: docs(walkthrough): fix Step 3 lede self-contradiction on .tf format count +- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane follow-ups +- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 F1 fix ### Features - :sparkles: feat(docs): gate example pack and format claims against dogfood @@ -140,6 +146,7 @@ repository still gets a decision, never by following the link; no release carrie - :bug: specs(p5-ex): make S10 schema freeze and S07 fence non-vacuous - :bug: fix(docs-gates): fail on unmapped format tokens and de-confound the tests-dir mutation (EX-S01 review) - :bug: fix(examples): make privilege-tier's proving fixture load-bearing +- :bug: fix(docs): pin case+decision pairs, not just case names, in the walkthrough (F1) ### Refactoring - :recycle: refactor(examples): discover dogfood packs instead of a hardcoded 3-name loop diff --git a/docs/usage/walkthrough.md b/docs/usage/walkthrough.md index 344f7ed..5ec11fc 100644 --- a/docs/usage/walkthrough.md +++ b/docs/usage/walkthrough.md @@ -10,18 +10,34 @@ ## The repo -`topic-registry` — one YAML file per Kafka topic: +`topic-registry` — one YAML file per Kafka topic, map-at-root keyed by topic name. +Trimmed from a real shipped fixture +(`examples/packs/topic-registry/.assent/tests/topics/wildcard-grant/head/topics/prod/billing.invoices.v1.yaml`): ```yaml -# topics/prod/orders.yaml -name: orders -owner: team-orders -partitions: 12 -retentionMs: 604800000 +# topics/prod/billing.invoices.v1.yaml +billing.invoices.v1: + owner: billing-team + partitions: 6 + replication_factor: 3 + retention_hours: 168 + schema: + format: avro + subject: billing.invoices.v1-value + compatibility: BACKWARD + acl: + grants: + billing-team: read + finance-team: read ``` Today every MR waits for a platform engineer. Goal: routine changes merge themselves. +`partitions`, `replication_factor` and `retention_hours` are scalar fields; `acl.grants` +and (on other topics) `consumers` are nested maps. The starter packs also govern nested +JSON objects, nested tfvars maps, and — opaquely, falling back to REVIEW — HCL `.tf` +blocks. Step 3 below is real `assent test` output across all four. + ## Step 1 — scaffold a policy tree > **Planned — `assent init` does not exist.** There is no scaffolding subcommand in the @@ -35,10 +51,12 @@ assent lint . ``` That gives you `.assent/config.yaml` (environments prod/dev by path, class -`kafka-topic`), `.assent/bindings.yaml`, `.assent/packs/topics/` (ownership, -bounded-change, non-destructive, schema-valid) and `.assent/tests/topics/` with a passing -fixture for every rule — `assent lint` rejects a rule with no test case, so the two ship -together by construction. +`kafka-topic`), `.assent/bindings.yaml`, `.assent/packs/topics/rules/` — nine rule files +today (`ownership`, `bounded-change`, `non-destructive`, `schema-valid`, +`schema-compatibility`, `list-no-shrink`, `wildcard-grant`, `soft-delete`, +`referenced-resource-ownership`) — and `.assent/tests/topics/` with a passing fixture for +every rule — `assent lint` rejects a rule with no test case, so the two ship together by +construction. Committed starter packs: [`examples/packs/`](https://github.com/PlatformRelay/assent/tree/main/examples/packs) (`topic-registry`, `service-catalog`, `infra-vars`). @@ -69,22 +87,109 @@ hard errors, not warnings. $ assent test . PASS topics/bounded-change (APPROVE) PASS topics/bounded-change/negative (REVIEW) +PASS topics/list-no-shrink (APPROVE) +PASS topics/list-no-shrink/negative (REVIEW) PASS topics/ownership (APPROVE) PASS topics/ownership/negative (REVIEW) +PASS topics/quota-ceiling (APPROVE) +PASS topics/quota-ceiling/facts-omitted (REVIEW) +PASS topics/quota-ceiling/negative (REVIEW) +PASS topics/resource-ownership (APPROVE) +PASS topics/resource-ownership/negative (REVIEW) +PASS topics/schema-compatibility (APPROVE) +PASS topics/schema-compatibility/negative (REVIEW) PASS topics/schema-valid (APPROVE) PASS topics/schema-valid/negative (BLOCK) +PASS topics/soft-delete (APPROVE) +PASS topics/soft-delete/negative (REVIEW) +PASS topics/soft-delete/unrelated-modify (APPROVE) +PASS topics/wildcard-grant (APPROVE) +PASS topics/wildcard-grant/negative (BLOCK) PASS topics/non-destructive (APPROVE) PASS topics/non-destructive-delete (REVIEW) ``` -That is real output from the shipped starter pack. Each case is a directory with -`base/`, `head/`, `facts.yaml` and `expect.yaml` (or an inline `cases.yaml` entry); a +That is real output from the shipped `topic-registry` starter pack — **YAML** with nested +`acl.grants` and keyed `consumers` maps, not a single flat field. Each case is a directory +with `base/`, `head/`, `facts.yaml` and `expect.yaml` (or an inline `cases.yaml` entry); a failing case prints the expected and actual decision plus the findings that differ. `--update` rewrites expectations from the produced actuals (refused when `CI` is set), `--coverage` is the read-only both-polarity completeness gate. Exit `0` every case matched; `1` a mismatch, write or load error; `2` usage, discovery, or the CI guard refusing `--update`. +`topic-registry` is one of **four** governed formats — three of them structurally +diffed, with `.tf` governed only as an opaque whole (below): + +```console +$ assent test examples/packs/service-catalog +PASS catalog/allowed-fields (APPROVE) +PASS catalog/allowed-fields/negative (REVIEW) +PASS catalog/context-fresh (APPROVE) +PASS catalog/context-fresh/negative (REVIEW) +PASS catalog/nested-fields (APPROVE) +PASS catalog/nested-fields/negative (REVIEW) +PASS catalog/non-destructive (APPROVE) +PASS catalog/non-destructive/negative (REVIEW) +PASS catalog/ownership (APPROVE) +PASS catalog/ownership/negative (REVIEW) +PASS catalog/privilege-tier (REVIEW) +PASS catalog/privilege-tier/negative (REVIEW) +PASS catalog/schema-valid (APPROVE) +PASS catalog/schema-valid/negative (BLOCK) +PASS catalog/unkeyed-list-opaque (REVIEW) +PASS catalog/file-non-destructive (APPROVE) +PASS catalog/file-non-destructive-delete (BLOCK) +``` + +`service-catalog` is **JSON** with nested objects (`catalog/nested-fields`) and a tier +allow-list (`catalog/privilege-tier`). `catalog/unkeyed-list-opaque` is a measured, not +wished-for, limit: an *unkeyed* list is opaque to the differ by design (D-061), so a +change inside one falls back to REVIEW rather than a false-precision partial diff. + +```console +$ assent test examples/packs/infra-vars +PASS vars/bounded-change (APPROVE) +PASS vars/bounded-change/negative (REVIEW) +PASS vars/max-replicas-change (APPROVE) +PASS vars/max-replicas-change/negative (REVIEW) +PASS vars/min-replicas-change (APPROVE) +PASS vars/min-replicas-change/negative (REVIEW) +PASS vars/nested-map-change (APPROVE) +PASS vars/nested-map-change/negative (REVIEW) +PASS vars/ownership (APPROVE) +PASS vars/ownership/negative (REVIEW) +PASS vars/placement (APPROVE) +PASS vars/placement/negative (REVIEW) +PASS vars/tf-opaque (REVIEW) +PASS vars/companion-delete (REVIEW) +``` + +`infra-vars` is **tfvars** — keyed `workloads.*` maps, including the deeper +`vars/nested-map-change` case. `vars/tf-opaque` and `vars/companion-delete` are the two +honest edges of the current differ, both pinned as expected **REVIEW**, never a silent +APPROVE: + +- **`.tf` is governed but not structurally diffed.** The differ only routes the + `.tfvars` extension to the HCL parser; a `.tf` file's content — blocks or bare literals + alike — is opaque and falls back to REVIEW, never a partial parse. Assent does **not** + understand Terraform expressions or resource blocks; it treats a whole changed `.tf` + file as one un-provable unit. This is a permanent v1 limitation, not a bug — see + `examples/README.md`. +- **A companion file outside the pack's class match** (e.g. a `NOTES.md` next to the + `*.tfvars` files) deleted alongside a real change is caught only by the class-agnostic + unmatched-whole-file-delete fail-safe — REVIEW, no obligation attached. v1 does not + correlate "delete A and append B" across files; that is out of engine scope today. + +These packs also carry the **REF-EX C1–C8** governance patterns: keyed-map entry removal +(C1, `topics/list-no-shrink`), a tier allow-list (C2, `catalog/privilege-tier`), +wildcard-grant blocking (C3, `topics/wildcard-grant`), soft-delete-as-field-add (C4, +`topics/soft-delete`), a fact-derived quota ceiling (C5, `topics/quota-ceiling`), a +placement allow-list (C6, `vars/placement`), referenced-resource ownership (C7, +`topics/resource-ownership`), and the companion-file-delete REVIEW above (C8, +`vars/companion-delete`) — all runnable today from +[`examples/packs/`](https://github.com/PlatformRelay/assent/tree/main/examples/packs). + ## Step 4 — backtest before trusting it > **Planned — `assent scan` and `assent stats` do not exist.** There is no historical diff --git a/hack/docs/truthlag_pins_test.sh b/hack/docs/truthlag_pins_test.sh index 2297506..b39d489 100755 --- a/hack/docs/truthlag_pins_test.sh +++ b/hack/docs/truthlag_pins_test.sh @@ -228,6 +228,136 @@ else fail "EX-S01: docs-gates does not run hack/docs/example_format_inventory_test.sh" fi +# --- REQ-EX-S09-01 — every (case name, decision) pair the walkthrough shows is real ------- +# +# The walkthrough's console blocks are supposed to be COPIED from a real `assent test` +# run, not invented. This builds the binary (same pattern as hack/dogfood-examples.sh / +# readme_smoke_test.sh) and runs it against every example pack, then diffs the +# `name (DECISION)` pairs against every `PASS ()` line the walkthrough +# shows. NAME ALONE IS NOT ENOUGH: an earlier version of this pin extracted only the +# case name (awk '{print $2}'), so a doc that renamed `PASS vars/tf-opaque (REVIEW)` to +# `(APPROVE)` stayed green — the walkthrough's own prose claims tf-opaque and +# companion-delete are "pinned as expected REVIEW, never a silent APPROVE", and the pin +# did not actually check that. Comparing the full pair closes it, and generalizes past +# just those two cases: ANY decision flip on ANY shown case now reddens. The comparison +# is bidirectional (comm -23 AND comm -13): a real case silently missing from a console +# block that claims to be a complete pack run is drift too, not just an invented one. +WT=docs/usage/walkthrough.md +BIN="${ASSENT_BIN:-bin/assent}" +if [[ ! -x "$BIN" ]]; then + echo "== building $BIN for EX-S09 console pins ==" + CGO_ENABLED=0 go build -o "$BIN" ./cmd/assent +fi + +REAL_CASES="$(mktemp)" +WT_CASES="$(mktemp)" +trap 'rm -f "$MIRROR_EXPECTED" "$REAL_CASES" "$WT_CASES"' EXIT + +for pack in topic-registry service-catalog infra-vars; do + "$BIN" test "examples/packs/$pack" 2>/dev/null +done | awk '{print $2, $3}' | LC_ALL=C sort -u > "$REAL_CASES" + +grep -oE '^PASS [A-Za-z0-9/_-]+ \([A-Z]+\)' "$WT" | awk '{print $2, $3}' | LC_ALL=C sort -u > "$WT_CASES" + +wt_case_count="$(wc -l < "$WT_CASES" | tr -d ' ')" +if [[ "$wt_case_count" -eq 0 ]]; then + fail "REQ-EX-S09-01: $WT has no 'PASS ()' console lines to pin — vacuous" +else + missing_cases="$(comm -23 "$WT_CASES" "$REAL_CASES")" + extra_cases="$(comm -13 "$WT_CASES" "$REAL_CASES")" + if [[ -n "$missing_cases" ]]; then + fail "REQ-EX-S09-01: $WT shows case/decision pair(s) absent from real 'assent test' output (invented case, or a decision that doesn't match the real run):" + printf ' %s\n' "$missing_cases" >&2 + elif [[ -n "$extra_cases" ]]; then + fail "REQ-EX-S09-01: real 'assent test' output has case(s) the walkthrough's console blocks omit — a block claiming to be a complete run is now incomplete:" + printf ' %s\n' "$extra_cases" >&2 + else + pass "REQ-EX-S09-01: all $wt_case_count walkthrough console (case, decision) pairs are real, and no real case is missing" + fi +fi + +# --- REQ-EX-S09-02 — the walkthrough names all four governed formats, honestly ------------ +# +# Before EX-S09 the walkthrough's only console block was the topic-registry pack's +# original 8-line output; there was no evidence any other format was governed at all. +# Pin each pack's namespace AND the two hardest cases (the `.tf` opaque fallback and the +# C8 companion-file-delete REVIEW) by name, plus the four format words, so a revert to +# the old one-format sketch reddens on more than one axis. +declare -A ex_s09_prefix_hint=( + [topics/]="YAML (topic-registry)" + [catalog/]="JSON (service-catalog)" + [vars/]="tfvars (infra-vars)" +) +for prefix in "${!ex_s09_prefix_hint[@]}"; do + if grep -q "PASS ${prefix}" "$WT"; then + pass "REQ-EX-S09-02: $WT shows a real ${ex_s09_prefix_hint[$prefix]} case" + else + fail "REQ-EX-S09-02: $WT has no '${prefix}' case — missing ${ex_s09_prefix_hint[$prefix]} coverage" + fi +done + +for must_case in "vars/tf-opaque" "vars/companion-delete"; do + if grep -q "PASS $must_case" "$WT"; then + pass "REQ-EX-S09-02/03: $WT shows the $must_case case" + else + fail "REQ-EX-S09-02/03: $WT is missing the $must_case case (the .tf opaque / C8 REVIEW demonstrator)" + fi +done + +for word in "YAML" "JSON" "tfvars" "HCL"; do + if grep -q "$word" "$WT"; then + pass "REQ-EX-S09-02: $WT names format $word" + else + fail "REQ-EX-S09-02: $WT never names format $word" + fi +done + +# --- REQ-EX-S09-03 — HCL/.tf claims are honest: REVIEW/known-limitation, never "understands" - +for f in "$WT" examples/README.md; do + if grep -qiE 'assent (fully |now )?(understands|parses|supports) terraform|full terraform support|evaluates terraform expressions' "$f"; then + fail "REQ-EX-S09-03: $f overclaims Terraform/.tf support" + else + pass "REQ-EX-S09-03: no Terraform-support overclaim in $f" + fi +done + +if grep -qE '\.tf\b' "$WT"; then + # A bare "REVIEW|opaque" grep over the whole page is vacuous — those words already + # appear ~30 times in the console blocks above (every REVIEW-decision case line). + # Pin the STABLE SUBSTRING of the actual honest sentence instead, so softening or + # deleting that specific claim (not just any REVIEW/opaque word anywhere) reddens. + if grep -q 'opaque and falls back to REVIEW, never a partial parse' "$WT"; then + pass "REQ-EX-S09-03: $WT carries the measured .tf opaque/REVIEW sentence" + else + fail "REQ-EX-S09-03: $WT mentions .tf but lost the opaque/REVIEW-never-partial-parse sentence" + fi +else + fail "REQ-EX-S09-03: $WT no longer mentions .tf at all — the honesty pin would be vacuous" +fi + +# --- Step 1's rule-file inventory must match the shipped tree (same truth-lag class as +# REQ-EX-S09-01/02: a stale parenthetical rule list is exactly the kind of drift this +# story exists to kill, even though it is prose rather than a console block) ----------- +RULE_DIR="examples/packs/topic-registry/.assent/packs/topics/rules" +real_rule_count=0 +missing_rule_files=0 +for rf in "$RULE_DIR"/*.yaml; do + [[ -f "$rf" ]] || continue + real_rule_count=$((real_rule_count + 1)) + base="$(basename "$rf" .yaml)" + if ! grep -q "\`$base\`" "$WT"; then + echo " $WT Step 1 does not name rule file \`$base\`" >&2 + missing_rule_files=$((missing_rule_files + 1)) + fi +done +if [[ "$real_rule_count" -eq 0 ]]; then + fail "EX-S09: $RULE_DIR has no rule files — the Step 1 inventory pin would be vacuous" +elif [[ "$missing_rule_files" -ne 0 ]]; then + fail "EX-S09: $WT's Step 1 rule-file list is missing $missing_rule_files real rule file name(s)" +else + pass "EX-S09: $WT Step 1 names all $real_rule_count real topic-registry rule files" +fi + if [[ "$fails" -ne 0 ]]; then echo "FAILED: $fails truth-lag pin(s) reopened" >&2 exit 1