From b9a64416ed08a895be30e4774f116d691e62cb6d Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Sun, 16 Aug 2026 13:33:59 +0200 Subject: [PATCH 1/7] :memo: docs(walkthrough): pin the walkthrough to measured multi-format CLI output (EX-S09) Step 3's console block was 8 lines of stale topic-registry-only output and the intro implied partitions was the sole governed field. Replace it with real, captured `assent test` output across all three shipped packs (YAML nested maps, JSON nested objects, tfvars nested maps, plus the .tf opaque and C8 companion-delete REVIEW cases), and name the REF-EX C1-C8 patterns by real case name. Extend hack/docs/truthlag_pins_test.sh (REQ-EX-S09-01..03) so every case name the walkthrough shows must exist in real assent test output, all four formats must be named, and any HCL/.tf claim must stay REVIEW/opaque/ known-limitation, never an "understands Terraform" overclaim. --- docs/usage/walkthrough.md | 95 ++++++++++++++++++++++++++++++++- hack/docs/truthlag_pins_test.sh | 93 ++++++++++++++++++++++++++++++++ 2 files changed, 186 insertions(+), 2 deletions(-) diff --git a/docs/usage/walkthrough.md b/docs/usage/walkthrough.md index 344f7ed..42339d4 100644 --- a/docs/usage/walkthrough.md +++ b/docs/usage/walkthrough.md @@ -22,6 +22,11 @@ retentionMs: 604800000 Today every MR waits for a platform engineer. Goal: routine changes merge themselves. +That flat snippet is illustrative, not the limit of what ships: the starter packs also +govern nested YAML maps (`acl.grants`, keyed `consumers`), nested JSON objects, nested +tfvars maps, and — opaquely, falling back to REVIEW — HCL `.tf` blocks. Step 3 below is +real `assent test` output across all four. + ## Step 1 — scaffold a policy tree > **Planned — `assent init` does not exist.** There is no scaffolding subcommand in the @@ -69,22 +74,108 @@ hard errors, not warnings. $ assent test . PASS topics/bounded-change (APPROVE) PASS topics/bounded-change/negative (REVIEW) +PASS topics/list-no-shrink (APPROVE) +PASS topics/list-no-shrink/negative (REVIEW) PASS topics/ownership (APPROVE) PASS topics/ownership/negative (REVIEW) +PASS topics/quota-ceiling (APPROVE) +PASS topics/quota-ceiling/facts-omitted (REVIEW) +PASS topics/quota-ceiling/negative (REVIEW) +PASS topics/resource-ownership (APPROVE) +PASS topics/resource-ownership/negative (REVIEW) +PASS topics/schema-compatibility (APPROVE) +PASS topics/schema-compatibility/negative (REVIEW) PASS topics/schema-valid (APPROVE) PASS topics/schema-valid/negative (BLOCK) +PASS topics/soft-delete (APPROVE) +PASS topics/soft-delete/negative (REVIEW) +PASS topics/soft-delete/unrelated-modify (APPROVE) +PASS topics/wildcard-grant (APPROVE) +PASS topics/wildcard-grant/negative (BLOCK) PASS topics/non-destructive (APPROVE) PASS topics/non-destructive-delete (REVIEW) ``` -That is real output from the shipped starter pack. Each case is a directory with -`base/`, `head/`, `facts.yaml` and `expect.yaml` (or an inline `cases.yaml` entry); a +That is real output from the shipped `topic-registry` starter pack — **YAML** with nested +`acl.grants` and keyed `consumers` maps, not a single flat field. Each case is a directory +with `base/`, `head/`, `facts.yaml` and `expect.yaml` (or an inline `cases.yaml` entry); a failing case prints the expected and actual decision plus the findings that differ. `--update` rewrites expectations from the produced actuals (refused when `CI` is set), `--coverage` is the read-only both-polarity completeness gate. Exit `0` every case matched; `1` a mismatch, write or load error; `2` usage, discovery, or the CI guard refusing `--update`. +`topic-registry` is one of **four** formats the shipped packs govern nested structure in: + +```console +$ assent test examples/packs/service-catalog +PASS catalog/allowed-fields (APPROVE) +PASS catalog/allowed-fields/negative (REVIEW) +PASS catalog/context-fresh (APPROVE) +PASS catalog/context-fresh/negative (REVIEW) +PASS catalog/nested-fields (APPROVE) +PASS catalog/nested-fields/negative (REVIEW) +PASS catalog/non-destructive (APPROVE) +PASS catalog/non-destructive/negative (REVIEW) +PASS catalog/ownership (APPROVE) +PASS catalog/ownership/negative (REVIEW) +PASS catalog/privilege-tier (REVIEW) +PASS catalog/privilege-tier/negative (REVIEW) +PASS catalog/schema-valid (APPROVE) +PASS catalog/schema-valid/negative (BLOCK) +PASS catalog/unkeyed-list-opaque (REVIEW) +PASS catalog/file-non-destructive (APPROVE) +PASS catalog/file-non-destructive-delete (BLOCK) +``` + +`service-catalog` is **JSON** with nested objects (`catalog/nested-fields`) and a tier +allow-list (`catalog/privilege-tier`). `catalog/unkeyed-list-opaque` is a measured, not +wished-for, limit: an *unkeyed* list is opaque to the differ by design (D-061), so a +change inside one falls back to REVIEW rather than a false-precision partial diff. + +```console +$ assent test examples/packs/infra-vars +PASS vars/bounded-change (APPROVE) +PASS vars/bounded-change/negative (REVIEW) +PASS vars/max-replicas-change (APPROVE) +PASS vars/max-replicas-change/negative (REVIEW) +PASS vars/min-replicas-change (APPROVE) +PASS vars/min-replicas-change/negative (REVIEW) +PASS vars/nested-map-change (APPROVE) +PASS vars/nested-map-change/negative (REVIEW) +PASS vars/ownership (APPROVE) +PASS vars/ownership/negative (REVIEW) +PASS vars/placement (APPROVE) +PASS vars/placement/negative (REVIEW) +PASS vars/tf-opaque (REVIEW) +PASS vars/companion-delete (REVIEW) +``` + +`infra-vars` is **tfvars** — keyed `workloads.*` maps, including the deeper +`vars/nested-map-change` case. `vars/tf-opaque` and `vars/companion-delete` are the two +honest edges of the current differ, both pinned as expected **REVIEW**, never a silent +APPROVE: + +- **`.tf` is governed but not structurally diffed.** The differ only routes the + `.tfvars` extension to the HCL parser; a `.tf` file's content — blocks or bare literals + alike — is opaque and falls back to REVIEW, never a partial parse. Assent does **not** + understand Terraform expressions or resource blocks; it treats a whole changed `.tf` + file as one un-provable unit. This is a permanent v1 limitation, not a bug — see + `examples/README.md`. +- **A companion file outside the pack's class match** (e.g. a `NOTES.md` next to the + `*.tfvars` files) deleted alongside a real change is caught only by the class-agnostic + unmatched-whole-file-delete fail-safe — REVIEW, no obligation attached. v1 does not + correlate "delete A and append B" across files; that is out of engine scope today. + +These packs also carry the **REF-EX C1–C8** governance patterns: keyed-map entry removal +(C1, `topics/list-no-shrink`), a tier allow-list (C2, `catalog/privilege-tier`), +wildcard-grant blocking (C3, `topics/wildcard-grant`), soft-delete-as-field-add (C4, +`topics/soft-delete`), a fact-derived quota ceiling (C5, `topics/quota-ceiling`), a +placement allow-list (C6, `vars/placement`), referenced-resource ownership (C7, +`topics/resource-ownership`), and the companion-file-delete REVIEW above (C8, +`vars/companion-delete`) — all runnable today from +[`examples/packs/`](https://github.com/PlatformRelay/assent/tree/main/examples/packs). + ## Step 4 — backtest before trusting it > **Planned — `assent scan` and `assent stats` do not exist.** There is no historical diff --git a/hack/docs/truthlag_pins_test.sh b/hack/docs/truthlag_pins_test.sh index 2297506..92bdb4a 100755 --- a/hack/docs/truthlag_pins_test.sh +++ b/hack/docs/truthlag_pins_test.sh @@ -228,6 +228,99 @@ else fail "EX-S01: docs-gates does not run hack/docs/example_format_inventory_test.sh" fi +# --- REQ-EX-S09-01 — every case name the walkthrough shows in console output is real ------ +# +# The walkthrough's console blocks are supposed to be COPIED from a real `assent test` +# run, not invented. This builds the binary (same pattern as hack/dogfood-examples.sh / +# readme_smoke_test.sh) and runs it against every example pack, then diffs the case +# names against every `PASS ` line the walkthrough shows. A hand-typed case name +# that doesn't exist in the corpus (or a real case renamed without updating the doc) +# reddens here, not just at eyeball-review time. +WT=docs/usage/walkthrough.md +BIN="${ASSENT_BIN:-bin/assent}" +if [[ ! -x "$BIN" ]]; then + echo "== building $BIN for EX-S09 console pins ==" + CGO_ENABLED=0 go build -o "$BIN" ./cmd/assent +fi + +REAL_CASES="$(mktemp)" +WT_CASES="$(mktemp)" +trap 'rm -f "$MIRROR_EXPECTED" "$REAL_CASES" "$WT_CASES"' EXIT + +for pack in topic-registry service-catalog infra-vars; do + "$BIN" test "examples/packs/$pack" 2>/dev/null +done | awk '{print $2}' | LC_ALL=C sort -u > "$REAL_CASES" + +grep -oE '^PASS [A-Za-z0-9/_-]+' "$WT" | awk '{print $2}' | LC_ALL=C sort -u > "$WT_CASES" + +wt_case_count="$(wc -l < "$WT_CASES" | tr -d ' ')" +if [[ "$wt_case_count" -eq 0 ]]; then + fail "REQ-EX-S09-01: $WT has no 'PASS ' console lines to pin — vacuous" +else + missing_cases="$(comm -23 "$WT_CASES" "$REAL_CASES")" + if [[ -n "$missing_cases" ]]; then + fail "REQ-EX-S09-01: $WT shows case name(s) absent from real 'assent test' output:" + printf ' %s\n' "$missing_cases" >&2 + else + pass "REQ-EX-S09-01: all $wt_case_count walkthrough console case names are real" + fi +fi + +# --- REQ-EX-S09-02 — the walkthrough names all four governed formats, honestly ------------ +# +# Before EX-S09 the walkthrough's only console block was the topic-registry pack's +# original 8-line output; there was no evidence any other format was governed at all. +# Pin each pack's namespace AND the two hardest cases (the `.tf` opaque fallback and the +# C8 companion-file-delete REVIEW) by name, plus the four format words, so a revert to +# the old one-format sketch reddens on more than one axis. +declare -A ex_s09_prefix_hint=( + [topics/]="YAML (topic-registry)" + [catalog/]="JSON (service-catalog)" + [vars/]="tfvars (infra-vars)" +) +for prefix in "${!ex_s09_prefix_hint[@]}"; do + if grep -q "PASS ${prefix}" "$WT"; then + pass "REQ-EX-S09-02: $WT shows a real ${ex_s09_prefix_hint[$prefix]} case" + else + fail "REQ-EX-S09-02: $WT has no '${prefix}' case — missing ${ex_s09_prefix_hint[$prefix]} coverage" + fi +done + +for must_case in "vars/tf-opaque" "vars/companion-delete"; do + if grep -q "PASS $must_case" "$WT"; then + pass "REQ-EX-S09-02/03: $WT shows the $must_case case" + else + fail "REQ-EX-S09-02/03: $WT is missing the $must_case case (the .tf opaque / C8 REVIEW demonstrator)" + fi +done + +for word in "YAML" "JSON" "tfvars" "HCL"; do + if grep -q "$word" "$WT"; then + pass "REQ-EX-S09-02: $WT names format $word" + else + fail "REQ-EX-S09-02: $WT never names format $word" + fi +done + +# --- REQ-EX-S09-03 — HCL/.tf claims are honest: REVIEW/known-limitation, never "understands" - +for f in "$WT" examples/README.md; do + if grep -qiE 'assent (fully |now )?(understands|parses|supports) terraform|full terraform support|evaluates terraform expressions' "$f"; then + fail "REQ-EX-S09-03: $f overclaims Terraform/.tf support" + else + pass "REQ-EX-S09-03: no Terraform-support overclaim in $f" + fi +done + +if grep -qE '\.tf\b' "$WT"; then + if grep -qiE 'REVIEW|opaque|literal-only|known limitation|permanent v1 limitation' "$WT"; then + pass "REQ-EX-S09-03: $WT's .tf mention is paired with an honest REVIEW/opaque/limitation caveat" + else + fail "REQ-EX-S09-03: $WT mentions .tf without an honest caveat" + fi +else + fail "REQ-EX-S09-03: $WT no longer mentions .tf at all — the honesty pin would be vacuous" +fi + if [[ "$fails" -ne 0 ]]; then echo "FAILED: $fails truth-lag pin(s) reopened" >&2 exit 1 From 8c2e5c58558e68c9cc9846f5b1bedbdfa54a057a Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Sun, 16 Aug 2026 13:39:48 +0200 Subject: [PATCH 2/7] :memo: docs(walkthrough): close review gaps in the EX-S09 pins and repo snapshot Three follow-up fixes on the walkthrough pinning work: (1) the intro's "one YAML file" snippet showed invented fields (name/retentionMs) that exist nowhere in the shipped corpus - replace it with a trimmed excerpt of a real fixture; (2) Step 1's rule-file parenthetical still named the pre-S06 four-rule pack while the pack now ships nine, and add a filesystem-driven pin so that list can't go stale again; (3) the .tf/REVIEW honesty pin in truthlag_pins_test.sh grepped the whole page for "REVIEW|opaque", which can never fail once Step 3's console blocks exist (~30 REVIEW lines) - replace it with a stable-substring match on the actual honest sentence, reconfirmed by mutation to now redden when the sentence is softened. Also regenerate CHANGELOG.md (drift from the prior EX-S09 commit). --- CHANGELOG.md | 1 + docs/usage/walkthrough.md | 41 ++++++++++++++++++++++----------- hack/docs/truthlag_pins_test.sh | 33 +++++++++++++++++++++++--- 3 files changed, 58 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca64324..db05681 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -124,6 +124,7 @@ repository still gets a decision, never by following the link; no release carrie - :memo: docs(openspec): amend REQ-EX-S07-03/04 to match the measured implementation - :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S07 lane follow-ups - :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S07 lane +- :memo: docs(walkthrough): pin the walkthrough to measured multi-format CLI output (EX-S09) ### Features - :sparkles: feat(docs): gate example pack and format claims against dogfood diff --git a/docs/usage/walkthrough.md b/docs/usage/walkthrough.md index 42339d4..c847ac7 100644 --- a/docs/usage/walkthrough.md +++ b/docs/usage/walkthrough.md @@ -10,22 +10,33 @@ ## The repo -`topic-registry` — one YAML file per Kafka topic: +`topic-registry` — one YAML file per Kafka topic, map-at-root keyed by topic name. +Trimmed from a real shipped fixture +(`examples/packs/topic-registry/.assent/tests/topics/wildcard-grant/head/topics/prod/billing.invoices.v1.yaml`): ```yaml -# topics/prod/orders.yaml -name: orders -owner: team-orders -partitions: 12 -retentionMs: 604800000 +# topics/prod/billing.invoices.v1.yaml +billing.invoices.v1: + owner: billing-team + partitions: 6 + replication_factor: 3 + retention_hours: 168 + schema: + format: avro + subject: billing.invoices.v1-value + compatibility: BACKWARD + acl: + grants: + billing-team: read + finance-team: read ``` Today every MR waits for a platform engineer. Goal: routine changes merge themselves. -That flat snippet is illustrative, not the limit of what ships: the starter packs also -govern nested YAML maps (`acl.grants`, keyed `consumers`), nested JSON objects, nested -tfvars maps, and — opaquely, falling back to REVIEW — HCL `.tf` blocks. Step 3 below is -real `assent test` output across all four. +`partitions`, `replication_factor` and `retention_hours` are scalar fields; `acl.grants` +and (on other topics) `consumers` are nested maps. The starter packs also govern nested +JSON objects, nested tfvars maps, and — opaquely, falling back to REVIEW — HCL `.tf` +blocks. Step 3 below is real `assent test` output across all four. ## Step 1 — scaffold a policy tree @@ -40,10 +51,12 @@ assent lint . ``` That gives you `.assent/config.yaml` (environments prod/dev by path, class -`kafka-topic`), `.assent/bindings.yaml`, `.assent/packs/topics/` (ownership, -bounded-change, non-destructive, schema-valid) and `.assent/tests/topics/` with a passing -fixture for every rule — `assent lint` rejects a rule with no test case, so the two ship -together by construction. +`kafka-topic`), `.assent/bindings.yaml`, `.assent/packs/topics/rules/` — nine rule files +today (`ownership`, `bounded-change`, `non-destructive`, `schema-valid`, +`schema-compatibility`, `list-no-shrink`, `wildcard-grant`, `soft-delete`, +`referenced-resource-ownership`) — and `.assent/tests/topics/` with a passing fixture for +every rule — `assent lint` rejects a rule with no test case, so the two ship together by +construction. Committed starter packs: [`examples/packs/`](https://github.com/PlatformRelay/assent/tree/main/examples/packs) (`topic-registry`, `service-catalog`, `infra-vars`). diff --git a/hack/docs/truthlag_pins_test.sh b/hack/docs/truthlag_pins_test.sh index 92bdb4a..9481ca1 100755 --- a/hack/docs/truthlag_pins_test.sh +++ b/hack/docs/truthlag_pins_test.sh @@ -312,15 +312,42 @@ for f in "$WT" examples/README.md; do done if grep -qE '\.tf\b' "$WT"; then - if grep -qiE 'REVIEW|opaque|literal-only|known limitation|permanent v1 limitation' "$WT"; then - pass "REQ-EX-S09-03: $WT's .tf mention is paired with an honest REVIEW/opaque/limitation caveat" + # A bare "REVIEW|opaque" grep over the whole page is vacuous — those words already + # appear ~30 times in the console blocks above (every REVIEW-decision case line). + # Pin the STABLE SUBSTRING of the actual honest sentence instead, so softening or + # deleting that specific claim (not just any REVIEW/opaque word anywhere) reddens. + if grep -q 'opaque and falls back to REVIEW, never a partial parse' "$WT"; then + pass "REQ-EX-S09-03: $WT carries the measured .tf opaque/REVIEW sentence" else - fail "REQ-EX-S09-03: $WT mentions .tf without an honest caveat" + fail "REQ-EX-S09-03: $WT mentions .tf but lost the opaque/REVIEW-never-partial-parse sentence" fi else fail "REQ-EX-S09-03: $WT no longer mentions .tf at all — the honesty pin would be vacuous" fi +# --- Step 1's rule-file inventory must match the shipped tree (same truth-lag class as +# REQ-EX-S09-01/02: a stale parenthetical rule list is exactly the kind of drift this +# story exists to kill, even though it is prose rather than a console block) ----------- +RULE_DIR="examples/packs/topic-registry/.assent/packs/topics/rules" +real_rule_count=0 +missing_rule_files=0 +for rf in "$RULE_DIR"/*.yaml; do + [[ -f "$rf" ]] || continue + real_rule_count=$((real_rule_count + 1)) + base="$(basename "$rf" .yaml)" + if ! grep -q "\`$base\`" "$WT"; then + echo " $WT Step 1 does not name rule file \`$base\`" >&2 + missing_rule_files=$((missing_rule_files + 1)) + fi +done +if [[ "$real_rule_count" -eq 0 ]]; then + fail "EX-S09: $RULE_DIR has no rule files — the Step 1 inventory pin would be vacuous" +elif [[ "$missing_rule_files" -ne 0 ]]; then + fail "EX-S09: $WT's Step 1 rule-file list is missing $missing_rule_files real rule file name(s)" +else + pass "EX-S09: $WT Step 1 names all $real_rule_count real topic-registry rule files" +fi + if [[ "$fails" -ne 0 ]]; then echo "FAILED: $fails truth-lag pin(s) reopened" >&2 exit 1 From 4e98318cc0ba81a849ec438e9b9889f34b930e91 Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Sun, 16 Aug 2026 13:42:00 +0200 Subject: [PATCH 3/7] :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index db05681..fd2f782 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -125,6 +125,7 @@ repository still gets a decision, never by following the link; no release carrie - :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S07 lane follow-ups - :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S07 lane - :memo: docs(walkthrough): pin the walkthrough to measured multi-format CLI output (EX-S09) +- :memo: docs(walkthrough): close review gaps in the EX-S09 pins and repo snapshot ### Features - :sparkles: feat(docs): gate example pack and format claims against dogfood From 248e1d56dacd20d75db89f3e93dfd8649b27f61d Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Sun, 16 Aug 2026 13:45:09 +0200 Subject: [PATCH 4/7] :memo: docs(walkthrough): fix Step 3 lede self-contradiction on .tf format count --- docs/usage/walkthrough.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/usage/walkthrough.md b/docs/usage/walkthrough.md index c847ac7..5ec11fc 100644 --- a/docs/usage/walkthrough.md +++ b/docs/usage/walkthrough.md @@ -118,7 +118,8 @@ failing case prints the expected and actual decision plus the findings that diff matched; `1` a mismatch, write or load error; `2` usage, discovery, or the CI guard refusing `--update`. -`topic-registry` is one of **four** formats the shipped packs govern nested structure in: +`topic-registry` is one of **four** governed formats — three of them structurally +diffed, with `.tf` governed only as an opaque whole (below): ```console $ assent test examples/packs/service-catalog From f4fac611c322dd763fd302ff4a950ca7e1cb837f Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Sun, 16 Aug 2026 13:45:20 +0200 Subject: [PATCH 5/7] :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane follow-ups --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index fd2f782..9f098b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,6 +126,9 @@ repository still gets a decision, never by following the link; no release carrie - :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S07 lane - :memo: docs(walkthrough): pin the walkthrough to measured multi-format CLI output (EX-S09) - :memo: docs(walkthrough): close review gaps in the EX-S09 pins and repo snapshot +- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane +- :memo: docs(walkthrough): fix Step 3 lede self-contradiction on .tf format count +- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane follow-ups ### Features - :sparkles: feat(docs): gate example pack and format claims against dogfood From 2a49bcdfb13886299877560c3c0582b9a4664b7c Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Sun, 16 Aug 2026 13:57:02 +0200 Subject: [PATCH 6/7] :bug: fix(docs): pin case+decision pairs, not just case names, in the walkthrough (F1) --- hack/docs/truthlag_pins_test.sh | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/hack/docs/truthlag_pins_test.sh b/hack/docs/truthlag_pins_test.sh index 9481ca1..b39d489 100755 --- a/hack/docs/truthlag_pins_test.sh +++ b/hack/docs/truthlag_pins_test.sh @@ -228,14 +228,20 @@ else fail "EX-S01: docs-gates does not run hack/docs/example_format_inventory_test.sh" fi -# --- REQ-EX-S09-01 — every case name the walkthrough shows in console output is real ------ +# --- REQ-EX-S09-01 — every (case name, decision) pair the walkthrough shows is real ------- # # The walkthrough's console blocks are supposed to be COPIED from a real `assent test` # run, not invented. This builds the binary (same pattern as hack/dogfood-examples.sh / -# readme_smoke_test.sh) and runs it against every example pack, then diffs the case -# names against every `PASS ` line the walkthrough shows. A hand-typed case name -# that doesn't exist in the corpus (or a real case renamed without updating the doc) -# reddens here, not just at eyeball-review time. +# readme_smoke_test.sh) and runs it against every example pack, then diffs the +# `name (DECISION)` pairs against every `PASS ()` line the walkthrough +# shows. NAME ALONE IS NOT ENOUGH: an earlier version of this pin extracted only the +# case name (awk '{print $2}'), so a doc that renamed `PASS vars/tf-opaque (REVIEW)` to +# `(APPROVE)` stayed green — the walkthrough's own prose claims tf-opaque and +# companion-delete are "pinned as expected REVIEW, never a silent APPROVE", and the pin +# did not actually check that. Comparing the full pair closes it, and generalizes past +# just those two cases: ANY decision flip on ANY shown case now reddens. The comparison +# is bidirectional (comm -23 AND comm -13): a real case silently missing from a console +# block that claims to be a complete pack run is drift too, not just an invented one. WT=docs/usage/walkthrough.md BIN="${ASSENT_BIN:-bin/assent}" if [[ ! -x "$BIN" ]]; then @@ -249,20 +255,24 @@ trap 'rm -f "$MIRROR_EXPECTED" "$REAL_CASES" "$WT_CASES"' EXIT for pack in topic-registry service-catalog infra-vars; do "$BIN" test "examples/packs/$pack" 2>/dev/null -done | awk '{print $2}' | LC_ALL=C sort -u > "$REAL_CASES" +done | awk '{print $2, $3}' | LC_ALL=C sort -u > "$REAL_CASES" -grep -oE '^PASS [A-Za-z0-9/_-]+' "$WT" | awk '{print $2}' | LC_ALL=C sort -u > "$WT_CASES" +grep -oE '^PASS [A-Za-z0-9/_-]+ \([A-Z]+\)' "$WT" | awk '{print $2, $3}' | LC_ALL=C sort -u > "$WT_CASES" wt_case_count="$(wc -l < "$WT_CASES" | tr -d ' ')" if [[ "$wt_case_count" -eq 0 ]]; then - fail "REQ-EX-S09-01: $WT has no 'PASS ' console lines to pin — vacuous" + fail "REQ-EX-S09-01: $WT has no 'PASS ()' console lines to pin — vacuous" else missing_cases="$(comm -23 "$WT_CASES" "$REAL_CASES")" + extra_cases="$(comm -13 "$WT_CASES" "$REAL_CASES")" if [[ -n "$missing_cases" ]]; then - fail "REQ-EX-S09-01: $WT shows case name(s) absent from real 'assent test' output:" + fail "REQ-EX-S09-01: $WT shows case/decision pair(s) absent from real 'assent test' output (invented case, or a decision that doesn't match the real run):" printf ' %s\n' "$missing_cases" >&2 + elif [[ -n "$extra_cases" ]]; then + fail "REQ-EX-S09-01: real 'assent test' output has case(s) the walkthrough's console blocks omit — a block claiming to be a complete run is now incomplete:" + printf ' %s\n' "$extra_cases" >&2 else - pass "REQ-EX-S09-01: all $wt_case_count walkthrough console case names are real" + pass "REQ-EX-S09-01: all $wt_case_count walkthrough console (case, decision) pairs are real, and no real case is missing" fi fi From 49ea6e665d7d195dce2267ea3b18b7643de2be0b Mon Sep 17 00:00:00 2001 From: Konrad Heimel Date: Sun, 16 Aug 2026 13:57:39 +0200 Subject: [PATCH 7/7] :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 F1 fix --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f098b2..bcac63c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -129,6 +129,7 @@ repository still gets a decision, never by following the link; no release carrie - :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane - :memo: docs(walkthrough): fix Step 3 lede self-contradiction on .tf format count - :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane follow-ups +- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 F1 fix ### Features - :sparkles: feat(docs): gate example pack and format claims against dogfood @@ -145,6 +146,7 @@ repository still gets a decision, never by following the link; no release carrie - :bug: specs(p5-ex): make S10 schema freeze and S07 fence non-vacuous - :bug: fix(docs-gates): fail on unmapped format tokens and de-confound the tests-dir mutation (EX-S01 review) - :bug: fix(examples): make privilege-tier's proving fixture load-bearing +- :bug: fix(docs): pin case+decision pairs, not just case names, in the walkthrough (F1) ### Refactoring - :recycle: refactor(examples): discover dogfood packs instead of a hardcoded 3-name loop