Skip to content

fix: Guard the evidence-exception bound by construction, not by regex over prose - #615

Open
webbrain-one wants to merge 1 commit into
BaseInfinity:mainfrom
webbrain-one:webbrain/issue-608
Open

fix: Guard the evidence-exception bound by construction, not by regex over prose#615
webbrain-one wants to merge 1 commit into
BaseInfinity:mainfrom
webbrain-one:webbrain/issue-608

Conversation

@webbrain-one

Copy link
Copy Markdown
Contributor

Closes #608

State the first-in-this-root-task evidence-exception bound once in a canonical anchor and reference it throughout the docs. Replace the prose regex guard with a drift check so unbounded restatements are prevented structurally instead of chased through English phrasing.
@BaseInfinity

BaseInfinity commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Review of PR #615 — reconciled from two independent reviewers

First: sorry for the mess on this PR. It was closed, reopened, and buried under half a dozen overlapping comments in the space of an hour. That was us moving too fast, and none of it was a reflection on your work. This comment replaces all of that — it's the only one you need to read.

Status: open, and we want to merge it. Nothing is blocked on it either — #608 carries no milestone and isn't on the path to v1.99.0 or v2.0.0. No deadline, no pressure.


Your instinct is right

This repo runs significant changes past two independent models — Fable 5 and Sol (GPT-5.6). Both reviewed your branch separately, neither knowing what the other found. Both endorsed your core approach: state the rule once, reference it everywhere else. That is the correct direction, and it's the direction we're taking regardless of what happens to this PR.

They also independently found the same four defects, which is worth something — it means these are real properties of the code, not one reviewer's taste.

The four defects

1. The test kills itself before it can fail. On your branch:

$ bash tests/test-evidence-exception-bound.sh
PASS: canonical rule stated once in the snippet
$ echo $?
1

One PASS, exit 1, no failure message. set -euo pipefail plus an assignment fed by a pipeline — when the inner grep matches nothing it returns 1, pipefail propagates it, and set -e kills the script before your || fail line runs. The guard goes silent exactly where it should speak. Genuinely easy to miss.

2. The self-falsification block doesn't falsify the guard. Both mutations sed a shell variable and grep it in memory. The file is never changed and the guard is never re-run — so it proves grep -F needs its pattern present, not anything about your guard. This one matters most, because #608 exists precisely because the previous guard was trusted without being falsified.

3. The count check is >= 1, not equality with mention_count. The doc has two restatements; one compliant restatement satisfies the check while the other stays unbounded. The rule is "every restatement carries the bound", so the check has to be equality.

4. It isn't wired into .github/workflows/ci.yml. So it would never run. Combined with (1), it would sit in tests/ looking like coverage while enforcing nothing.

The one you couldn't have known

package.json "files": ["cli/","skills/","hooks/",".claude-plugin/",
                       "CLAUDE_CODE_SDLC_WIZARD.md","AI_SETUP_LANES.md","CHANGELOG.md"]

docs/ doesn't ship to consumers. CLAUDE_CODE_SDLC_WIZARD.md does. So a pointer from the wizard doc to docs/snippets/… would be broken in every installed copy. You'd have had to read package.json to catch that — as Fable put it, "That's on the repo for not documenting it, not on you."

Design direction both reviewers agreed on

Drop the separate snippet file. Line 4383 of CLAUDE_CODE_SDLC_WIZARD.md already carries the full FIRST-in-this-root-task bound — make that the canonical statement, mark the block explicitly (an HTML comment fence is fine, it survives rendering), and scope the guard to consistency within that one file.

Don't try to detect unmarked English paraphrases. That's what died in #606 — five near-misses that reversed the meaning while still matching the pattern, across three review rounds. Marked blocks make the problem winnable; paraphrase detection is the trap this issue was filed about.

What would get it merged

A demonstration, in the PR:

  1. Break the rule in the shipped doc on purpose
  2. Run the guard — show it failing, with its own message, non-zero exit
  3. Restore the doc, run it again — show it passing
  4. Wire the test into ci.yml

That's the bar. It's the same standard this repo holds its own changes to: a guard is believed once it's been watched catching a real break, not because it passes.

Both reviewers committed to it — Fable: "Do that and I'll clear it." Sol: "That evidence would directly answer my concerns."

Take it up if it interests you. If you'd rather not, that's completely fine — say so and we'll close it with your branch recorded as the evidence, and you'll still have saved us from shipping the wrong design.

@BaseInfinity BaseInfinity reopened this Aug 15, 2026
BaseInfinity added a commit that referenced this pull request Aug 15, 2026
…emembers (#540)

Round 2 fixed the wrong thing correctly. It replaced a stale local tracking ref
with `baseRefOid`, which both reviewers had independently prescribed — and which
live data falsifies. That field is the base commit associated with the pull
request, a snapshot that does not move when the branch does. On this repo PR
#615 carries baseRefOid f8ba12b while main is at d0e1c7b, and the trees differ.
So round 2 swapped an indefinitely stale local cache for an indefinitely stale
server-side snapshot: the same hole, relocated.

Both rounds failed the same way, and the shared mistake is the finding: read a
cached value, treat it as current. The base tip now comes from
`gh api repos/{owner}/{repo}/git/ref/heads/<base>` on every run, and
`baseRefOid` is removed from the parsed field list rather than left sitting
there for someone to reach for again.

Everything downstream of the OID is unchanged — the fetch-fallback, the
fail-closed refusal, the tree comparison were all reviewed and cleared.

The round-2 regression test agreed with the bug. It hand-set the base OID to the
advanced commit, which is precisely what GitHub does not do, so it could never
have caught this. The stub now drives the two knobs separately: baseRefOid stays
at the snapshot while the ref endpoint reports the moved tip. Reverting the
source to the snapshot field is red permanently.

A shipped comment in the commit hook asserted that the merge boundary reads
baseRefOid. It did, and it was wrong; the correction keeps the falsification
rather than quietly restating the conclusion.

Removing the field regressed the clearance suite to 40/9 — eight stubs answered
`pr view` and not the ref endpoint. All eight now answer both.

Mutation: swapping the source back to the snapshot gives 83/2 with the fixture
merging at exit 0, reproducing the round-2 defect. A first attempt broke the
JSON parse instead of the source and went red for an unrelated reason, which
proves nothing; it was redone as a clean swap.

The check-to-merge race is recorded as an accepted limit. --match-head-commit
binds the head atomically and GitHub offers no base equivalent, so a base move
inside that window cannot be closed from the client.

Suites: merge-gate 85/0, clearance-binds-to-tree 8/0, hooks 225/0,
cross-model-clearance 49/0.
BaseInfinity added a commit that referenced this pull request Aug 15, 2026
* fix(gate): bind certification to content, not to a SHA (#540)

The CERTIFIED lane keyed staleness on `commit_sha == HEAD`. PreToolUse runs
before the commit, so at check time HEAD is still the certified commit and the
gate allowed — then the commit landed carrying whatever was staged. The
resulting commit was by construction never the one that was reviewed. Sol found
it at file:line; Fable withdrew its own recommendation to extend the gate once
shown the counter-example.

The same key also taxed every commit with a re-pin call, ~10 wasted tool calls
in one measured cycle, to protect nothing a content pin does not.

Certification now names the tree it was issued over. The hook compares
`git write-tree` against `candidate_tree`; the merge gate compares the remote
head's tree, and the base it was read against, before merging.

An earlier ruling chose `git patch-id --stable`. Both advisors amended it after
measuring two failures: two diffs differing only in the indentation of an added
Python line produce the identical patch-id, and whitespace is semantic in
Python, YAML, Makefiles and string literals; and what merges is the resulting
tree, not the diff, so a rebase onto moved upstream keeps patch-id identical
while producing content nobody read.

Message-only amend, squash and reorder with no upstream movement all produce
the same tree, so certification survives them at zero cost. Only content that
moves invalidates.

The hook does not parse commit flags. Requiring a clean tracked worktree makes
`git commit` and `git commit -a` commit the same tree, so the index speaks for
both; modelling git's option grammar in a regex is what took #610 from 3 to 6
to 4 before round 4 deleted it. `base_tree` is checked only at the merge
boundary for the same reason: at commit time the base is HEAD or HEAD^
depending on whether it is an amend, and that costs nothing to defer because a
rebase changes the index tree anyway.

Three suites encoded the old contract and were rewritten with it. The stale-SHA
row became a moved-content row, because a new commit that changes nothing
reviewable is now exactly the case the ruling preserves. Two fixtures used a
made-up 40-char head with no git objects behind it, which cannot survive a
check that resolves a tree; both now build real ones.

Prior art read rather than derived, per the ruling: codex-sdlc-wizard#111's
requireFrozenIndex + candidateTree.

Each guard proven by mutating the file and re-running: hook 8/0 -> 6/2, and
either merge-gate comparison neutered lets the fixture merge.

* fix(gate): read the base from the PR, not from a local tracking ref (#540)

Round 1 of the cross-model review found one blocker, independently, in both
legs, and both constructed it rather than argued it. The base_tree check
resolved `origin/$BASE_BRANCH^{tree}` — a local cache. Absent, the
`[ -n "$CUR_BASE_TREE" ] &&` guard skipped the comparison outright: one
`update-ref -d` merged a certification carrying a bogus base_tree at exit 0.
Stale, the cached tree still satisfied the certification: a bare server repo
with main advanced and the local ref left behind reached the same exit 0. The
check was fail-open in exactly the situation it exists for, and its posture
contradicted the fail-closed candidate_tree check ten lines above it.

The base is now read from the PR's own baseRefOid, resolved with the same
fetch-fallback as the remote head, and blocks when it cannot be resolved. Both
reviewers prescribed this same fix.

A claim in the round-1 rationale was also false, and it was load-bearing. The
deferral of base_tree to the merge boundary was justified by "a rebase onto
moved upstream changes the index tree, so the hook's check already refuses it".
When upstream independently produces the candidate's final content, the rebase
drops the redundant branch change and the index tree does not move. The
deferral survives on the narrower ground that PreToolUse cannot observe
post-command state and the base ref is well-defined only at the merge boundary
— which is precisely why that boundary may not trust a cache.

The test that asserted the exact merge command read .fixture-sha after deleting
the fixture, so the substitution was empty and the grep passed on a prefix
match without checking the SHA.

Guards proven by mutation, reverted by inverse edit: neutering the fail-closed
branch gives 84/1, and reverting the base read to the tracking ref gives 83/2
with the fixture merging at exit 0. The first mutation attempt reverted with
`git checkout --`, which silently discarded the uncommitted fix along with the
mutation and made the reading meaningless.

The unreadable-base row first asserted only a non-zero exit and stayed green
through its own mutation, because the mismatch check below rejects an empty
tree anyway. Both new rows now assert the specific refusal.

* fix(gate): ask the server where the base branch is, not what the PR remembers (#540)

Round 2 fixed the wrong thing correctly. It replaced a stale local tracking ref
with `baseRefOid`, which both reviewers had independently prescribed — and which
live data falsifies. That field is the base commit associated with the pull
request, a snapshot that does not move when the branch does. On this repo PR
#615 carries baseRefOid f8ba12b while main is at d0e1c7b, and the trees differ.
So round 2 swapped an indefinitely stale local cache for an indefinitely stale
server-side snapshot: the same hole, relocated.

Both rounds failed the same way, and the shared mistake is the finding: read a
cached value, treat it as current. The base tip now comes from
`gh api repos/{owner}/{repo}/git/ref/heads/<base>` on every run, and
`baseRefOid` is removed from the parsed field list rather than left sitting
there for someone to reach for again.

Everything downstream of the OID is unchanged — the fetch-fallback, the
fail-closed refusal, the tree comparison were all reviewed and cleared.

The round-2 regression test agreed with the bug. It hand-set the base OID to the
advanced commit, which is precisely what GitHub does not do, so it could never
have caught this. The stub now drives the two knobs separately: baseRefOid stays
at the snapshot while the ref endpoint reports the moved tip. Reverting the
source to the snapshot field is red permanently.

A shipped comment in the commit hook asserted that the merge boundary reads
baseRefOid. It did, and it was wrong; the correction keeps the falsification
rather than quietly restating the conclusion.

Removing the field regressed the clearance suite to 40/9 — eight stubs answered
`pr view` and not the ref endpoint. All eight now answer both.

Mutation: swapping the source back to the snapshot gives 83/2 with the fixture
merging at exit 0, reproducing the round-2 defect. A first attempt broke the
JSON parse instead of the source and went red for an unrelated reason, which
proves nothing; it was redone as a clean swap.

The check-to-merge race is recorded as an accepted limit. --match-head-commit
binds the head atomically and GitHub offers no base equivalent, so a base move
inside that window cannot be closed from the client.

Suites: merge-gate 85/0, clearance-binds-to-tree 8/0, hooks 225/0,
cross-model-clearance 49/0.
@BaseInfinity

Copy link
Copy Markdown
Owner

Thanks for this — the design is right and it's the design #608 actually ruled: state the rule once, reference it everywhere, guard the reference rather than the prose. The self-falsification block at the end is a genuinely nice touch and is exactly the standard this repo asks for.

Three concrete things stand between this and merge. All are mechanical, and the first one is the reason CI wouldn't have caught them.

1. The test fails against this PR

Run in a clean checkout with the PR applied:

$ bash tests/test-evidence-exception-bound.sh
PASS: canonical rule stated once in the snippet
$ echo $?
1

It fails at the canonical_count check. CLAUDE_CODE_SDLC_WIZARD.md has two restatements of the evidence-only exception (lines 4316 and 4383), and neither carries the canonical bound nor references the snippet — the PR adds the snippet and the guard but never edits the doc the guard checks.

So the PR needs a third file: the two restatements updated to either carry the canonical line verbatim or say see docs/snippets/evidence-exception-bound.md. That's the actual behaviour change; right now the guard is correct and has nothing to pass against.

2. The test isn't registered in CI

.github/workflows/ci.yml invokes each suite as its own explicit step — there's no runner script that discovers tests/*.sh. grep -c evidence-exception-bound .github/workflows/ci.yml returns 0, so this suite would never run and item 1 would have gone unnoticed indefinitely. Needs a step alongside the other suites.

3. docs/snippets/ isn't shipped, but the doc that references it is

package.json files is ["cli/", "skills/", "hooks/", ".claude-plugin/", "CLAUDE_CODE_SDLC_WIZARD.md", "AI_SETUP_LANES.md", "CHANGELOG.md"]. CLAUDE_CODE_SDLC_WIZARD.md ships to every consumer; docs/snippets/ does not. Once the doc says "see docs/snippets/evidence-exception-bound.md", every installed copy carries a pointer to a file that isn't there.

Two ways out, and the second is probably better:

  • add docs/snippets/ to files, or
  • have the shipped doc carry the canonical line verbatim and keep the snippet as the repo-local source of truth. The guard already accepts verbatim-or-reference, so this needs no test change — and it keeps the consumer-facing doc self-contained, which matters more for a file people read offline.

Smaller notes, non-blocking

  • grep -EiF -e ... -e ... passes -E and -F together. Behaviour when both are given isn't guaranteed across implementations; since both patterns here are literals, plain -F says what's meant.
  • set -euo pipefail is right, but grep -c ... || true under pipefail still works because the || true is on the command substitution — worth a comment so nobody "fixes" it later.

Milestone

Tagged for v2.0.0 rather than the current milestone. v1.99.0 is scoped to the merge-gate work in flight and I don't want to hold this behind it.

Happy to push the three fixes onto your branch if you'd rather not — say the word. Otherwise it's yours; the change is small and the reasoning in the header comment is already better than most of what's in tests/.

@BaseInfinity

Copy link
Copy Markdown
Owner

Update on the three blockers above — plus a fourth we hit while fixing them. All four are fixed and green locally; the patch is staged and lands on this branch shortly. maintainerCanModify is on, so it goes on your branch and the PR stays yours.

1. The guard failed against its own PR. CLAUDE_CODE_SDLC_WIZARD.md was never edited, so ref_count >= 1 failed. Both restatements (4316, 4383) now carry the bound verbatim. Verbatim rather than a see docs/snippets/... pointer because docs/snippets/ is not in package.json files — consumers read the wizard doc offline, and a repo-relative pointer in shipped material is a dangling instruction. That also means the snippet dir does not need to ship, which keeps CLAUDE.md's shipped-files list and the doc-consistency suite out of it.

2. Your test contradicted your own snippet. The snippet says restatements may be "verbatim or reference"; the final check required a reference unconditionally. Amended to accept either — this aligns the test with the contract it states, it does not weaken it. Proven non-vacuous: with the bound stripped from the doc and the phrase renamed so the earlier check is skipped, the amended check fails on its own.

3. Not registered in CI. Added after the doc-consistency step in .github/workflows/ci.yml.

4. New — #513 fence-only-assertions rejected the test. grep -F "$CANONICAL_RULE" "$DOC" interpolates a shell variable, so the scanner cannot verify what the assertion actually asserts. Both greps against $DOC now inline the literal. Was 136 passed, 1 failed; now clean.

Evidence:

  • ./tests/test-evidence-exception-bound.sh → all checks pass, exit 0
  • RED against the doc as this PR left it (git show HEAD:CLAUDE_CODE_SDLC_WIZARD.md) → exit 1, which is blocker 1 reproduced
  • ./tests/test-doc-consistency.sh → all pass (was 136/1)

On timing, and it is scheduling, not a hold-up: an ancestry bind is in flight on our side that pins a clearance to a specific base_sha. Landing this first would go stale the moment that merges and force a second review round on your PR. Ours-first costs your PR one round instead of two. The review itself is one clean round for a change this size.

The design call in #608 — state it once, guard by construction rather than by regex over prose — is right, and it is the part that was hard. Thanks for taking it on.

@BaseInfinity

Copy link
Copy Markdown
Owner

Thank you for this, @webbrain-one — this is the right diagnosis and the right shape of fix.

The regex-over-prose guard that #606 deleted failed for exactly the reason you name in the header comment: it recognized a vocabulary, not a meaning, and five meaning-reversing near-misses walked past it over three rounds. Stating the rule once and checking that every restatement either carries it verbatim or points at it is the correct inversion. The self-falsification block at the end is the part I want to call out specifically — a guard that does not demonstrate it can fail is a guard nobody should believe, and shipping that demonstration inside the script rather than in a commit message is better than what we had.

maintainerCanModify is on, so I'm going to push the adaptation to your branch and merge this PR rather than reimplementing it. Here is everything I'm changing and why, so nothing lands as a surprise:

1. The canonical block moves from docs/ into CLAUDE_CODE_SDLC_WIZARD.md. This is the one real defect, and it isn't visible from inside the repo. docs/ is not in package.json's files list, so it is not in the published npm package — I verified against npm pack --dry-run, not the files field, because npm also auto-includes files the field doesn't mention. Consumers receive CLAUDE_CODE_SDLC_WIZARD.md, which under this PR would reference docs/snippets/evidence-exception-bound.md — a path that does not exist in their install. A dangling pointer to the single source of truth is worse than the prose it replaced.

So the canonical statement goes into the shipped doc inside an explicitly marked block (<!-- CANONICAL:evidence-exception-bound --><!-- /CANONICAL -->), and the guard anchors on those markers. Marked, not paraphrase-detected — that keeps your core insight intact.

2. The guard has to actually pass. As written, ref_count requires CLAUDE_CODE_SDLC_WIZARD.md to reference the snippet, and today it doesn't — so this fails on checkout. That's a fine RED state to have arrived at, but it needs the corresponding doc edit to go green. Folded into (1).

3. Strengthening the self-falsification. Right now the two mutations are applied to a shell variable and then greped:

_mut1=$(sed '/FIRST-in-this-root-task/d' "$SNIPPET")
printf '%s' "$_mut1" | grep -qF "$CANONICAL_RULE" && fail "..."

That proves grep can miss a string it wasn't given — it never re-runs the guard. I'm rewriting it to write each mutant to a temp tree and invoke the checking logic against it, so a failure means the guard failed. Same intent, actually load-bearing. _tmpdir and the trap you already set up are what make this cheap.

4. Restoring the full mutation set. #608's bar asks for the five near-misses that defeated the old guard, recoverable from git show 7f11280:tests/test-doc-consistency.sh. Your two are a subset. All five get run.

5. Wiring into .github/workflows/ci.yml, which #608 requires and this PR doesn't do yet. A suite that isn't in CI is a suite that stops running.

One smaller thing: grep -EiF passes both -E and -F, which are mutually exclusive and resolve differently across grep implementations — that becomes moot in the rewrite.

None of this changes what your guard decides. It changes where the canonical text lives so consumers get it, and makes the falsification prove the thing it claims. I'll run the full mutation set and post the fail/restore/pass output here before merging.

@BaseInfinity

Copy link
Copy Markdown
Owner

@webbrain-one — an update, and thanks again.

Your guard is going in. The continuation is #669, and the credit is yours: your commit fa4051f is preserved as authored by you, and every commit on top carries your Co-authored-by trailer. The single-source-of-truth design and the self-falsification requirement — the two things that made this work — are your contribution.

I opened a separate PR rather than pushing to your branch because the work rebases onto current main, and updating this PR in place would have meant force-pushing over your fork. That's not mine to do. I'm leaving this PR open; closing it is the maintainer's call, not mine.

What happened to it since:

  • The rebase justified itself immediately — main had grown two new statements of the rule that your guard correctly rejected.
  • One real problem in the contributed version: the check accepted canonical_count >= 1 while printing "all N restatements carry the bound". I reproduced it with a 6-mention fixture where 5 reversed the rule, and it printed PASS. Now every detected mention must comply.
  • Eight rounds of cross-model review found a demonstrated defect every round. Three of them were spent trying to stop a markdown renderer from capturing the rule, and each fix was falsified by the next render. That ended in a scope decision rather than a fourth patch — render capture is now a stated limit with its threat model, because anyone who can wrap the rule in <section hidden> has commit access and could simply delete it.
  • Your self-falsification idea is what carried the whole thing. It's now 20 mutations, each declaring which check must reject it, plus 3 must-pass fixtures — those catch checks that are wrong in the other direction, and all three caught defects nothing else would have.

The header records what the guard does not do before what it does. That framing came out of your PR's discipline about proving the guard can fail.

BaseInfinity added a commit that referenced this pull request Aug 18, 2026
* fix(review): forbid raw HTML before the rule, and make the harness verify its own claim (#608)

Seat 1 falsified this repair's load-bearing claim with a render, which is what
it was asked to do.

P1 — I asserted from the GFM spec that a block surrounded by blank lines cannot
be captured by a span opened outside it. Raw HTML is not an inline span and does
cross blank lines: a raw <s> opened before the blank line renders the complete
rule under html>body>s>p>strong while the checker exits 0 with markers isolated,
blank lines intact and the body byte-identical. Enumerating <s> would repeat the
mistake, since <del>, <code>, <div hidden>, <script> and <template> retract or
hide the rule the same way. The invariant is now that NO HTML element opens
before the canonical block — verified empirically as already true, since the
only tag-shaped strings in that region are prose placeholders like <branch> and
<sha>, which are not HTML elements.

P2 — the claim "all mutations fail for their intended reason" was false. Form C
bypass #1 had drifted onto the marker-isolation check instead of the equality
check it was written for, and because it was still being caught, nothing said
so. That is the same overclaim class this PR has been enforcing on others. The
fixture is repaired, and each mutation now DECLARES which check must reject it,
with run_mutation failing if a different one does. Every fixture also asserts
its own preconditions, so it cannot degrade into a weaker test than its label.

P3 — the mutation-count diagnostic still read "expected 16 mutations".

19/19, each verified by the harness to fail via the specific check it targets.

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>

* fix(review): stop an inline span from retracting the canonical rule (#608)

Seat 1 was asked to demonstrate a disputed round-2 finding rather than restate
it, and did: a cmark-gfm render in which the complete canonical rule sits inside
<del> — struck through, i.e. retracted — while the checker exits 0. It is
achieved by inlining the CLOSING marker into the following prose and opening a
`~~` span before the block. The normalised body is byte-identical, so the
equality check cannot see it. Dispute withdrawn on the evidence.

The fix is structural rather than lexical: each marker must be ALONE on its own
line, and the block must be surrounded by blank lines. GFM inline spans cannot
cross a blank line, so no inline construct opened outside the block can capture
the rule. Chasing `~~` would have been the treadmill again — an unclosed <span>
or a blockquote does the same job. A `~~`-inside-the-block check stays as cheap
belt-and-braces.

Seat 1's exact reproduction is Form E, and its fixture asserts the normalised
body is unchanged, so it keeps testing the reported hole instead of quietly
degrading into a text-changed mutation.

Round 3 also returned SHAPE: SOUND, verified the two round-2 repairs, and
accepted the measured dispute on punctuation-joined reversals, which is now a
named non-goal in the header.

18/18 mutations rejected, each for its intended reason.

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>

* Guard evidence-exception bound by construction

State the first-in-this-root-task evidence-exception bound once in a canonical anchor and reference it throughout the docs. Replace the prose regex guard with a drift check so unbounded restatements are prevented structurally instead of chased through English phrasing.

* fix(review): guard the evidence-exception bound by construction (#608)

Adapts PR #615 by @webbrain-one. The single-source-of-truth design and the
self-falsification requirement are theirs.

- move the canonical statement into the shipped CLAUDE_CODE_SDLC_WIZARD.md;
  docs/ is not in the npm pack, so a canonical block there reaches no consumer
  install and the shipped doc would reference a nonexistent path
- keep the doc's own certified wording as canonical; the PR's snippet had
  drifted the rule from 'a later evidence-only finding is filed' to 'handed off
  for human review'
- require EVERY mention to carry the reference token in its own sentence; the
  contributed check was 'at least one' while reporting 'all', and passed a doc
  in which five of six restatements reversed the rule
- re-run the real checker as a subprocess against mutated files, instead of
  grepping a string in memory
- run all five predecessor near-misses, each in-block and each injected
  adjacent to a compliant sentence
- honor TMPDIR; wire into ci.yml and CONTRIBUTING.md

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>

* test(608): make the fence check claim what it enforces, and sweep by quantifier

Two findings from seat 1's round-7 leg. Neither is repaired the way it was
reported, and the reasoning matters more than either fix.

P1 — the fence scanner treats a fence left unclosed inside a list item as
still open past that list item. Seat 1 is right about CommonMark: container-
nested fences close at the end of their container, so that document is valid
and was being rejected.

Resolved by REDEFINITION, not by parsing. 1b now requires every fence opened
before the rule to be EXPLICITLY CLOSED IN THE SOURCE — deliberately stricter
than CommonMark. Under that rule the list-item case is the check firing, not a
false positive, on a shape this document can simply not write. The alternative
is a container-block parser, and the alternative to that is a full markdown
implementation. Rounds 3-5 of this PR already proved a source-text checker
cannot win an argument about rendering; a claim about what the SOURCE contains
is exact and settleable. Recorded as the third stated limit.

The error message is corrected too. It said the rule "would render as a code
sample" — a render claim, and false for exactly the list-item case raised. It
now makes the source claim only.

P2 — the header still promised to keep "every honest edit reachable" while its
own stated limit says otherwise. This was the third sweep, and the two before
it each fixed the sentence seat 1 named and left the class behind. So this one
sweeps by QUANTIFIER (every|all|any|cannot|never|always) rather than by report.
Three universals were live and all three are bounded: the guard binds each
restatement IT DETECTS; "never defeated" is dated to the rounds that tested it;
and the header now states plainly that it does NOT keep every honest edit
reachable.

Seat 1's counterexample is reproduced rather than assumed — appending "Stale
evidence buys another pass." leaves the checker exiting 0, because MENTION does
not match that sentence. That is stated limit #1 working as documented, and it
stays. What changed is that the header may no longer promise past it.

No parser added. No fixtures added. Still 20 mutations and 2 must-pass
fixtures.

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>

* test(608): measure fence indent in columns, not characters

Seat 1's merge-clearance leg found one blocking defect, and it is mine rather
than a scope argument: the fence scanner measured indent in CHARACTERS. A tab
advances to the next 4-column tab stop, so a single leading tab is already 4
columns and its line is indented code, not a fence. Measuring characters saw
1 and treated a tab-indented literal ``` as an unclosed fence, rejecting a
valid document.

Verified all three shapes after the fix, not just the reported one:
  tab-indented literal ```      -> ACCEPT (was REJECT)
  3-space-indented open fence   -> REJECT
  unindented open fence         -> REJECT

Added as must-pass fixture #3. Indent arithmetic is exactly the kind of thing
a later edit re-simplifies, and no must-fail mutation can catch a check that
is wrong in this direction.

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>

* fix(review): close five bypasses seat 1 demonstrated against the bound guard (#608)

Cross-model review (gpt-5.6-sol, high) returned WRONG_SHAPE with five IN-CARD
findings, each demonstrated by running the guard against a constructed mutant.

- the canonical block was containment-checked, so a contradiction inserted
  beside the rule passed; it is now matched EXACTLY
- "Otherwise STOP." had been severed from the certified sentence by the
  explanatory prose, so flipping it to "Otherwise CONTINUE." passed; the
  default is back inside the block and the prose moved after it
- a reversing clause joined by a semicolon shared the token's sentence; the
  clause split now breaks on ';' as well
- MENTION missed "Invalid verification evidence" because invalidat\w* does not
  match "Invalid"; added an alternative, with the vocabulary limit stated
- the Form-B injector cut at the '.' in "test-evidence-exception-bound.sh" and
  spliced mid-filename, so no Form-B mutant realised the adjacency its label
  claimed; it now targets a clause carrying both a mention and the token, and
  asserts placement, non-mangling, and that the injected text is detectable
- the [bound: ...] token no longer sits inside the rendered ASCII diagram; the
  caption points at the SCOPE RULE instead of restating it
- the CI step name is quoted, so '#' no longer truncates it

Sol's four bypasses are kept as permanent regression mutations, and the harness
now reports why each mutation was rejected — a bypass fixed without a test is a
bypass that returns, and a mutation caught for the wrong reason is not evidence.

14/14 mutations rejected, each for its intended reason.

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>

* fix(review): close the marker-counting and code-fence holes (#608)

Round 2 of seat 1's review. Two findings repaired, two disputed with evidence.

FIXED — marker uniqueness counted matched PAIRS, so a stray unmatched marker
survived (reproduced: a bare closing marker appended at EOF, rc=0). A later
edit could pair with it and move the block. Now requires exactly one occurrence
of each marker in the raw text, with regression mutations for a stray closing
AND a stray opening marker.

FIXED — the canonical block could be swallowed by an unclosed code fence
earlier in the document, rendering the rule as an inert code sample. Added a
fence-parity check from doc start to the opening marker, and a mutation that
wraps the block in an unclosed fence. Deliberately not extended toward general
markdown-rendering semantics.

DISPUTED — reversals joined to the token by an em dash, parenthetical, colon,
list break or table cell. Measured rather than argued: widening the clause split
to cover them splits the reference token at its own colon and produces eight
false violations, against zero today. No punctuation class is an attachment
boundary in English. Recorded as a NAMED NON-GOAL in the header so it is not
rediscovered as a defect.

DISPUTED — that the rule could render as deleted text. Strikethrough inside the
markers already fails the equality check, and GFM strikethrough is inline-level.
Seat 1 demonstrated the fence case with a cmark render but asserted this one;
a demonstration is requested, on the same standard the driver is held to.

Also folded in seat 2's P3: the header claimed evasion "requires discussing the
rule without using its words", which is false — "Stale evidence buys another
pass" uses its vocabulary and matches nothing listed.

Recorded in the handoff: two of round 2's three findings were PROMPT-SEEDED by
the driver's own round-2 prompt, which listed the evasion categories that came
back as findings. The shipped doc forbids exactly that. Round 3's prompt has no
attack sections.

17/17 mutations rejected, each for its intended reason.

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>

* test(608): delete the render-containment invariant, close the fence class by spec

Seat 1 falsified the raw-HTML element scan from both sides in one round, and
dispositioned it DELETE rather than REPAIR. Seat 2 ratified. It was too narrow
(<section hidden> is in no hand-written element list; a ~~~ fence captures the
block using no element name at all) and too broad (a closed <kbd>Ctrl</kbd>,
which can capture nothing, failed the suite).

Three consecutive rounds each closed one render-capture hole and each opened
another. The exit is a scope decision, not a fourth patch: every check that
reads source text for a rendered format is falsifiable in render-space by
construction. A render can falsify a claim about what a check catches; it
cannot falsify a decision about what the check is for.

- Delete check 1d entirely, leaving its history in place so it is not
  reintroduced.
- Record render capture as a SECOND STATED LIMIT, with the threat model that
  makes it the right scope: anyone who can open <section hidden> here has
  commit access and could delete the rule outright. Review is the control for
  a committer-adversary; this guard catches accidents.
- Extend fence parity to ~~~. CommonMark defines exactly two fence characters,
  so this set is closed by spec and cannot grow again. The two are counted
  separately, never summed — a ``` line inside an open ~~~ block is literal
  content, and a combined count would corrupt parity either way.
- Scope 1c's claim to GFM inline spans. It said "any inline construct", which
  is false: raw HTML is not an inline span and does cross blank lines.
- Swap Form F (raw <s>) for Form G (unclosed ~~~ fence). Still 19 mutations,
  each asserted to fail via its declared check.
- Add a must-PASS fixture for <kbd>Ctrl</kbd>. A deleted check leaves no
  trace, so nothing else would stop the overbreadth from returning.

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>

* test(608): track fence state per CommonMark instead of counting prefixes

Seat 1 accepted the render-capture scope ruling this round (SHAPE: SOUND, no
finding against the named non-goal), then found the fence check I wrote to
replace the deleted one was wrong in both directions.

It counted prefix-shaped lines, which is not fence state. A properly closed
```` block containing a literal ``` line was REJECTED — a false positive on a
valid document. The same block left unclosed was ACCEPTED and captured the
rule — a false negative on the exact attack the check exists for.

- Track fence state per CommonMark: a closing fence must use the same
  character, be at least as long, and carry no info string; while a fence is
  open, every other line is literal content. Skip 4+ space indents (indented
  code blocks) and backtick openers whose info string contains a backtick.
- Add Form H (unclosed ```` must fail) and a second must-PASS fixture (closed
  ```` containing ``` must be accepted). Each asserts that naive prefix parity
  would have gotten it wrong, so neither can degrade into a test a counter
  would also pass. A must-fail mutation cannot detect a check that is wrong in
  the other direction; that is what the must-pass fixtures are for.
- 19 -> 20 mutations.

Also corrects two more overclaims seat 1 found in this file's own header: the
canonical block is compared VERBATIM AFTER WHITESPACE NORMALISATION, not
byte-for-byte, and the reachability promise is scoped to what MENTION detects.
A paraphrase avoiding every MENTION alternative is neither flagged nor bound.
The honest claim is the narrow one: what this guard finds, it binds.

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>

---------

Co-authored-by: webbrain-one <webbrain-one@users.noreply.github.com>
Co-authored-by: webbrain-one <295484252+webbrain-one@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Guard the evidence-exception bound by construction, not by regex over prose

2 participants