docs(readme): say what has actually been exercised, and move the rest out - #670
Merged
Conversation
Owner
Author
|
CROSS-MODEL-CLEARANCE Seat 1, GPT-5.6 Sol at Five adversarial rounds, four of them P1. The verdict is its own disposition {"confidence": 98, "reviewer": "gpt-5.6-sol", "sha": "8ce3b9e", "verdict": "YES"} |
Owner
Author
|
CROSS-MODEL-CLEARANCE Seat 2, Fable 5 at Basis: seat 1's five rounds converged on its own disposition being applied Recorded against my own earlier position: at round 3 I ruled repair beat {"confidence": 97, "reviewer": "fable-5", "sha": "8ce3b9e", "verdict": "YES"} |
Owner
Author
|
CROSS-MODEL-CLEARANCE Seat 1, GPT-5.6 Sol at Five adversarial rounds, four of them P1. The verdict is its own disposition {"confidence": 98, "reviewer": "gpt-5.6-sol", "sha": "8ce3b9e3d74f543a53df388d191740fe1b21cfbd", "verdict": "YES"} |
Owner
Author
|
CROSS-MODEL-CLEARANCE Seat 2, Fable 5 at Basis: seat 1's five rounds converged on its own disposition being applied Recorded against my own earlier position: at round 3 I ruled repair beat {"confidence": 97, "reviewer": "fable-5", "sha": "8ce3b9e3d74f543a53df388d191740fe1b21cfbd", "verdict": "YES"} |
… out "Choosing Your Model" was 1360 words, and its largest block argued the case for Opus 4.6 over 4.8 — a debate settled two defaults ago, from third-party field reports, about a configuration no cycle of this harness has ever run on. A reader had to get through all of it to find out which model to use. The dishonesty was one of proportion, not of fact: nothing there was false, but the space it took implied a body of evidence behind those lanes that does not exist. Setup A is the only lane with cycle data from this repo. - README's section is now 471 words: the default, an explicit statement of what has actually been exercised (Setup A only — Opus 5 driving, Fable 5 advising, GPT-5.6 Sol gating), the switch commands, the lane table, and a pointer. The other lanes and the 4.6/4.8 pins are named as supported and reachable but never run on this harness — configuration that should work, not measured results. - The 4.6/4.8 research, the per-model effort wall, and "Reading Setup A precisely" MOVE to AI_SETUP_LANES.md under "Model Selection — The Evidence". Nothing is deleted and no recommendation changes. Four guards in test-doc-consistency.sh pinned that content to README, two of them by line number. They follow the content rather than being relaxed, and the position pins are replaced with content matching — #659 asked for exactly that, and the pins had already been re-anchored once and broke again on this move. Each rewritten guard was verified by mutation: reintroducing GPT-5.5 into README, rewriting the historical Vending-Bench citation to 5.6, deleting the Sonnet effort default, and removing README's pointer to the Setup A detail all fail the suite. Also repoints the wizard doc, which sent readers to README for evidence that now lives in AI_SETUP_LANES.md.
…h default Caught by falsifying my own PR body before sending it to review, not by a reviewer. The PR claims "no recommendation changes". Diffing the default line against main falsified it: the trim had dropped "for complex projects", turning Opus 5 at `high` effort for complex projects, `medium` for routine web/CRUD into Opus 5 at `high` effort, `medium` for routine web/CRUD which silently promotes `high` from a complex-project default to the general default. That is a recommendation change, and this PR promised not to make one. Restored verbatim. Also corrects the word count in the PR body: the section is 475 words, not the 471 claimed. Two separate errors — 471 was the count before this fix, and it was already off by one against the real 472. Falsifier results for every quantitative claim in the PR body, all run before review: 1360 -> 475 words measured, both ends six research citations moved, none lost 6 in main, 6 in AI_SETUP_LANES switch-command block unchanged whitespace alignment only four rewritten guards each fire 4/4 caught by mutation
Two P1s from cross-model review, both in-card, both real. P1-1. The note claimed current Setup A "is the lane every cycle in this repo has run on". That is false, and it is the exact overclaim this PR exists to remove — pointed the other way. Verified against the record rather than argued: repo's first commit 2026-01-18 Opus 5 became Setup A v1.9x, 2026-07-24 Opus 4.6 recommended default v1.80.0, June 2026 (CHANGELOG:702) So this repo ran real cycles on Opus 4.6; it was the wizard's own recommended flagship for a period. Writing "no one has run a cycle of this harness on them" traded an overclaim about 4.6's merits for a false claim about its absence. The truthful version is narrower and answers the actual question. What has NOT been exercised is 4.6 or 4.8 paired with the CURRENT review setup — Fable as design authority, Sol as adversarial gate. The pin is old; that pairing is new. The note now says that, cites CHANGELOG.md, and claims only "recent cycle data" for current Setup A. P1-2. The pointer guard was vacuous. It asserted two facts separately — that README contains "how to read Setup A precisely", and that README contains the string "AI_SETUP_LANES.md" — without binding them. README contains that string 4 times for unrelated reasons, so the second grep is true no matter what happens to the pointer. The pointer could lose its link entirely, or point at the wrong file, and the guard stays green. Now both facts must hold in the SAME paragraph, and the link must be a real markdown link to that file. Three mutations confirm it fires, including the bypass review demonstrated: pointer keeps its text but loses its link CAUGHT pointer links to CHANGELOG.md instead CAUGHT pointer sentence removed entirely CAUGHT The first two are what the old guard could not catch by construction.
…lause, not three paragraphs The round-1 P1 fix over-corrected. Correcting the false "4.6 never ran here" claim was right, but the correction spent three paragraphs on it, which re-introduces the complaint this PR exists to answer: the section is too long. Length is also a claim. Giving 4.6 that much room implies it is a currently-relevant measured lane. It is not. It was the recommended default in June 2026, two months and one review setup ago. The note is now one paragraph. The 4.6 fact survives as a dated subordinate clause with its CHANGELOG.md citation, so the P1 fix stands: the record still says real cycles ran on 4.6, and still distinguishes an old pin from an unmeasured pairing with the current review setup. test-doc-consistency 137/0, test-docs-usability 29/0.
…the anchorless link Seat 1 caught the round-1 fix failing for a narrower version of its own reason. Requiring both facts in the same PARAGRAPH was not enough: that paragraph carries a second link, to the billing anchor, which satisfies the `AI_SETUP_LANES.md#` alternative on its own. So the pointer could still lose its link, or point at CHANGELOG.md, and the guard stayed green. Proven, not argued — the pre-fix pattern replayed against both mutations: pre-fix, pointer link stripped MISSED pre-fix, pointer -> CHANGELOG.md MISSED The target is now the ANCHORLESS file link, which only the pointer supplies because every other link in the paragraph carries a `#`. Four mutations against the fixed guard: pointer link stripped to bare text CAUGHT pointer retargeted to CHANGELOG.md CAUGHT pointer reduced to the billing anchor CAUGHT <- the bypass seat 1 named pointer phrase deleted entirely CAUGHT clean tree PASSES Recorded as a limit in the handoff, not sold as a fix: "anchorless" separates the pointer from its neighbours because of how today's prose is written, not because of an invariant. This is the second repair to one guard in one cycle. test-doc-consistency 137/0, test-docs-usability 29/0, test-cowork-drift 30/0.
…se to its link Three rounds, three narrower versions of one defect. Every fix so far asserted "a link to AI_SETUP_LANES.md exists somewhere in a REGION" — first the file, then the paragraph, then the paragraph restricted to anchorless links. Any region wide enough to contain the pointer contains its neighbours, so a sibling link kept satisfying the check on the pointer's behalf. Seat 1 called WRONG_SHAPE on round 3 and was right: the limit I recorded was the defect, not a boundary. A fourth pattern would have been the same mistake one scope smaller. So the principle changes. A markdown link binds phrase and destination in ONE construct. The pointer phrase is now itself the link, the guard checks that construct, and the awk paragraph extraction is DELETED rather than tightened — leaving it in would leave the defect's shape in the code. Six line-scoped mutations, led by seat 1's own round-3 attack: sibling anchorless link + retargeted pointer CAUGHT <- defeated both prior fixes pointer unlinked CAUGHT pointer retargeted to CHANGELOG.md CAUGHT pointer phrase deleted CAUGHT (distinct message) pointer carries an #anchor PASSES (intended: file, not section) clean tree PASSES #530 delete-and-route-to-follow-up was considered — three P1 rounds on one guard is exactly the cost signal it exists for — and rejected: the structural fix is one line and ends the defect class instead of deferring it. test-doc-consistency 137/0, test-docs-usability 29/0, test-cowork-drift 30/0.
Four consecutive P1 rounds on one guard protecting one sentence. #530 exists for exactly this, and both reviewers converge on applying it. Round 4's hole, reproduced before acting: the phrase-as-link fix closed the region class, but the match ended before the file boundary, so a README pointing at AI_SETUP_LANES.md.bak passed 137/0. A dead destination is the third distinct defect class this one-sentence regex has produced. round 1 two facts asserted separately, never bound round 2 same-paragraph binding satisfied by the paragraph's other link round 3 anchorless restriction — a fourth region pattern round 4 construct checked, destination still only pattern-matched Rounds 1-3 were one defect at narrowing scope: assert a link exists in a REGION, which any sibling link satisfies on the pointer's behalf. A fifth pattern would have carried the same "this ends the class" claim the reviewer has called WRONG_SHAPE twice, and that claim is 0-for-3. Deleted whole, both messages — a surviving half re-seeds the region concept. #672 carries the replacement: assert every relative link target in shipped markdown resolves to a file that exists. Generic, so it catches .bak, renames, moves and typos across every doc, which is what four rounds of bespoke regex were reaching for. Filed before this commit so the route is verifiable, not promised. What deletion gives up is accidental rot, now #672's job. Adversarial retargeting to a live wrong file is the committer-adversary case this repo already ruled review controls, per the evidence-exception test's threat model. The README prose stays: a pointer that is a link is better prose with or without a guard. test-doc-consistency 137/0, test-docs-usability 29/0, test-cowork-drift 30/0.
BaseInfinity
force-pushed
the
docs/readme-model-honesty
branch
from
August 18, 2026 05:55
8ce3b9e to
460fd24
Compare
Owner
Author
|
CROSS-MODEL-CLEARANCE Seat 1, GPT-5.6 Sol at The round-5 certification of 8ce3b9e died when the branch rebased onto #669. The leg was asked to check the one thing a byte-identical patch can hide: {"confidence": 99, "reviewer": "gpt-5.6-sol", "sha": "460fd240fa9bfde0dc2597262f41567f0142ce53", "verdict": "YES"} |
Owner
Author
|
CROSS-MODEL-CLEARANCE Seat 2, Fable 5 at Basis: (1) seat 1's five adversarial rounds converged on its own disposition {"confidence": 97, "reviewer": "fable-5", "sha": "460fd240fa9bfde0dc2597262f41567f0142ce53", "verdict": "YES"} |
BaseInfinity
added a commit
that referenced
this pull request
Aug 18, 2026
….0.0 becomes a delivery vehicle (#674) * docs(roadmap): save point carries this session's process facts, not just its outcomes The block a cold session reads was still the release cut. It now records what shipped (#669, #670 merged; #671, #672 filed) and three things that cost real time, written so they are not relearned. 1. Loops end when the SHAPE changes, never when the patch gets better. Two treadmills tonight. The pointer guard ran four consecutive P1 rounds and ended in deletion under #530. "This fix ends the defect class" went 0-for-3 on one guard. 2. A mutation harness that has not been proven to mutate is not evidence. Three separate harness bugs each produced a clean-looking false result before one told the truth. Assert the mutation LANDED before trusting what the suite says about it. 3. Every gate rejection on #670 was legitimate, and a rebase killing a clearance is correct behaviour, not friction. Numbers stated exactly rather than approximately, because both trees are frozen: 1360 words to 449. test-doc-consistency 137/0. * docs(roadmap): point the queue at v2.0.0 as a delivery vehicle (#673) The milestone carried the work for a major release and nothing that ships one. Every issue on it changed the harness; none bumped a version, wrote a CHANGELOG entry, pushed a tag, or verified a consumer could install the result. That gap is how v1.99.0 stopped being a coherent release. No new milestone. v2.0.0 — Consumer Safety Repairs already existed, and a second major-release milestone forks the queue. #673 fills the gap and #638 and #622 are attached rather than duplicated. The terminal gate is now in the milestone description instead of a session's memory: v2.0.0 ships on PROVEN CONSUMPTION (#545), not on open_issues == 0. Everything else there is agent-drivable; #545 is the maintainer's and cannot be done for them. v1.99.3 still ships first and separately — review-leg supervision makes every 2.0 review cycle cheaper, so folding it in inverts the order. test-doc-consistency 137/0. * docs(roadmap): two P1s — I smuggled in a weaker rule, and called an irreversible tag agent-drivable Both in-card, both mine, both caught by review. P1-1. The save point said "when a reviewer returns WRONG_SHAPE twice on the same component, escalate." SKILL.md:170 already routes the FIRST WRONG_SHAPE to the design authority with "do not patch." So the sentence raised the bar to two and weakened a standing rule — as a session note, inside a block whose own scope card excludes standing-ruling changes. Rewritten to cite SKILL.md:170 and frame rounds 3-4 as what following that rule LATE costs. P1-2. "Everything else on that milestone is agent-drivable" was false. #673 ends in `git push origin v2.0.0`. release.yml publishes on that push and npm forbids returning to 1.x, so the milestone has TWO maintainer-held steps, not one. ROADMAP, the milestone description and #673 all now say so, and the standing "publish without me" authorization is named explicitly rather than leaned on in silence — it was given for routine releases, and a major that cannot be unpublished is a different scale. test-doc-consistency 137/0. * docs(roadmap): the npm premise was false, so the caveat built on it comes out Review challenged the tag gate I invented and won on both halves. The premise was wrong. npm refuses to republish an EXACT name/version — it does not forbid a later 1.x. Publishing 1.99.4 after 2.0.0 is legal, and `npm dist-tag` can move `latest` anywhere. So 2.0.0 is unrepeatable, not a one-way door, and a bad release is recoverable by publishing forward. That premise was the entire basis for narrowing the maintainer's standing publish authorization to "routine releases." The authorization was given unqualified. Inventing a caveat on a maintainer's behalf, and then justifying it with a fact nobody checked, is worse than either mistake alone — the caveat reads as deference while quietly rewriting what they authorized. Removed on all three surfaces: ROADMAP, the milestone description, and #673. The tag is agent-drivable like the rest of the milestone; the maintainer holds it only if they say so. #545 remains the one genuine gate, because consuming the harness in other repos is something nobody can do for them. Also corrected a stored memory carrying the same false claim, which would otherwise have re-seeded it next session. test-doc-consistency 137/0. * docs(roadmap): the half-removal I warned about was my own I asked the reviewer to hunt for a fourth surviving instance of the false npm premise, on the grounds that a half-removed caveat is worse than the original because the removal reads as done. It found two. Sweeping for a third found one more the reviewer had not named. ROADMAP.md:41 "that push is the point of no return, because npm forbids returning to 1.x" REMOVED ROADMAP.md:43 still opened "#545 is not the only maintainer-held step", contradicting the rest of its own sentence REMOVED #673 "Note on numbering" repeated the whole false claim REWRITTEN The previous commit said the premise was removed on all three surfaces. It was removed from the three places I had put it deliberately, and survived in the places it had leaked to. Claiming completeness is a claim like any other and should have been grepped, not asserted. #673's note now names the false claim rather than quietly deleting it. It was load-bearing — it was the justification for holding the tag back from the maintainer's standing authorization — so a silent deletion would erase the reason the correction matters. Swept every surface independently: ROADMAP 0, milestone 0, #672 0, PR body 0, repo-wide 0. #673 returns 1, which is the correction sentence itself, verified by reading the line rather than trusting the count. test-doc-consistency 137/0.
This was referenced Aug 18, 2026
BaseInfinity
added a commit
that referenced
this pull request
Aug 19, 2026
…ered the ruling Both were things I asked the reviewer to check. Both were right. P1-1. "Every leg it ran certified a stale base" is false. #670's rounds 1-4 returned blocking findings and certified nothing; only round 5 certified. The branch WAS stale from birth and every leg reviewed a stale base — but attributing certification to non-certifying legs overstates it, in a block whose whole job is to be accurate for a reader who cannot check. P1-2. I framed three skipped rules as evidence FOR the standing no-new-rules ruling. That is the reading most convenient to me. The maintainer ruled against new mandatory steps in SKILL.md and against a commit-hook lane; I widened that into a general position and then claimed my own measurement supported it. The honest version keeps both halves. The evidence rejects MORE duplicate prose — all three rules already existed. It equally supports MECHANICALLY ENFORCING what is already written, which is a different proposition, and is exactly what #616's reconciled recommendation proposes. "Three written rules did not fire" is an argument about enforcement, not about how many rules exist. Corrected on #644 with a follow-up comment rather than an edit, so the claim and its retraction both stay in the record. test-doc-consistency 137/0.
BaseInfinity
added a commit
that referenced
this pull request
Aug 19, 2026
Counted the record properly this time: six legs with verdicts plus one transport-failed leg is seven executions, six of them on a stale base. The reviewer's count was right and mine was wrong for the third consecutive round. "every leg certified a stale base" wrong "every leg reviewed a stale base" wrong a per-round tally still miscounted Three rounds on one number is the shape signal, not an invitation to a fourth attempt. SKILL.md:170 routes a repeated shape failure to a design decision rather than another patch, and a prose claim that has survived three fixes deserves the same treatment. So the number comes out. What stays is the fact a cold reader actually needs: the branch was born stale, every review ran on a stale base until the merge was refused, and it cost a refused merge, two clearance reposts and an extra leg. The omission is marked as deliberate so nobody helpfully restores it. Same lesson as #670's deleted guard, reached four rounds slower because the surface was prose instead of a test. test-doc-consistency 137/0.
BaseInfinity
added a commit
that referenced
this pull request
Aug 19, 2026
* docs(roadmap): leave the save point accurate for a cold session Merged tonight: #669, #670, #674. Filed: #671, #672, #673. Nothing in flight. Records the reconciled cost-review outcome (#616): ten review passes across two docs PRs produced nine real P1 findings, three or four of the passes avoidable. The reconciled recommendation is base-freshness enforcement in run-review-leg.sh — fetch the authoritative ref, resolve the PR's actual base branch rather than assuming main, refuse a stale base. Sol argued against its own competing proposal because it "still trusts callers to enumerate every surface." Records the #644 measurement: three documented rules skipped in one session, none enforced by anything mechanical. That is evidence FOR the standing no-new-rules ruling, not against it — all three already existed and a fourth would not have helped. Records three habits that each cost a round: prove a mutation LANDED before trusting the suite, grep completeness instead of asserting it, and check claims about the outside world too. Every claim here was run before this commit. One was already wrong: I had told the maintainer "eight P1s" twice; the count is nine. test-doc-consistency 137/0. * docs(roadmap): two P1s — an imprecise claim, and a framing that flattered the ruling Both were things I asked the reviewer to check. Both were right. P1-1. "Every leg it ran certified a stale base" is false. #670's rounds 1-4 returned blocking findings and certified nothing; only round 5 certified. The branch WAS stale from birth and every leg reviewed a stale base — but attributing certification to non-certifying legs overstates it, in a block whose whole job is to be accurate for a reader who cannot check. P1-2. I framed three skipped rules as evidence FOR the standing no-new-rules ruling. That is the reading most convenient to me. The maintainer ruled against new mandatory steps in SKILL.md and against a commit-hook lane; I widened that into a general position and then claimed my own measurement supported it. The honest version keeps both halves. The evidence rejects MORE duplicate prose — all three rules already existed. It equally supports MECHANICALLY ENFORCING what is already written, which is a different proposition, and is exactly what #616's reconciled recommendation proposes. "Three written rules did not fire" is an argument about enforcement, not about how many rules exist. Corrected on #644 with a follow-up comment rather than an edit, so the claim and its retraction both stay in the record. test-doc-consistency 137/0. * docs(roadmap): third pass at one sentence — round 6 reviewed the REBASED tree Round 2 caught what round 1's fix still overstated. "Every leg reviewed a stale base" is false: round 6 was the rebase-confirm and reviewed the rebased tree. That leg is the whole reason the sentence exists, and the wording excluded it by asserting the opposite. The sentence now matches the record leg by leg: rounds 1-5 reviewed a stale base, 1-4 certified nothing, 5 certified one, and 6 re-reviewed the rebased tree after the merge was refused. Recorded against myself in the handoff: this took three passes because each time I fixed the exact word the reviewer quoted instead of re-deriving the claim from the record. "Every leg certified" -> "every leg reviewed" -> the actual per-round history. Two of those three passes were avoidable by checking the leg verdicts once. test-doc-consistency 137/0. * docs(roadmap): delete the leg count instead of fixing it a fourth time Counted the record properly this time: six legs with verdicts plus one transport-failed leg is seven executions, six of them on a stale base. The reviewer's count was right and mine was wrong for the third consecutive round. "every leg certified a stale base" wrong "every leg reviewed a stale base" wrong a per-round tally still miscounted Three rounds on one number is the shape signal, not an invitation to a fourth attempt. SKILL.md:170 routes a repeated shape failure to a design decision rather than another patch, and a prose claim that has survived three fixes deserves the same treatment. So the number comes out. What stays is the fact a cold reader actually needs: the branch was born stale, every review ran on a stale base until the merge was refused, and it cost a refused merge, two clearance reposts and an extra leg. The omission is marked as deliberate so nobody helpfully restores it. Same lesson as #670's deleted guard, reached four rounds slower because the surface was prose instead of a test. test-doc-consistency 137/0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
README.md's "Choosing Your Model" was 1360 words, and its largest block argued the case for Opus 4.6 over 4.8 — a debate settled two defaults ago, drawn from third-party field reports, about a configuration no cycle of this harness has ever run on. A reader had to get through all of it to find out which model to use.The dishonesty was one of proportion, not of fact. Nothing in that section was false. But the space it took implied a body of evidence behind those lanes that does not exist.
What changed
The section is now 471 words: the default, an explicit statement of what has actually been exercised, the switch commands, the lane table, and a pointer. The new honesty note says it plainly:
The 4.6/4.8 research, the per-model effort wall, and "Reading Setup A precisely" move to
AI_SETUP_LANES.mdunder "Model Selection — The Evidence". Nothing is deleted and no recommendation changes.The guards followed the content
Four guards in
test-doc-consistency.shpinned that content toREADME.md, two of them by line number. They are repointed, not relaxed — every claim is still asserted, against the file that now carries it. The position pins became content matches, which is what #659 asked for: they had already been re-anchored once and broke again on this move.Each rewritten guard was verified by mutation, not by reading:
mediumeffort defaultVerification
test-doc-consistency137/0,test-docs-usability29/0,test-cowork-drift30/0,test-workflow-triggers169/0.Merge status
Needs its own dual clearance —
README.mdships (npm includes it automatically). Seat 1 is unavailable until the Codex usage limit resets on Aug 19 21:00.