G6 complete: Phase B measured (Phase A was survivorship bias) + the missing deployment docs - #24
Merged
Merged
Conversation
…tered Phase A is published, Phase B has not started. PROTOCOL.md §6 fixed WHAT is repaired, the 30-minute cap and the six categories, but left four operational questions open — and each of them changes what the repair-effort number means. Deciding them in the executing session instead of here would have been exactly the post-hoc curation this protocol exists to prevent. - A2.1 parallel, mutually blind repairers, one agent instance per cell: a serial repairer accumulates familiarity, so cell ORDER becomes a hidden variable. Cost stated, not hidden: the numbers are per-cell COLD-START effort, an upper bound on one cell in isolation, never a batch estimate for a human team. - A2.2 repair may only repair — no test method the model did not write. Without this Phase B measures the repairer, not the generator. - A2.3 a cell whose recorded output is `null` is ABANDONED at 0 minutes, and the report must not average it in: zero-because-unrepairable is not a cheap cell. - A2.4 truncated output is salvaged (keep complete methods, drop the tail), never completed. Nothing is re-run, re-prompted or re-recorded; no Phase-A number changes. Harness 24/24 green after the edit (the prompt-drift test reads §4, untouched). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ENGINEERING_STANDARDS §7 has required `docs/deployment.md` since the standard was written. The file did not exist — and the gap was not in DEVIATIONS.md either, which is worse: the ledger's whole purpose is that there are no silent holes. It was found by a doc audit, not by the ledger. Both halves are now closed. New: - `docs/deployment.md` — prerequisites per module (the JDK matrix existed nowhere as a table), first run, both stands, the database, every suite, reproducing CI, troubleshooting. Every command in it was executed against this repo before being written down. §10 states plainly that production deployment does NOT exist and invents no steps for it. - `docs/MANUAL_TASKS.md` — the human-only steps as checklists: once per machine, per session, per PR, per stage, GitHub admin, the API key, golden re-capture. Beginner-fatal defects fixed, all verified rather than assumed: - **`--wait` was missing from every human-facing quickstart** while all three CI workflows have always used it. Copy-paste the docs and the next suite run hits a stand that is up but not healthy, failing on timing in a way that reads like a test bug. Fixed in README, CLAUDE.md and all four module READMEs. - **characterization silently ignored `-Dtarget`.** `-Dtarget=modern` went GREEN against the LEGACY stand — an equivalence proof that proved nothing. A net that can be aimed at the wrong stand without saying so is not a net, so it now fails fast with the three flags it actually needs. Verified: guard fires, normal runs stay green. - **`./mvnw verify -f modern/pom.xml` needs a running Docker daemon** (Testcontainers) and nothing said so. - characterization README started only the legacy stand, then targeted the modern one. - `legacy/` build failure explained correctly for the first time: measured on JDK 26, `compile` and `surefire` SUCCEED — it dies in `maven-war-plugin:2.6`, which reflects into `java.util` internals JDK 16 sealed. It was never the Java 8 source level. That is the project's thesis in miniature: the blocker is the build plugins. - `modern/frontend/README.md` was unedited CLI boilerplate telling you to run `ng test` with Vitest — neither is installed. Rewritten, and the absence of frontend unit tests is now stated as a position rather than left to be discovered. - stale: "module tests arrive with G6" (they exist since 2026-07-31) in CLAUDE.md, SPEC and modern-ci; SPEC's "three workflows" (there are four, supplying three of the seven required checks); the selector-map comment claiming the Angular Testability API (the app is zoneless — Testability observes nothing); `.env.example`'s cost estimate (€1.20–1.50 estimated vs €0.65 actually spent). Grounded in official Docker, Adoptium/Maven, Selenium, PostgreSQL, GitHub and npm/OpenRouter documentation; claims those sources could not confirm were dropped rather than filled in from memory. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…bias Amendment A2 fixed the Phase-B method BEFORE any repair ran: mutually blind parallel repairers, repair-may-not-add-tests, `null` output is ABANDONED at zero and must not be averaged in, truncated output is salvaged not completed. 11 cells repaired, one isolated agent each, 30-minute cap, live fix log. A2.2 was verified mechanically rather than asserted: @test counts identical in all 24 cells, sole exception the A2.4 salvage that dropped one cut-off method. Phase A Phase B green cells 12/24 21/24 test methods 154 (3 red) 421 (0 red) line / branch 100% / 100% 90.5% / 78.8% mutation score 99.2% 73.2% The quality metrics got WORSE, and that is the result. Phase A's perfect figures were computed only over the cells that happened to compile — exclusively the small controllers; the God class was not in the denominator. Repair puts it in. Phase A was survivorship bias, visible only because the protocol forced every failed cell to be measured to the end. Repaired, the God class reaches ~83% line coverage at a 44-56% mutation score: half the injected faults survive the one class this whole experiment is about. A coverage gate would have passed it. Three more findings: one model wrote 134 test methods where the other wrote 13, at byte-identical measured value (21/21 lines, 13/13 mutants); 15 wrong-expectation repairs found ZERO real defects in production code — these tests pin behaviour, they do not audit it; and the open-weight model emitted getStatusCodeValue(), present in corpus A's Spring 4.3 and REMOVED in corpus B's Spring 7 — migrating to a very new stack makes LLM support temporarily worse. Two defects in our own work, disclosed rather than quietly fixed: - A3: one cell in 24 was decided by our extraction rule, not the model. It wrote a draft, rejected it in plain text, then wrote a correct 26-test class; our "first fenced block" rule kept the draft. NOT re-extracted — a rule does not become wrong because it cost a point. 12/24 stands; the REPORT's taxonomy row is corrected. - A4: the repair-effort clock is contaminated by A2.1's own parallelism (one cell: 24.7 wall-clock minutes for a build that worked 5.6 seconds). Minutes published as a measured upper bound; the transferable figures are the 52 fixes and categories. Adoption (ADR-0011): one class per unit, highest mutation score, ties to FEWER test methods — the rule that rejects the 134-method class. Selected a mixed set (2 frontier, 4 open-weight). 88 methods adopted, 99 module tests green, coverage 37.3% -> 81.3%, ratchet 0.35 -> 0.80. §3's 80% target reached rather than declared for the first time; stated next to it that branch coverage still lags 23 points and the adopted God-class suite kills only 44.1% of mutants. Verification: char 47/47 vs legacy AND modern - e2e 34/34 vs legacy AND modern - harness 24/24 - both testbed validations - modern verify 99 tests at the 0.80 gate. The safety net is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes milestone G6. Two independent tracks.
Track A — G6 Phase B
Amendment A2 fixed the method before any repair ran: mutually blind parallel repairers, repair may not add a test the model did not write,
nulloutput isABANDONEDat zero and must not be averaged in, truncated output is salvaged not completed.The quality metrics got worse, and that is the finding. Phase A's perfect figures were computed only over the cells that happened to compile — exclusively the small controllers. The God class was never in the denominator. Repair puts it in. Phase A was survivorship bias, visible only because the protocol forced every failed cell to be measured to the end.
Repaired, the God class reaches ~83 % line coverage at a 44–56 % mutation score — half the injected faults survive the one class the experiment is about. A coverage gate would have passed it.
Also measured:
BUG-FOUNDacross 15 wrong-expectation repairs — these tests pin behaviour, they do not audit it;getStatusCodeValue(), present in corpus A's Spring 4.3 and removed in corpus B's Spring 7 — migrating to a very new stack makes LLM support temporarily worse.Two defects in our own work, disclosed not fixed away:
Adoption (ADR-0011): one class per unit, highest mutation score, ties to fewer test methods — the rule that rejects the 134-method class, and which selected a mixed set (2 frontier, 4 open-weight). 88 methods adopted → 99 module tests green, coverage 37.3 % → 81.3 %, ratchet 0.35 → 0.80. §3's 80 % target reached rather than declared for the first time; branch coverage still lags 23 points and the adopted God-class suite kills 44.1 % — both stated next to the number.
Track B — the deployment docs
ENGINEERING_STANDARDS.md§7 has always requireddocs/deployment.md. It did not exist — and the gap was not inDEVIATIONS.mdeither. Written now, every command executed before being written down, plusdocs/MANUAL_TASKS.mdas the by-hand checklist. Production deployment gets no invented steps: §10 states what does not exist.Beginner-fatal fixes:
--waitwas missing from every human quickstart while all three CI workflows always used it; characterization silently ignored-Dtarget, so-Dtarget=modernwent green against the legacy stand — now a fail-fast; modernverifyneeds a Docker daemon and nothing said so. Corrected: thelegacy/build failure is not the Java 8 source level — on JDK 26compileandsurefiresucceed and it dies inmaven-war-plugin:2.6reflecting intojava.utilinternals JDK 16 sealed.Verification
char 47/47 vs legacy AND modern · e2e 34/34 vs legacy AND modern · harness 24/24 · both testbed validations · modern
verify99 tests at the 0.80 gate. The safety net is untouched.🤖 Generated with Claude Code