Skip to content

G6 complete: Phase B measured (Phase A was survivorship bias) + the missing deployment docs - #24

Merged
SebastianKrn merged 3 commits into
masterfrom
g6-phase-b-repair
Aug 2, 2026
Merged

G6 complete: Phase B measured (Phase A was survivorship bias) + the missing deployment docs#24
SebastianKrn merged 3 commits into
masterfrom
g6-phase-b-repair

Conversation

@SebastianKrn

Copy link
Copy Markdown
Contributor

Closes milestone G6. Two independent tracks.

Track A — G6 Phase B

Amendment A2 fixed the method before any repair ran: mutually blind parallel repairers, repair may not add a test the model did not write, null output is ABANDONED at zero and must not be averaged in, truncated output is salvaged not completed.

Phase A Phase B
green cells 12/24 21/24
test methods 154 (3 red) 421 (0 red)
line / branch coverage 100 % / 100 % 90.5 % / 78.8 %
mutation score 99.2 % 73.2 %

The quality metrics got worse, and that is the finding. Phase A's perfect figures were computed only over the cells that happened to compile — exclusively the small controllers. The God class was never in the denominator. Repair puts it in. Phase A was survivorship bias, visible only because the protocol forced every failed cell to be measured to the end.

Repaired, the God class reaches ~83 % line coverage at a 44–56 % mutation score — half the injected faults survive the one class the experiment is about. A coverage gate would have passed it.

Also measured:

  • one model wrote 134 test methods where the other wrote 13, at byte-identical value (21/21 lines, 13/13 mutants);
  • zero BUG-FOUND across 15 wrong-expectation repairs — these tests pin behaviour, they do not audit it;
  • the open-weight model emitted getStatusCodeValue(), present in corpus A's Spring 4.3 and removed in corpus B's Spring 7 — migrating to a very new stack makes LLM support temporarily worse.

Two defects in our own work, disclosed not fixed away:

  • A3 — one cell in 24 was decided by our extraction rule, not the model: it wrote a draft, rejected it in plain text, then wrote a correct 26-test class; our "first fenced block" rule kept the draft. Not re-extracted; 12/24 stands. The failure taxonomy is corrected instead.
  • A4 — the repair-effort clock is contaminated by A2.1's own parallelism (one cell: 24.7 wall-clock minutes for a build that worked 5.6 s). Minutes published as a measured upper bound; the transferable figures are the 52 fixes and their categories.

Adoption (ADR-0011): one class per unit, highest mutation score, ties to fewer test methods — the rule that rejects the 134-method class, and which selected a mixed set (2 frontier, 4 open-weight). 88 methods adopted → 99 module tests green, coverage 37.3 % → 81.3 %, ratchet 0.35 → 0.80. §3's 80 % target reached rather than declared for the first time; branch coverage still lags 23 points and the adopted God-class suite kills 44.1 % — both stated next to the number.

Track B — the deployment docs

ENGINEERING_STANDARDS.md §7 has always required docs/deployment.md. It did not exist — and the gap was not in DEVIATIONS.md either. Written now, every command executed before being written down, plus docs/MANUAL_TASKS.md as the by-hand checklist. Production deployment gets no invented steps: §10 states what does not exist.

Beginner-fatal fixes: --wait was missing from every human quickstart while all three CI workflows always used it; characterization silently ignored -Dtarget, so -Dtarget=modern went green against the legacy stand — now a fail-fast; modern verify needs a Docker daemon and nothing said so. Corrected: the legacy/ build failure is not the Java 8 source level — on JDK 26 compile and surefire succeed and it dies in maven-war-plugin:2.6 reflecting into java.util internals JDK 16 sealed.

Verification

char 47/47 vs legacy AND modern · e2e 34/34 vs legacy AND modern · harness 24/24 · both testbed validations · modern verify 99 tests at the 0.80 gate. The safety net is untouched.

🤖 Generated with Claude Code

SebastianKrn and others added 3 commits August 2, 2026 10:47
…tered

Phase A is published, Phase B has not started. PROTOCOL.md §6 fixed WHAT is
repaired, the 30-minute cap and the six categories, but left four operational
questions open — and each of them changes what the repair-effort number means.
Deciding them in the executing session instead of here would have been exactly
the post-hoc curation this protocol exists to prevent.

- A2.1 parallel, mutually blind repairers, one agent instance per cell: a serial
  repairer accumulates familiarity, so cell ORDER becomes a hidden variable.
  Cost stated, not hidden: the numbers are per-cell COLD-START effort, an upper
  bound on one cell in isolation, never a batch estimate for a human team.
- A2.2 repair may only repair — no test method the model did not write. Without
  this Phase B measures the repairer, not the generator.
- A2.3 a cell whose recorded output is `null` is ABANDONED at 0 minutes, and the
  report must not average it in: zero-because-unrepairable is not a cheap cell.
- A2.4 truncated output is salvaged (keep complete methods, drop the tail),
  never completed.

Nothing is re-run, re-prompted or re-recorded; no Phase-A number changes.
Harness 24/24 green after the edit (the prompt-drift test reads §4, untouched).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ENGINEERING_STANDARDS §7 has required `docs/deployment.md` since the standard was
written. The file did not exist — and the gap was not in DEVIATIONS.md either,
which is worse: the ledger's whole purpose is that there are no silent holes. It
was found by a doc audit, not by the ledger. Both halves are now closed.

New:
- `docs/deployment.md` — prerequisites per module (the JDK matrix existed nowhere
  as a table), first run, both stands, the database, every suite, reproducing CI,
  troubleshooting. Every command in it was executed against this repo before being
  written down. §10 states plainly that production deployment does NOT exist and
  invents no steps for it.
- `docs/MANUAL_TASKS.md` — the human-only steps as checklists: once per machine,
  per session, per PR, per stage, GitHub admin, the API key, golden re-capture.

Beginner-fatal defects fixed, all verified rather than assumed:
- **`--wait` was missing from every human-facing quickstart** while all three CI
  workflows have always used it. Copy-paste the docs and the next suite run hits a
  stand that is up but not healthy, failing on timing in a way that reads like a
  test bug. Fixed in README, CLAUDE.md and all four module READMEs.
- **characterization silently ignored `-Dtarget`.** `-Dtarget=modern` went GREEN
  against the LEGACY stand — an equivalence proof that proved nothing. A net that
  can be aimed at the wrong stand without saying so is not a net, so it now fails
  fast with the three flags it actually needs. Verified: guard fires, normal runs
  stay green.
- **`./mvnw verify -f modern/pom.xml` needs a running Docker daemon** (Testcontainers)
  and nothing said so.
- characterization README started only the legacy stand, then targeted the modern one.
- `legacy/` build failure explained correctly for the first time: measured on JDK 26,
  `compile` and `surefire` SUCCEED — it dies in `maven-war-plugin:2.6`, which reflects
  into `java.util` internals JDK 16 sealed. It was never the Java 8 source level. That
  is the project's thesis in miniature: the blocker is the build plugins.
- `modern/frontend/README.md` was unedited CLI boilerplate telling you to run `ng test`
  with Vitest — neither is installed. Rewritten, and the absence of frontend unit tests
  is now stated as a position rather than left to be discovered.
- stale: "module tests arrive with G6" (they exist since 2026-07-31) in CLAUDE.md,
  SPEC and modern-ci; SPEC's "three workflows" (there are four, supplying three of the
  seven required checks); the selector-map comment claiming the Angular Testability API
  (the app is zoneless — Testability observes nothing); `.env.example`'s cost estimate
  (€1.20–1.50 estimated vs €0.65 actually spent).

Grounded in official Docker, Adoptium/Maven, Selenium, PostgreSQL, GitHub and
npm/OpenRouter documentation; claims those sources could not confirm were dropped
rather than filled in from memory.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…bias

Amendment A2 fixed the Phase-B method BEFORE any repair ran: mutually blind
parallel repairers, repair-may-not-add-tests, `null` output is ABANDONED at zero
and must not be averaged in, truncated output is salvaged not completed.

11 cells repaired, one isolated agent each, 30-minute cap, live fix log. A2.2 was
verified mechanically rather than asserted: @test counts identical in all 24 cells,
sole exception the A2.4 salvage that dropped one cut-off method.

                     Phase A          Phase B
  green cells        12/24            21/24
  test methods       154 (3 red)      421 (0 red)
  line / branch      100% / 100%      90.5% / 78.8%
  mutation score     99.2%            73.2%

The quality metrics got WORSE, and that is the result. Phase A's perfect figures
were computed only over the cells that happened to compile — exclusively the small
controllers; the God class was not in the denominator. Repair puts it in. Phase A
was survivorship bias, visible only because the protocol forced every failed cell
to be measured to the end. Repaired, the God class reaches ~83% line coverage at a
44-56% mutation score: half the injected faults survive the one class this whole
experiment is about. A coverage gate would have passed it.

Three more findings: one model wrote 134 test methods where the other wrote 13, at
byte-identical measured value (21/21 lines, 13/13 mutants); 15 wrong-expectation
repairs found ZERO real defects in production code — these tests pin behaviour, they
do not audit it; and the open-weight model emitted getStatusCodeValue(), present in
corpus A's Spring 4.3 and REMOVED in corpus B's Spring 7 — migrating to a very new
stack makes LLM support temporarily worse.

Two defects in our own work, disclosed rather than quietly fixed:
- A3: one cell in 24 was decided by our extraction rule, not the model. It wrote a
  draft, rejected it in plain text, then wrote a correct 26-test class; our "first
  fenced block" rule kept the draft. NOT re-extracted — a rule does not become wrong
  because it cost a point. 12/24 stands; the REPORT's taxonomy row is corrected.
- A4: the repair-effort clock is contaminated by A2.1's own parallelism (one cell:
  24.7 wall-clock minutes for a build that worked 5.6 seconds). Minutes published as
  a measured upper bound; the transferable figures are the 52 fixes and categories.

Adoption (ADR-0011): one class per unit, highest mutation score, ties to FEWER test
methods — the rule that rejects the 134-method class. Selected a mixed set (2 frontier,
4 open-weight). 88 methods adopted, 99 module tests green, coverage 37.3% -> 81.3%,
ratchet 0.35 -> 0.80. §3's 80% target reached rather than declared for the first time;
stated next to it that branch coverage still lags 23 points and the adopted God-class
suite kills only 44.1% of mutants.

Verification: char 47/47 vs legacy AND modern - e2e 34/34 vs legacy AND modern -
harness 24/24 - both testbed validations - modern verify 99 tests at the 0.80 gate.
The safety net is untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@SebastianKrn
SebastianKrn merged commit c856a0f into master Aug 2, 2026
7 checks passed
@SebastianKrn
SebastianKrn deleted the g6-phase-b-repair branch August 2, 2026 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant