replay: corpus replay pattern — buzz history through pg-sprite, generalizable to any project

[Kiran01bm]

Summary

Adds replay/ — a throwaway-Docker pattern that replays a real project's schema-change history through pg-sprite from scratch, asserting per statement that pg-sprite either executes the change for real or produces exactly the expected typed refusal (a reason mismatch is a failure, not a pass). The machinery is project-agnostic; the first project is the SQLx migration history of block/buzz (range partitioning, composite FKs, enums, generated TSVECTOR columns, GIN/partial indexes, PL/pgSQL triggers), replayed 0002 → 0032 with zero mismatches.

What

A replay project is defined by three inputs — repository, pinned commit, migrations path — captured in replay/<project>/project.conf alongside the corpus file list and a curated manifest:

• init.sh <project> <repo> <ref> <path> — scaffolds a new project from those inputs: resolves the ref to a commit, captures the corpus file list at that pin, picks a free host port, writes project.conf, .gitignore (corpus is fetched, never vendored), and a skeleton assessment.tsv.
• fetch.sh <project> — downloads the corpus at the pinned commit (explicit file list, never a directory scrape — the corpus stays synchronized with the assessment written against it). fetch.sh <project> refresh [ref] reports files beyond the pin via the GitHub API without touching anything; picking up new history is a deliberate pin-bump + re-curation.
• harness.sh <project> up|reset|psql|dsn|down — the project's pinned postgres image with state confined to the container; up applies the project baseline via psql (bootstrap DDL on an empty database has no online-safety problem), reset returns to exactly that state. Per-project ports let harnesses coexist.
• assessment.tsv — one row per replay step, <migration> <line-range> <execute|refuse:<reason>|psql> [class]. Line ranges index into the pinned corpus, so a pin bump forces re-curation — an assessment never silently applies to a corpus it was not written against. The optional class on refuse rows separates three very different refusals: capability boundary (default — pg-sprite is expected to handle this eventually), no-online-safety-problem (bootstrap CREATE TABLE — nothing reads the object yet, so there is no concurrent-access problem for an online engine to solve), and by-design (refused deliberately because a safer form exists).
• make replay [REPLAY_PROJECT=<project>] — the one-shot: build the binary, fetch the pinned corpus, start (or reset) the project's container, replay. make replay-refresh reports corpus drift; make replay-down removes the container. Default project: buzz.
• replay.sh <project> — starts or resets the harness to the pristine baseline, then walks the manifest in strict corpus order:
  • execute steps run pg-sprite migrate --alter '…' --url … --json and require exit 0 with outcome executed-natively; pg-sprite itself mutates the database — real execution, not dry-run.
  • refuse:<reason> steps require exit 2 and exactly that reason, then psql-apply the same statement so the remaining history replays against true state.
  • psql steps (data changes, PL/pgSQL functions/triggers, dynamic DO blocks, session LOCK/SET LOCAL) are applied via psql in one transaction and never assessed.
  • Ends with a per-statement results table and a bucket summary; exits non-zero on any mismatch.

Why

The Go suite tests the engine against schemas written for the tests. Replaying an independent project's full history measures something different: how much of a real service's schema-change workload lands in each support tier, with the refusals firing for the right reasons. Buzz is public, so the whole harness lives in the open — and any other project can be assessed the same way from its repo URL, a commit, and a migrations path.

Buzz replay result at the pinned corpus (94 steps, 0 mismatches)

== bucket summary
executed natively by pg-sprite (T1)                  37
refusal: capability boundary (T2)                    6
  not-native-safe-rewrite-required                   3
  unsupported-partitioned-parent                     2
  backend-unavailable                                1
refusal: no online-safety problem (bootstrap DDL)    25
refusal: by design (safer form exists)               1
out-of-scope content, psql only (T3)                 25
mismatches                                           0

Two findings surfaced by curating against real verdicts (kept as manifest comments):

• CREATE INDEX IF NOT EXISTS is refused by design — a name-only no-op cannot prove the existing index is valid or even the requested one.
• Buzz's unnamed ADD FOREIGN KEY is refused (not-native-safe-rewrite-required) while the same constraint written with CONSTRAINT <name> executes via the safer NOT VALID + VALIDATE sequence — the rewrite needs a name to reference.

Known follow-up: the refusal class lives in the manifest, not the verdict

The class split above is curated — the engine returns the same machine-readable reason (unsupported-statement) for a data backfill, a bootstrap CREATE TABLE, and CREATE INDEX IF NOT EXISTS, so a machine consumer parsing --json cannot distinguish "wrong tool class" from "missing capability". A verdict-taxonomy change will follow as its own PR (additive class field on the refusal verdict, exit codes unchanged, capabilities/limitations/README/demo updated together per the capability-statement rule); once the engine emits the class, the replay will assert it from the verdict instead of carrying its own column.

Before / after

before: engine validated only against test-authored schemas
after:  replay/ replays any pinned public corpus end to end:
        init.sh <project> <repo> <ref> <path> ──▶ project.conf (pin + file list)
        fetch.sh <project> ──▶ corpus/ ──▶ harness.sh <project> up ──▶ baseline
                                                │
        assessment.tsv ──▶ replay.sh <project> ─┴─▶ per-statement verdict assertions
             execute  ──▶ pg-sprite runs the change (exit 0, executed-natively)
             refuse:r ──▶ exact typed refusal (exit 2) ──▶ psql advance
             psql     ──▶ out-of-scope content applied, never assessed

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[aparajon]

🤖 Adversarial correctness review, requested by @aparajon and performed by their agent. Reviewed at head a30c984, in a worktree, against a live PostgreSQL — I ran the whole thing: cold docker pull, fetch, replay, 37 / 32 / 25 with 0 mismatches, exit 0, reproducing the summary in the body exactly. Then I attacked the harness's own integrity claims rather than the engine's, because that is where a replay tool can lie to you.

Verdict: the assertions are genuinely strict — a wrong reason fails, a refusal advances state only on exit 2, and a mid-plan failure never gets blindly re-applied — and the buzz curation is meticulous (I verified programmatically that its 94 ranges tile every substantive line of 0002–0032 with no overlaps and no gaps). Two things to fix, both in the machinery rather than the assessment. The pin does not identify the corpus on disk: I bumped COMMIT to forty zeros, re-ran fetch.sh, and it made zero network calls, kept a byte-identical corpus, and printed corpus complete: 2 files at block/buzz@000000000000. And nothing checks that the manifest covers the corpus, so an omitted range is silently unassessed — which matters because the tier counts are the deliverable. Both are cheap. Neither is a reason to hold the PR if you'd rather land the pattern and follow up, but the README currently states the first as a guarantee.

Findings

1. fetch.sh caches by filename, so the pin records what you last typed, not what the corpus is. Line 59 short-circuits on [ -s corpus/$f ] before any network call, so a pin bump re-fetches only new filenames and leaves every existing file at the old pin's content — then the banner reports the whole corpus as being at the new commit. Demonstrated end to end:

$ ./fetch.sh zzpin                      # COMMIT=a2d8be5efa12
fetch 0001_initial_schema.sql
fetch 0013_push_endpoint_state.sql
corpus complete: 2 files at block/buzz@a2d8be5efa12

$ sed -i '' 's/^COMMIT=.*/COMMIT="000…000"/' zzpin/project.conf
$ ./fetch.sh zzpin
have  0001_initial_schema.sql
have  0013_push_endpoint_state.sql
corpus complete: 2 files at block/buzz@000000000000     # commit does not exist
                                                        # corpus md5 unchanged

This is exactly the failure the README says is prevented by design — "an assessment must never silently apply to a corpus it was not written against" (README line 99) and "the pin in project.conf records exactly which commit the assessment ran against" (line 121). Today the coupling is aspirational: what actually protects you is that stale line ranges will probably produce mismatches, which is a smoke alarm, not an interlock. And the documented refresh procedure — bump COMMIT, extend FILES, re-fetch — is precisely the sequence that produces the mixed corpus, since the pre-existing files are the ones the cache keeps. The fix is small: write the resolved commit (ideally with per-file sha256) to corpus/.pin at fetch time; re-fetch everything when it differs from COMMIT, and have replay.sh refuse to run when corpus/.pin and COMMIT disagree. That turns the sentence in the README into something the code enforces.

2. Nothing checks that the manifest tiles the corpus, and the uncovered case is silent. replay.sh walks the rows it is given; a statement no row claims is never assessed, never counted, and leaves no trace in the results table. Since the output is the measurement — "how much of a real service's workload lands in each tier" — a curator who skips the hard statements gets a green run and a summary that quietly understates the workload, which is the one failure mode this tool cannot afford. Worth saying clearly: buzz's manifest does not have this problem. I reconstructed coverage from the pinned corpus and every non-blank, non-comment line of 0002 through 0032 is claimed by exactly one range, ranges never overlap, and they advance monotonically within each file, with only the baseline 0001 outside — which is correct. That is what makes the check cheap to add and free to pass: about twenty lines that assert each file's ranges are non-overlapping, ordered, and cover every substantive line, run once before the replay starts. Right now that property is a fact about the curator, not about the tool, and the PR's headline is that the tool generalizes.

3. A divergence mid-run keeps producing verdicts, and the engine's own explanation is thrown away. The baseline path already refuses to replay a wrong world — "a results table that looks legitimate but assesses the wrong world" (replay.sh line 55) — and dies. The same hazard arrives mid-loop with a continue: a psql step that fails to apply (line 83) and an execute step that fails with exit 1 (line 114's deliberate non-advance) both leave the database missing content the rest of the corpus assumes, and the loop keeps assessing against it. I truncated one psql range so it would fail:

0006  15-40   psql     apply-error            FAIL  CREATE TABLE moderation_reports …
0006  53-54   execute  unparseable (exit 1)   FAIL  CREATE INDEX idx_moderation_reports…
0006  56-58   execute  unparseable (exit 1)   FAIL  …
0006  59-61   execute  unparseable (exit 1)   FAIL  …
0006  63-64   execute  unparseable (exit 1)   FAIL  …
0006 128-130  refuse:not-native-safe-rewrite-required  unparseable (exit 1)  FAIL
0029 263-575  psql     apply-error            FAIL  INSERT INTO _ope…

One root cause, seven mismatches, a cascade reaching a migration twenty-three files later — and a complete-looking bucket summary of 33 / 30 / 24 computed against a world where moderation_reports never existed. The exit code is 1, so a scripted caller is safe; a human reading the table is not, and those tier counts are the quotable artifact. I don't think the answer is to die — during curation you want the whole table in one pass — but once the database has diverged from the corpus's true state, the rows after it are unassessable and should be labelled and excluded from the buckets rather than counted as evidence.

The second half of that output is its own defect: six rows say unparseable because line 89 discards stderr with 2>/dev/null, and on exit 1 pg-sprite writes nothing to stdout. What it does write is exactly what you need — pg-sprite: error: table not found: moderation_reports is not visible on the session search_path. A harness whose purpose is to explain mismatches should capture stderr and print it on FAIL rows.

4. (nit) wait_ready can pass against the postgres image's init server. pg_isready runs over the container's unix socket, which the entrypoint's temporary init server answers while listen_addresses is still empty; the real server starts afterwards, so up can race into apply_baseline around the restart. pg_isready -h localhost inside the container is only true once the real server is listening, which closes it without a sleep.

5. (nit) init.sh picks an unclaimed port, not a free one — max(PORT)+1 across existing project.conf files, with no check against the host. The summary says "picks a free host port"; docker run will fail loudly, so this is a wording fix or a one-line nc -z probe.

Action items

1. (Finding 1) Record the resolved commit (and ideally per-file digests) in corpus/.pin at fetch time; re-fetch when it differs from COMMIT, and have replay.sh refuse a corpus whose recorded pin doesn't match. Then the README's two guarantees are enforced rather than intended.
2. (Finding 2) Add a pre-flight coverage assertion — ranges non-overlapping, ordered, and covering every substantive line of every non-baseline corpus file — so a manifest gap fails the run instead of shrinking the buckets.
3. (Finding 3) Stop counting rows that follow a divergence: mark them unassessable and keep them out of the bucket summary (or stop the run), and capture pg-sprite's stderr so a FAIL row shows the engine's message instead of unparseable.
4. (Findings 4, 5) Wait on TCP readiness inside the container; either probe the host port in init.sh or soften the "free port" wording.
5. (optional) The replay surfaced an engine message bug worth fixing on its own: the unnamed ADD FOREIGN KEY refusal advises "submit each operation as its own single-operation statement so the engine can build its safer form" — but the statement already is one, and the real blocker is the missing constraint name, which buzz/README.md diagnoses correctly. That detail sends an operator down a path that cannot work.
6. (optional) shellcheck -S warning is already clean on all five scripts apart from sourced-library false positives in common.sh (add # shellcheck shell=bash and the SC2034 export note), so wiring it into make lint costs almost nothing and this PR triples the repo's shell surface.

Verified (tried to break, couldn't)

The headline reproduces exactly: from a cold postgres:17-alpine pull through 94 steps, I got 37 executed / 32 typed refusals (26 unsupported-statement, 3 not-native-safe-rewrite-required, 2 unsupported-partitioned-parent, 1 backend-unavailable) / 25 psql-only, 0 mismatches, exit 0 — and those numbers match the manifest's own expectation counts field-for-field, so the body is not rounding anything. The strictness claims hold: a wrong reason is a FAIL because the comparison is on the full refused:<reason> string, not the exit code; unparseable JSON is a FAIL rather than a skip; and the state-advance logic is careful in both directions — it advances only on exit 2 (a true refusal executed nothing) and deliberately refuses to re-apply after exit 1, where a committed prefix could exist, which is the same committed-prefix hazard #53's loop reasons about. --single-transaction means a failed psql step rolls back whole rather than half-landing. Both boundary facts in buzz/README.md are independently true — I probed a fresh table pair and the unnamed FK refuses not-native-safe-rewrite-required while the identical constraint with CONSTRAINT <name> executes as ADD … NOT VALID then VALIDATE CONSTRAINT, and CREATE INDEX IF NOT EXISTS refuses unsupported-statement with the name-only-no-op detail. The throwaway story is real: all state is container-confined, the database name rides psql's :"db" identifier quoting instead of string splicing (so a hyphenated project survives), the project name is charset-bounded before it becomes a container name and a filename, down is total, and corpus/ is gitignored rather than vendored. go build ./... passes at head, shellcheck is clean, there are zero test deletions or weakened assertions, CI is green 12/12 across PostgreSQL 14–18, and the leak check passes. Process note: the only bot comment is Codex reporting it is out of quota, so no automated review ran — same as #52, #53, and #55.

This review was generated by Claude Code (claude-opus-5).

[aparajon]

🤖 Second pass, same head (a30c984), through the two lenses @aparajon asks pg-sprite changes to be judged on: how easily an outside team adopts this, and the seam an orchestrator embedding the engine consumes. Correctness findings are in the comment above; nothing here blocks.

Lens 1 — OSS adoption

This is the first artifact in the repo that answers an evaluator's actual first question — "will it handle my schema?" — with a number instead of a promise, and that changes what pg-sprite can claim in public. Every other form of evidence here is the project grading its own homework: the Go suite tests the engine against schemas written for the tests, and the demo tours a schema chosen to tour well. Replaying somebody else's real history, curated against observed verdicts rather than predictions, is the one form of evidence a skeptical reader can't discount — and pinning it to a public commit means they can re-run it and get the same table. The design choices that make it credible are the unglamorous ones: real execution rather than dry-run, an exact reason match so a refusal that lands for the wrong cause is a failure, and psql-advancing after each refusal so statement 90 is assessed against a database that actually has statements 1–89 in it.

The one change I'd make before this becomes the number people quote: the bucket summary undersells the engine by about four times, and the fix is regrouping, not engineering. The headline reads 37 executed against 32 typed refusals, which invites the conclusion that pg-sprite handles a bit over half of a real workload. I broke down what those 32 refusals actually are:

refusal	count	what it means to an adopter
unsupported-statement — CREATE TABLE	25	bootstrap DDL, no online-safety problem to solve; outside migrate's intake by design
unsupported-statement — CREATE INDEX IF NOT EXISTS	1	refused by design; a name-only no-op proves nothing
not-native-safe-rewrite-required	3	real gap (and one of the three is the missing-constraint-name case)
unsupported-partitioned-parent	2	real gap, planned capability
backend-unavailable	1	real gap, pending copy-and-swap

So of the 69 table-shape statements buzz put in front of the engine, 62 are either executed online today or bootstrap DDL that is safe by definition, and six are genuine capability gaps — three of which are one feature. That is a far stronger and equally honest story than "32 refusals", and the current grouping actively hides it, because the biggest bucket in the refusal column is the one that isn't a limitation at all. Splitting the summary into executed / safe by definition, outside intake / capability gap / out of scope would make the table say what the data says. The README already explains this distinction in prose (lines 23–29) — the summary just doesn't reflect it.

Second adoption note: the fastest way for another team to reject this is the curation cost, and nothing sets that expectation. The scripts scaffold in one command and the docs are genuinely good, but the actual work is hand-writing 94 line ranges against a corpus and iterating until green — and the ranges are line numbers, so they are re-derived by hand on every pin bump. That is fine for a flagship exemplar and worth every hour it took here; it is a surprise to somebody who reads "assess your own project" and expects to spend an afternoon. Either say so in README.md ("expect a few hours per thousand lines of history"), or reduce it — a --suggest mode that walks a corpus file, splits it into statements, probes each one through migrate --json, and emits a candidate manifest with the observed verdicts pre-filled would turn curation into review-and-correct. The curator's judgment is still needed to mark psql steps and sanity-check the reasons, but the line-range bookkeeping is mechanical, and it is the part that has to be redone every time the pin moves.

Lens 2 — the seam an orchestrator consumes

Whether or not it was meant as one, this is now the repo's only end-to-end test of the CLI contract an embedder actually integrates against — exit codes and --json outcome/reason — and it is explicitly excluded from CI. Everything an orchestrator binds to is asserted here on real input: exit 0 with executed-natively, exit 2 with a specific reason, and stdout being parseable JSON in both cases. That is a compatibility suite. And README.md says, correctly for today, that it is "not part of make test or CI — the Go suite remains the correctness oracle." The risk that creates is the usual one for out-of-band suites: a reason rename or an exit-code change stays green everywhere that runs automatically, and the replay only notices the next time somebody thinks to run it. The pin is what makes the fix easy — the corpus is deterministic, so the run is reproducible — and this doesn't belong on every PR (docker plus network plus a few minutes), but a scheduled or manually-dispatched job would keep the contract honest at nearly no cost. Related: #53 added verdict.Reasons() and a docs test that pins every reason token to a row in the refusal table. The replay now pins a subset of those same tokens from the outside, so a rename has two places to fail — worth connecting them deliberately (assert the manifest's refuse:<reason> values are all members of Reasons()) rather than leaving two independent copies to drift.

The deeper seam question is which front door this measures. The replay drives migrate --alter, one statement at a time, with the harness supplying the ordering, the state advance, and the decision about what is out of scope. That is the imperative door — and after #53, it is not the door a real adopter uses. The declarative loop is the product story ("give us the file and we'll converge the table"), and the interesting version of this measurement is the one that hands RunDesired a desired-state file and asks what fraction of a real history it converges without an operator in the loop. It would also exercise the parts of #53 that the current tests only cover one at a time: committed-prefix semantics over a long plan, the admission gates against real drift, and the pinned-fingerprint retry. I'd expect it to score worse than the imperative number — the whole-plan destructive refusal and the size guard bite harder when a file is the unit — and that gap is the single most useful thing this harness could measure, because it is the distance between what the engine can do and what a team can adopt. The 25 CREATE TABLE refusals are a preview: through the declarative door those aren't refusals at all, they're just table creation, which is why the two doors would produce genuinely different tier counts on the same corpus.

One thing worth keeping exactly as it is: the choice to curate against observed verdicts rather than predicted ones, and to write the two surprises down in buzz/README.md as boundary facts. I re-derived both independently and they hold — and the unnamed-ADD FOREIGN KEY one is the more valuable of the two, because it caught a real defect in the engine's own advice (noted in the comment above). That is the pattern paying for itself on its first project: not "the engine passed", but "the engine's error message is wrong, and we only found out by feeding it somebody else's schema."

This review was generated by Claude Code (claude-opus-5).

[aparajon]

🤖 Approved. I ran the replay end to end at head a30c984 — cold container pull through 94 steps, 37 / 32 / 25 with 0 mismatches, matching the manifest's own expectation counts field-for-field — and independently re-derived both boundary facts in buzz/README.md. The assertions are strict in the ways that matter, and the buzz curation tiles the corpus exactly. Findings are non-blocking and both concern the machinery rather than the assessment: the pin does not identify the corpus on disk (fetch.sh caches by filename), and nothing checks that the manifest covers the corpus. Details: #58 (comment)

This review was generated by Claude Code (claude-opus-5).