From e5944991dd1e0c63eb0b2e942ebb82a838cf774e Mon Sep 17 00:00:00 2001 From: Kiran Muddukrishna Date: Sun, 23 Aug 2026 13:03:30 +1000 Subject: [PATCH 1/2] docs: split T3 marks, add online-safety column, state the Unix philosophy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A wall of โŒ read as "unsupported" when most T3 rows are scope boundaries, not gaps. T3 now carries three marks โ€” โšช (no online-safety problem to solve), ๐Ÿ”ต (a different tool class owns it), โŒ (no online mechanism exists in PostgreSQL) โ€” and every matrix table gains an "Online-safety problem?" column whose "No" cells name the tool class users should reach for instead. The same boundary is now stated loudly as the design stance it is: vision.md, README.md, and capabilities.md call out the Unix philosophy โ€” do one thing (online table-shape change under concurrent load) and do it perfectly โ€” so scope limits read as intent, not missing features. --- README.md | 7 ++ docs/capabilities.md | 150 +++++++++++++++++++++++++------------------ docs/vision.md | 14 +++- 3 files changed, 107 insertions(+), 64 deletions(-) diff --git a/README.md b/README.md index 5359207..83fa388 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,13 @@ change to the safest sequence that exists, and refuses with a structured verdict when it can't prove one (see [docs/postgres-online-ddl-reference.md](docs/postgres-online-ddl-reference.md)). +pg-sprite embraces the Unix design philosophy โ€” **do one thing, and do it +perfectly**: change the shape of live PostgreSQL tables while applications +keep reading and writing them. Anything that is not that one thing โ€” data +backfills, catalog bootstrap, GitOps orchestration, access control โ€” is +deliberately another tool's job, and the engine says so with a typed refusal +that names the tool class ([docs/capabilities.md](docs/capabilities.md)). + **Status: Phases 1 and 2.1โ€“2.5.** The parse boundary, declarative diff, classifier, router seam, versioned dry-run plan report, offline linter, and advisory `suggest` command are implemented. `pg-sprite migrate --alter 'โ€ฆ'` classifies and diff --git a/docs/capabilities.md b/docs/capabilities.md index 56d2bb1..de6a1bd 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -37,6 +37,10 @@ types it models but whether a change it accepts can hurt a production workload. positioning is [vision.md](vision.md); how it differs from planners and imperative copy tools by *problem class* is [architecture.md](architecture.md). +This is the Unix design philosophy applied to schema changes โ€” **do one thing, and do it +perfectly**. The one thing is online table-shape change under concurrent load; this whole +page is the map of where that one thing ends and another tool's job begins. + Two consequences follow, and they explain most of this page: 1. **Tables and their indexes are the model, by design.** Online safety is a @@ -57,6 +61,24 @@ Two consequences follow, and they explain most of this page: | **T2 โ€” planned** | A known online pattern exists (or requires the copy-and-swap engine); building it is on the roadmap | A **typed refusal** naming the reason, exit 2 โ€” never a silent fallback to a blocking form | | **T3 โ€” out of scope by design** | No online-safety problem to solve, or solving it belongs to a different tool class | A typed refusal or a parse-level rejection, with the reason stating *why it is not planned* | +T3 rows carry one of three marks, because they mean different things โ€” and only one of +them is a limitation: + +- **โšช no online-safety problem to solve** โ€” the operation is already safe to run + directly: transactional catalog work, or bootstrap on an object nothing reads yet. + There is nothing for an *online* engine to add; run it through owner tooling or psql. +- **๐Ÿ”ต a different tool class owns it** โ€” the job is real but belongs to another kind of + tool (data-change runners, provisioning/IaC, convergence planners, expand/contract + frameworks). The row's "Online-safety problem?" column names the class to look for. +- **โŒ no online mechanism exists** โ€” PostgreSQL itself provides no online pattern to + build on, so pg-sprite refuses rather than silently run the blocking form. These are + the only rows where "unsupported" is the honest reading. + +Every matrix table carries an **"Online-safety problem?"** column: "Yes" means there is a +readers-and-writers problem for an online engine to solve (pg-sprite solves it, plans to, +or โ€” โŒ โ€” nothing can today); "No" states which tool class users should reach for +instead. + The invariant: **every T2 row is a tracked roadmap item; T3 rows deliberately have none.** If a refusal message points at a "planned" capability, that plan exists โ€” otherwise the refusal says out-of-scope and names the tool class that owns the job. @@ -81,90 +103,92 @@ the canonical example. ## Support matrix Status legend: โœ… T1 (supported today) ยท ๐ŸŸก T2 (planned; typed refusal today) ยท -โŒ T3 (out of scope by design). +โšช T3 (out of scope; **no online-safety problem** โ€” run directly) ยท +๐Ÿ”ต T3 (out of scope; **a different tool class owns it**) ยท +โŒ T3 (out of scope; **no online mechanism exists** in PostgreSQL). ### Column changes -| Operation | Status | Behavior and why | -| --- | --- | --- | -| `ADD COLUMN` (no default, or constant default) | โœ… | Metadata-only / fast default (PG 11+); executes instantly under bounded locks | -| `ADD COLUMN` with volatile default (`now()`, `gen_random_uuid()`, โ€ฆ) | ๐ŸŸก | Table rewrite; routes to copy-and-swap and is refused until that engine lands | -| `ADD COLUMN ... GENERATED ... STORED` | ๐ŸŸก | Table rewrite; copy-and-swap route. The copy engine must **recompute, never copy,** generated columns on the shadow table | -| `ADD COLUMN` with inline `UNIQUE`/`PRIMARY KEY`/`REFERENCES`/`CHECK` | ๐ŸŸก | The inline constraint does its index build or validation scan under the `ADD COLUMN`'s `ACCESS EXCLUSIVE` lock; refused with guidance to add the column first, then build the constraint online | -| `DROP COLUMN` | โœ… | Metadata-only; flagged **destructive** in the plan report | -| `ALTER COLUMN TYPE`, binary-coercible (proven against live column facts) | โœ… | Catalog relabel, e.g. `varchar(50)` โ†’ `varchar(100)`, `varchar` โ†’ `text` | -| `ALTER COLUMN TYPE`, general (or with `USING`) | ๐ŸŸก | Table rewrite; copy-and-swap route, refused today | -| `SET DEFAULT` / `DROP DEFAULT` / `DROP NOT NULL` | โœ… | Metadata-only | -| `SET NOT NULL` | โœ… | Executed as the native four-step pattern: `ADD CONSTRAINT ... CHECK (col IS NOT NULL) NOT VALID` โ†’ online `VALIDATE` โ†’ `SET NOT NULL` (catalog flip, PG 12+) โ†’ drop the scaffold check | -| `RENAME COLUMN` / `RENAME TABLE` | โœ… | Metadata-only for PostgreSQL but **app-breaking** across deployed instances; executed with a typed reason so lint/plan consumers can steer away | -| `SET TABLESPACE` | ๐ŸŸก | Physical relocation is a rewrite; copy-and-swap route | +| Operation | Status | Online-safety problem? | Behavior and why | +| --- | --- | --- | --- | +| `ADD COLUMN` (no default, or constant default) | โœ… | Yes | Metadata-only / fast default (PG 11+); executes instantly under bounded locks | +| `ADD COLUMN` with volatile default (`now()`, `gen_random_uuid()`, โ€ฆ) | ๐ŸŸก | Yes | Table rewrite; routes to copy-and-swap and is refused until that engine lands | +| `ADD COLUMN ... GENERATED ... STORED` | ๐ŸŸก | Yes | Table rewrite; copy-and-swap route. The copy engine must **recompute, never copy,** generated columns on the shadow table | +| `ADD COLUMN` with inline `UNIQUE`/`PRIMARY KEY`/`REFERENCES`/`CHECK` | ๐ŸŸก | Yes | The inline constraint does its index build or validation scan under the `ADD COLUMN`'s `ACCESS EXCLUSIVE` lock; refused with guidance to add the column first, then build the constraint online | +| `DROP COLUMN` | โœ… | Yes | Metadata-only; flagged **destructive** in the plan report | +| `ALTER COLUMN TYPE`, binary-coercible (proven against live column facts) | โœ… | Yes | Catalog relabel, e.g. `varchar(50)` โ†’ `varchar(100)`, `varchar` โ†’ `text` | +| `ALTER COLUMN TYPE`, general (or with `USING`) | ๐ŸŸก | Yes | Table rewrite; copy-and-swap route, refused today | +| `SET DEFAULT` / `DROP DEFAULT` / `DROP NOT NULL` | โœ… | Yes | Metadata-only | +| `SET NOT NULL` | โœ… | Yes | Executed as the native four-step pattern: `ADD CONSTRAINT ... CHECK (col IS NOT NULL) NOT VALID` โ†’ online `VALIDATE` โ†’ `SET NOT NULL` (catalog flip, PG 12+) โ†’ drop the scaffold check | +| `RENAME COLUMN` / `RENAME TABLE` | โœ… | Yes | Metadata-only for PostgreSQL but **app-breaking** across deployed instances; executed with a typed reason so lint/plan consumers can steer away | +| `SET TABLESPACE` | ๐ŸŸก | Yes | Physical relocation is a rewrite; copy-and-swap route | ### Constraints -| Operation | Status | Behavior and why | -| --- | --- | --- | -| `ADD PRIMARY KEY` / `ADD UNIQUE` (plain key columns) | โœ… | Rewritten to the online sequence: `CREATE UNIQUE INDEX CONCURRENTLY` โ†’ `ADD CONSTRAINT ... USING INDEX` | -| `ADD CHECK` / `ADD FOREIGN KEY` (imperative) | โœ… | Rewritten to the online sequence: `ADD CONSTRAINT ... NOT VALID` (brief metadata lock) โ†’ `VALIDATE CONSTRAINT` (writes keep flowing during the scan) | -| `ADD CONSTRAINT ... NOT VALID` / `... USING INDEX` / `VALIDATE CONSTRAINT` | โœ… | Already the online idiom; executed as-is | -| `ADD FOREIGN KEY ... NOT VALID` on a **partitioned parent** | ๐ŸŸก | PostgreSQL supports this only from version 18; refused on 14โ€“17 | -| `EXCLUDE` constraints (and unrecognized constraint forms) | โŒ | No online pattern exists in PostgreSQL โ€” the build scans under `ACCESS EXCLUSIVE` with no `NOT VALID`/`USING INDEX` equivalent. Refused; revisit only if PostgreSQL grows one | -| `DROP CONSTRAINT` | โœ… | Metadata-only; flagged **destructive** | +| Operation | Status | Online-safety problem? | Behavior and why | +| --- | --- | --- | --- | +| `ADD PRIMARY KEY` / `ADD UNIQUE` (plain key columns) | โœ… | Yes | Rewritten to the online sequence: `CREATE UNIQUE INDEX CONCURRENTLY` โ†’ `ADD CONSTRAINT ... USING INDEX` | +| `ADD CHECK` / `ADD FOREIGN KEY` (imperative) | โœ… | Yes | Rewritten to the online sequence: `ADD CONSTRAINT ... NOT VALID` (brief metadata lock) โ†’ `VALIDATE CONSTRAINT` (writes keep flowing during the scan) | +| `ADD CONSTRAINT ... NOT VALID` / `... USING INDEX` / `VALIDATE CONSTRAINT` | โœ… | Yes | Already the online idiom; executed as-is | +| `ADD FOREIGN KEY ... NOT VALID` on a **partitioned parent** | ๐ŸŸก | Yes | PostgreSQL supports this only from version 18; refused on 14โ€“17 | +| `EXCLUDE` constraints (and unrecognized constraint forms) | โŒ | Yes โ€” unsolvable today | No online pattern exists in PostgreSQL โ€” the build scans under `ACCESS EXCLUSIVE` with no `NOT VALID`/`USING INDEX` equivalent. Refused; revisit only if PostgreSQL grows one | +| `DROP CONSTRAINT` | โœ… | Yes | Metadata-only; flagged **destructive** | ### Indexes -| Operation | Status | Behavior and why | -| --- | --- | --- | -| `CREATE [UNIQUE] INDEX` on a plain table โ€” including partial, expression, covering (`INCLUDE`), GIN/GiST/BRIN | โœ… | Executed as (or rewritten to) `CREATE INDEX CONCURRENTLY`, with validity verification and typed invalid-index outcomes ([runbook](invalid-index-recovery.md)) | -| `DROP INDEX` | โœ… | Rewritten to `DROP INDEX CONCURRENTLY`; flagged **destructive** | -| `REINDEX` | โœ… | Rewritten to `REINDEX ... CONCURRENTLY` | -| Index build on a **partitioned parent** | ๐ŸŸก | PostgreSQL has no parent-level `CONCURRENTLY`; the blocking form is refused by policy (`--force` does not bypass it). The partition-aware flow โ€” `CREATE INDEX ON ONLY` โ†’ per-partition CIC โ†’ `ATTACH PARTITION`, with crash-resume per leaf โ€” is planned | -| `ADD CONSTRAINT ... USING INDEX` on a partitioned parent | โŒ | PostgreSQL does not support adopting an index on a partitioned parent in any supported version; refused before execution | +| Operation | Status | Online-safety problem? | Behavior and why | +| --- | --- | --- | --- | +| `CREATE [UNIQUE] INDEX` on a plain table โ€” including partial, expression, covering (`INCLUDE`), GIN/GiST/BRIN | โœ… | Yes | Executed as (or rewritten to) `CREATE INDEX CONCURRENTLY`, with validity verification and typed invalid-index outcomes ([runbook](invalid-index-recovery.md)) | +| `DROP INDEX` | โœ… | Yes | Rewritten to `DROP INDEX CONCURRENTLY`; flagged **destructive** | +| `REINDEX` | โœ… | Yes | Rewritten to `REINDEX ... CONCURRENTLY` | +| Index build on a **partitioned parent** | ๐ŸŸก | Yes | PostgreSQL has no parent-level `CONCURRENTLY`; the blocking form is refused by policy (`--force` does not bypass it). The partition-aware flow โ€” `CREATE INDEX ON ONLY` โ†’ per-partition CIC โ†’ `ATTACH PARTITION`, with crash-resume per leaf โ€” is planned | +| `ADD CONSTRAINT ... USING INDEX` on a partitioned parent | โŒ | Yes โ€” unsolvable today | PostgreSQL does not support adopting an index on a partitioned parent in any supported version; refused before execution | ### Partitioned tables -| Operation | Status | Behavior and why | -| --- | --- | --- | -| `CREATE TABLE ... PARTITION OF` | โœ… | Executed, with a typed warning: creating a partition takes a brief `ACCESS EXCLUSIVE` on the **parent** and queues behind long-running queries | -| `ATTACH PARTITION` | โœ… | Executed; the safer idiom (pre-prove the bound with a validated `CHECK` so the attach skips its scan) is surfaced as guidance. A classify-first flow that constructs the proof itself is planned | -| `DETACH PARTITION [CONCURRENTLY]` | โœ… | `CONCURRENTLY` is the idiom; the blocking form is rewritten to it | -| Partitioned parents in the **declarative model** | ๐ŸŸก | Typed refusal: the model does not yet carry partition keys, and rendering a partitioned parent as a plain `CREATE TABLE` would be silently wrong | -| Partitioned tables in **copy-and-swap** | ๐ŸŸก | Root-vs-leaf publication semantics and per-partition swap; sequenced after the copy engine core | +| Operation | Status | Online-safety problem? | Behavior and why | +| --- | --- | --- | --- | +| `CREATE TABLE ... PARTITION OF` | โœ… | Yes | Executed, with a typed warning: creating a partition takes a brief `ACCESS EXCLUSIVE` on the **parent** and queues behind long-running queries | +| `ATTACH PARTITION` | โœ… | Yes | Executed; the safer idiom (pre-prove the bound with a validated `CHECK` so the attach skips its scan) is surfaced as guidance. A classify-first flow that constructs the proof itself is planned | +| `DETACH PARTITION [CONCURRENTLY]` | โœ… | Yes | `CONCURRENTLY` is the idiom; the blocking form is rewritten to it | +| Partitioned parents in the **declarative model** | ๐ŸŸก | Yes | Typed refusal: the model does not yet carry partition keys, and rendering a partitioned parent as a plain `CREATE TABLE` would be silently wrong | +| Partitioned tables in **copy-and-swap** | ๐ŸŸก | Yes | Root-vs-leaf publication semantics and per-partition swap; sequenced after the copy engine core | ### The declarative model (desired files, diff, pull) -| Table shape | Status | Behavior and why | -| --- | --- | --- | -| Plain tables + their indexes | โœ… | `diff`, `pull`, and desired-file rendering round-trip the canonical model | -| Tables that own **or are referenced by** foreign keys | ๐ŸŸก | Typed refusal on both sides โ€” an incoming FK cannot be expressed in the table's own desired file, and a lossy description would be worse than none. Declarative FK support (composite keys as the primary case, two-phase `NOT VALID` โ†’ `VALIDATE` execution) is planned | -| Unlogged tables | ๐ŸŸก | Typed refusal: persistence is not modeled, converging it (`SET LOGGED`) is a full rewrite, and rendering the table as plain `CREATE TABLE` would silently change crash-safety | -| Explicit column collations | ๐ŸŸก | Typed refusal: dropping a `COLLATE` clause from a rendered baseline silently changes sort order and index semantics; a collation delta cannot converge without a rewrite | -| Columns whose default uses a sequence the column does not own | ๐ŸŸก | Typed refusal: in a desired-state model that sequence exists only inside the scratch transaction, so no derived plan can reference it. Column-owned (`serial`-style) sequences are fine | -| Greenfield `CREATE TABLE` apply (bootstrap an empty database from desired files) | โŒ | No online-safety problem โ€” a new table has no readers or writers to protect. `diff --sql` emits the statement; applying it belongs to owner tooling or a convergence planner, not this engine | +| Table shape | Status | Online-safety problem? | Behavior and why | +| --- | --- | --- | --- | +| Plain tables + their indexes | โœ… | Yes | `diff`, `pull`, and desired-file rendering round-trip the canonical model | +| Tables that own **or are referenced by** foreign keys | ๐ŸŸก | Yes | Typed refusal on both sides โ€” an incoming FK cannot be expressed in the table's own desired file, and a lossy description would be worse than none. Declarative FK support (composite keys as the primary case, two-phase `NOT VALID` โ†’ `VALIDATE` execution) is planned | +| Unlogged tables | ๐ŸŸก | Yes | Typed refusal: persistence is not modeled, converging it (`SET LOGGED`) is a full rewrite, and rendering the table as plain `CREATE TABLE` would silently change crash-safety | +| Explicit column collations | ๐ŸŸก | Yes | Typed refusal: dropping a `COLLATE` clause from a rendered baseline silently changes sort order and index semantics; a collation delta cannot converge without a rewrite | +| Columns whose default uses a sequence the column does not own | ๐ŸŸก | Yes | Typed refusal: in a desired-state model that sequence exists only inside the scratch transaction, so no derived plan can reference it. Column-owned (`serial`-style) sequences are fine | +| Greenfield `CREATE TABLE` apply (bootstrap an empty database from desired files) | โšช | No โ€” owner tooling or a convergence planner | A new table has no readers or writers to protect. `diff --sql` emits the statement; applying it belongs to owner tooling or a convergence planner, not this engine | ### Types and non-table objects -| Object / operation | Status | Behavior and why | -| --- | --- | --- | -| Enum-typed columns on plain tables | ๐ŸŸก | Tolerance end to end (introspection already canonicalizes via `format_type`; desired-file admission and scratch-database mechanics are being verified) | -| `ALTER TYPE ... ADD VALUE` | ๐ŸŸก | Metadata-only and online-safe (PG 14+ allows it in a transaction; the value is usable after commit) โ€” planned as an owned operation. No peer online executor owns it | -| Enum value rename / removal | ๐ŸŸก | PostgreSQL has no `DROP VALUE`; this is a type swap + table rewrite โ€” routes to a typed refusal toward copy-and-swap | -| Enum/domain type creation and drop | โŒ | Bootstrap/catalog work with no concurrent-access problem; owner tooling applies it in the same change that ships the code | -| Views, materialized views | โŒ | `CREATE OR REPLACE VIEW` is transactional catalog work; no online-safety problem for an executor to own. (Materialized-view *refresh* is a data operation โ€” also out) | -| Triggers and PL/pgSQL function bodies | โŒ | Bootstrap/catalog objects with no online-safety problem; no peer online executor owns them either | -| Extensions (`CREATE EXTENSION`) | โŒ | Same: catalog bootstrap, owner tooling | -| Grants, roles, row-level-security policies | โŒ | Access control, not table shape; belongs to provisioning (see [engine-role.md](engine-role.md) for what the *engine's own* role needs) | -| Standalone sequences, publications/subscriptions | โŒ | Not table shape; no online pattern to provide | +| Object / operation | Status | Online-safety problem? | Behavior and why | +| --- | --- | --- | --- | +| Enum-typed columns on plain tables | ๐ŸŸก | Yes | Tolerance end to end (introspection already canonicalizes via `format_type`; desired-file admission and scratch-database mechanics are being verified) | +| `ALTER TYPE ... ADD VALUE` | ๐ŸŸก | Yes | Metadata-only and online-safe (PG 14+ allows it in a transaction; the value is usable after commit) โ€” planned as an owned operation. No peer online executor owns it | +| Enum value rename / removal | ๐ŸŸก | Yes | PostgreSQL has no `DROP VALUE`; this is a type swap + table rewrite โ€” routes to a typed refusal toward copy-and-swap | +| Enum/domain type creation and drop | โšช | No โ€” owner tooling (psql, shipped with the code change) | Bootstrap/catalog work with no concurrent-access problem; owner tooling applies it in the same change that ships the code | +| Views, materialized views | โšช | No โ€” owner tooling | `CREATE OR REPLACE VIEW` is transactional catalog work; no online-safety problem for an executor to own. (Materialized-view *refresh* is a data operation โ€” also out) | +| Triggers and PL/pgSQL function bodies | โšช | No โ€” owner tooling | Bootstrap/catalog objects with no online-safety problem; no peer online executor owns them either | +| Extensions (`CREATE EXTENSION`) | โšช | No โ€” owner tooling | Same: catalog bootstrap, owner tooling | +| Grants, roles, row-level-security policies | ๐Ÿ”ต | No โ€” provisioning / IaC | Access control, not table shape; belongs to provisioning (see [engine-role.md](engine-role.md) for what the *engine's own* role needs) | +| Standalone sequences, publications/subscriptions | โšช | No โ€” owner tooling / replication provisioning | Not table shape: sequence DDL is transactional catalog work, and publications/subscriptions are replication provisioning โ€” owner tooling applies both | ### Data and whole-table operations -| Operation | Status | Behavior and why | -| --- | --- | --- | -| Data backfills, `UPDATE`/`DELETE` batches, DML of any kind | โŒ | pg-sprite changes table *shape*, never table *contents*. Versioned-script runners and application jobs own data changes | -| Column-transform expressions during a copy-and-swap rewrite | ๐ŸŸก | The one principled exception: when a rewrite is already copying every row, deriving a new column's value by expression is part of the shape change, not a data job. Planned as part of the copy engine | -| Online table rebuild with no shape change (bloat reclamation) | ๐ŸŸก | A copy-and-swap with an identical target shape โ€” the pg_repack use case with checksum-gated cutover and crash-resume. Planned once the copy engine lands | -| Whole-schema convergence (apply a directory of desired files, dependency-ordered) | โŒ | Convergence planning across objects is a planner's job (pg-schema-diff, pgschema, pgdelta); pg-sprite stays the execution engine for the table-shape subset | -| Versioned schema-change-file workflow (Flyway-style ordered scripts) | โŒ | Declarative-only by design; see [vision.md](vision.md) | -| Expand/contract dual-schema versions (pgroll/reshape style) | โŒ | Rejected: application invisibility is a core invariant; see [vision.md](vision.md) | +| Operation | Status | Online-safety problem? | Behavior and why | +| --- | --- | --- | --- | +| Data backfills, `UPDATE`/`DELETE` batches, DML of any kind | ๐Ÿ”ต | No โ€” data-change runners, application batch jobs | pg-sprite changes table *shape*, never table *contents*. Versioned-script runners and application jobs own data changes | +| Column-transform expressions during a copy-and-swap rewrite | ๐ŸŸก | Yes | The one principled exception: when a rewrite is already copying every row, deriving a new column's value by expression is part of the shape change, not a data job. Planned as part of the copy engine | +| Online table rebuild with no shape change (bloat reclamation) | ๐ŸŸก | Yes | A copy-and-swap with an identical target shape โ€” the pg_repack use case with checksum-gated cutover and crash-resume. Planned once the copy engine lands | +| Whole-schema convergence (apply a directory of desired files, dependency-ordered) | ๐Ÿ”ต | No โ€” convergence planners (pg-schema-diff, pgschema, pgdelta) | Convergence planning across objects is a planner's job; pg-sprite stays the execution engine for the table-shape subset | +| Versioned schema-change-file workflow (Flyway-style ordered scripts) | ๐Ÿ”ต | No โ€” versioned-script runners (Flyway-style) | Declarative-only by design; see [vision.md](vision.md) | +| Expand/contract dual-schema versions (pgroll/reshape style) | ๐Ÿ”ต | No โ€” pgroll/reshape own this model | Rejected: application invisibility is a core invariant; see [vision.md](vision.md) | ## Peers share these limits โ€” for different reasons diff --git a/docs/vision.md b/docs/vision.md index 0865271..cd61c38 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -17,6 +17,16 @@ checksum before any destructive step, a durable checkpoint so a mid-change crash instead of orphaning, and managed-platform failover (Aurora) treated as a modeled state rather than an unhandled surprise. +**pg-sprite embraces the Unix design philosophy: do one thing, and do it perfectly.** The +one thing is changing the *shape* of live PostgreSQL tables while applications keep +reading and writing them. Everything inside that problem โ€” classification, online idioms, +rewrites, verification, crash-resume โ€” belongs in this engine and is done to the safety +bar below. Everything outside it โ€” data changes, catalog bootstrap, GitOps orchestration, +access control โ€” is deliberately another tool's job, and the engine says so with a typed +refusal that names the tool class ([capabilities.md](capabilities.md)). "One tool for all +schema changes" and "do one thing" are the same claim read from two sides: depth across +every table-shape change, never sprawl across object types. + ## The five pillars ### 1. All schema changes, one engine @@ -160,7 +170,9 @@ pg-sprite has succeeded when: ## What pg-sprite is not -Scope honesty keeps the vision credible: pg-sprite is not the GitOps layer itself โ€” it does +This section is the do-one-thing philosophy applied โ€” every "not" below is the same +boundary drawn from a different angle. Scope honesty keeps the vision credible: +pg-sprite is not the GitOps layer itself โ€” it does not watch git, manage pull requests, or schedule fleets; it is the engine that layer (SchemaBot) drives. It is not an application-rollout coordinator (the multi-version schema window some expand/contract tools offer is a real capability we deliberately trade away From d6ce6beb2808664066512b85b47b09ced59a12ac Mon Sep 17 00:00:00 2001 From: Kiran Muddukrishna Date: Mon, 24 Aug 2026 16:10:48 +1000 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20tighten=20T3=20marks=20per=20review?= =?UTF-8?q?=20=E2=80=94=20lock=20caveats,=20row=20splits?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The โšช mark overclaimed: triggers, view replaces, and greenfield FKs take brief locks on live tables that queue behind long-running queries. Tighten the definition to no-scan/no-rewrite, add queue caveats, split rows that carried one mark for two reasons, and stop claiming the engine names the tool class (this page does; refusals are undifferentiated today). --- README.md | 5 +++-- docs/capabilities.md | 35 +++++++++++++++++++++++------------ docs/vision.md | 8 ++++---- 3 files changed, 30 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 83fa388..409c7b2 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,9 @@ pg-sprite embraces the Unix design philosophy โ€” **do one thing, and do it perfectly**: change the shape of live PostgreSQL tables while applications keep reading and writing them. Anything that is not that one thing โ€” data backfills, catalog bootstrap, GitOps orchestration, access control โ€” is -deliberately another tool's job, and the engine says so with a typed refusal -that names the tool class ([docs/capabilities.md](docs/capabilities.md)). +deliberately another tool's job: the engine refuses it with a typed verdict, +and [docs/capabilities.md](docs/capabilities.md) names the tool class that +owns each job. **Status: Phases 1 and 2.1โ€“2.5.** The parse boundary, declarative diff, classifier, router seam, versioned dry-run plan report, offline linter, and diff --git a/docs/capabilities.md b/docs/capabilities.md index de6a1bd..d246de5 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -59,14 +59,17 @@ Two consequences follow, and they explain most of this page: | --- | --- | --- | | **T1 โ€” supported today** | The engine executes the change through an online-safe pattern | Execution (or the safer rewritten sequence), exit 0 | | **T2 โ€” planned** | A known online pattern exists (or requires the copy-and-swap engine); building it is on the roadmap | A **typed refusal** naming the reason, exit 2 โ€” never a silent fallback to a blocking form | -| **T3 โ€” out of scope by design** | No online-safety problem to solve, or solving it belongs to a different tool class | A typed refusal or a parse-level rejection, with the reason stating *why it is not planned* | +| **T3 โ€” out of scope by design** | No online-safety problem to solve, solving it belongs to a different tool class, or PostgreSQL offers no online mechanism to build on | A typed refusal or a parse-level rejection, with the reason stating *why it is not planned* | T3 rows carry one of three marks, because they mean different things โ€” and only one of them is a limitation: -- **โšช no online-safety problem to solve** โ€” the operation is already safe to run - directly: transactional catalog work, or bootstrap on an object nothing reads yet. - There is nothing for an *online* engine to add; run it through owner tooling or psql. +- **โšช no online-safety problem to solve** โ€” the operation does no table scan and no + rewrite; at most it takes a brief catalog lock. There is nothing for an *online* + engine to add; run it through owner tooling or psql. Where that brief lock lands on a + **live** table (a trigger, a view swap, a greenfield foreign key), the row says so: + the statement queues behind long-running queries and blocks sessions behind it while + it waits, so run it under a `lock_timeout`. - **๐Ÿ”ต a different tool class owns it** โ€” the job is real but belongs to another kind of tool (data-change runners, provisioning/IaC, convergence planners, expand/contract frameworks). The row's "Online-safety problem?" column names the class to look for. @@ -81,7 +84,9 @@ instead. The invariant: **every T2 row is a tracked roadmap item; T3 rows deliberately have none.** If a refusal message points at a "planned" capability, that plan exists โ€” -otherwise the refusal says out-of-scope and names the tool class that owns the job. +otherwise the change is out of scope by design, and this page โ€” not the refusal text, +which today is one undifferentiated unsupported-statement reason for everything outside +the imperative front door โ€” names the tool class that owns the job. ## The two front doors @@ -102,6 +107,9 @@ the canonical example. ## Support matrix +**51 operations: 17 supported today, 18 planned behind a typed refusal, 14 out of scope +by design, and 2 with no online mechanism in PostgreSQL to build on.** + Status legend: โœ… T1 (supported today) ยท ๐ŸŸก T2 (planned; typed refusal today) ยท โšช T3 (out of scope; **no online-safety problem** โ€” run directly) ยท ๐Ÿ”ต T3 (out of scope; **a different tool class owns it**) ยท @@ -163,7 +171,7 @@ Status legend: โœ… T1 (supported today) ยท ๐ŸŸก T2 (planned; typed refusal today | Unlogged tables | ๐ŸŸก | Yes | Typed refusal: persistence is not modeled, converging it (`SET LOGGED`) is a full rewrite, and rendering the table as plain `CREATE TABLE` would silently change crash-safety | | Explicit column collations | ๐ŸŸก | Yes | Typed refusal: dropping a `COLLATE` clause from a rendered baseline silently changes sort order and index semantics; a collation delta cannot converge without a rewrite | | Columns whose default uses a sequence the column does not own | ๐ŸŸก | Yes | Typed refusal: in a desired-state model that sequence exists only inside the scratch transaction, so no derived plan can reference it. Column-owned (`serial`-style) sequences are fine | -| Greenfield `CREATE TABLE` apply (bootstrap an empty database from desired files) | โšช | No โ€” owner tooling or a convergence planner | A new table has no readers or writers to protect. `diff --sql` emits the statement; applying it belongs to owner tooling or a convergence planner, not this engine | +| Greenfield `CREATE TABLE` apply (the table does not exist yet โ€” a fresh database or a new table in a live one) | โšช | No โ€” owner tooling or a convergence planner | The new table has no readers or writers to protect, but a `REFERENCES` clause takes a brief `SHARE ROW EXCLUSIVE` on each **referenced** table and queues behind long-running queries โ€” run it under a `lock_timeout`. `diff --sql` emits the statement; applying it belongs to owner tooling or a convergence planner, not this engine | ### Types and non-table objects @@ -173,11 +181,14 @@ Status legend: โœ… T1 (supported today) ยท ๐ŸŸก T2 (planned; typed refusal today | `ALTER TYPE ... ADD VALUE` | ๐ŸŸก | Yes | Metadata-only and online-safe (PG 14+ allows it in a transaction; the value is usable after commit) โ€” planned as an owned operation. No peer online executor owns it | | Enum value rename / removal | ๐ŸŸก | Yes | PostgreSQL has no `DROP VALUE`; this is a type swap + table rewrite โ€” routes to a typed refusal toward copy-and-swap | | Enum/domain type creation and drop | โšช | No โ€” owner tooling (psql, shipped with the code change) | Bootstrap/catalog work with no concurrent-access problem; owner tooling applies it in the same change that ships the code | -| Views, materialized views | โšช | No โ€” owner tooling | `CREATE OR REPLACE VIEW` is transactional catalog work; no online-safety problem for an executor to own. (Materialized-view *refresh* is a data operation โ€” also out) | -| Triggers and PL/pgSQL function bodies | โšช | No โ€” owner tooling | Bootstrap/catalog objects with no online-safety problem; no peer online executor owns them either | +| Views, materialized views (create and replace) | โšช | No โ€” owner tooling | Transactional catalog work, but `CREATE OR REPLACE VIEW` takes a brief `ACCESS EXCLUSIVE` on the view and queues behind in-flight readers โ€” run it under a `lock_timeout` | +| `REFRESH MATERIALIZED VIEW` | ๐Ÿ”ต | No โ€” data jobs / owner tooling | A data operation, not catalog work: the plain form holds `ACCESS EXCLUSIVE` on the matview for the whole rebuild (`CONCURRENTLY` needs a unique index and trades the lock for churn). Scheduling refreshes belongs to data jobs | +| PL/pgSQL function bodies (`CREATE OR REPLACE FUNCTION`) | โšช | No โ€” owner tooling | Transactional catalog work that takes no lock on any relation; nothing for an online engine to add. No peer online executor owns it either | +| Triggers (`CREATE TRIGGER`) | โšช | No โ€” owner tooling | Catalog work โ€” no scan, no rewrite โ€” but it takes a brief `SHARE ROW EXCLUSIVE` on the table, queues behind long-running queries, and blocks writers while it waits โ€” run it under a `lock_timeout` | | Extensions (`CREATE EXTENSION`) | โšช | No โ€” owner tooling | Same: catalog bootstrap, owner tooling | | Grants, roles, row-level-security policies | ๐Ÿ”ต | No โ€” provisioning / IaC | Access control, not table shape; belongs to provisioning (see [engine-role.md](engine-role.md) for what the *engine's own* role needs) | -| Standalone sequences, publications/subscriptions | โšช | No โ€” owner tooling / replication provisioning | Not table shape: sequence DDL is transactional catalog work, and publications/subscriptions are replication provisioning โ€” owner tooling applies both | +| Standalone sequences | โšช | No โ€” owner tooling | Transactional catalog work on an object with no readers-and-writers problem | +| Publications, subscriptions | ๐Ÿ”ต | No โ€” replication provisioning / IaC | Replication provisioning, not table shape (`ALTER PUBLICATION ... ADD TABLE` also takes `SHARE UPDATE EXCLUSIVE` on the table) | ### Data and whole-table operations @@ -211,9 +222,9 @@ line sits where it does, and what happens when you cross it: is cheap when you don't own what happens under concurrent load. pg-sprite's position: model narrowly, execute what the model covers with provable online -safety, and make every boundary a **typed refusal that states its tier** โ€” planned (with -a real plan) or out-of-scope (with the tool class that owns the job). The scope limit is -explicit, documented on this page, and machine-checkable via exit codes. +safety, and make every boundary a **typed refusal**, with this page stating its tier โ€” +planned (with a real plan) or out of scope (with the tool class that owns the job). The +scope limit is explicit, documented on this page, and machine-checkable via exit codes. ## Why typed refusal, not passthrough diff --git a/docs/vision.md b/docs/vision.md index cd61c38..ab175eb 100644 --- a/docs/vision.md +++ b/docs/vision.md @@ -22,10 +22,10 @@ one thing is changing the *shape* of live PostgreSQL tables while applications k reading and writing them. Everything inside that problem โ€” classification, online idioms, rewrites, verification, crash-resume โ€” belongs in this engine and is done to the safety bar below. Everything outside it โ€” data changes, catalog bootstrap, GitOps orchestration, -access control โ€” is deliberately another tool's job, and the engine says so with a typed -refusal that names the tool class ([capabilities.md](capabilities.md)). "One tool for all -schema changes" and "do one thing" are the same claim read from two sides: depth across -every table-shape change, never sprawl across object types. +access control โ€” is deliberately another tool's job: the engine refuses it with a typed +reason, and [capabilities.md](capabilities.md) names the tool class that owns each job. +"One tool for all schema changes" and "do one thing" are the same claim read from two +sides: depth across every table-shape change, never sprawl across object types. ## The five pillars