postgres: add migration guide for self-managed sources via remote replica - #1039
postgres: add migration guide for self-managed sources via remote replica#1039tamalsaha wants to merge 6 commits into
Conversation
…lica Migrating a PostgreSQL that KubeDB does not manage -- a bare StatefulSet, a VM in another datacenter, a hardened install whose superuser is never shared -- into a KubeDB cluster: seed with pg_basebackup, stream until lag is zero, stop writes verifiably, cut over. Every step was executed against two live source flavours before being written down: a stock postgres:17.4 with shared superuser credentials, and a hardened source with scram auth, a replication-only user, a restricted pg_hba.conf and no "postgres" role at all (initdb run as a different superuser). Measured write gap for the cutover was 43s (bare, from server-side row timestamps) and 37s (hardened, cutover-initiated to first accepted write). The guide leads with the limitation table: self-managed sources only, matching major version, remote-config inapplicable (hand-crafted AppBinding shown), port 5432 fixed, the postgres-role-must-exist rule with its one-line fix that replicates through WAL, libc/collation mismatch with the datcollversion remedy, extensions, tablespaces, and slotless WAL retention. Notable mechanics it documents from the live runs: reading the address to allowlist out of the source's own pg_hba rejection log (the seed retries forever, so the fix needs no restart); running the warm phase with the migration user's credentials as the authSecret so health checks pass without the source's superuser password; and the authSecret rule at cutover -- keep it if its username is postgres, remove it otherwise and let the operator generate a fresh superuser secret that promotion re-keys the copied catalog to. Signed-off-by: Tamal Saha <tamal@appscode.com>
![kodiak-appscode[bot]](https://avatars.githubusercontent.com/in/92481?s=60&v=4){kind=small}
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
💤 Files with no reviewable changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThis change adds a PostgreSQL remote physical replication migration guide and Kubernetes manifests. It covers source preparation, warm-replica deployment, lag monitoring, cutover, promotion, verification, and cleanup. ChangesPostgreSQL remote replica migration
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant PostgreSQLSource
participant KubeDB
participant pg-mig
participant Coordinator
participant MigrationOperator
MigrationOperator->>PostgreSQLSource: prepare replication and capture cutover state
KubeDB->>pg-mig: deploy remote replica
pg-mig->>PostgreSQLSource: replicate WAL changes
Coordinator->>pg-mig: monitor replay lag
MigrationOperator->>pg-mig: verify synchronization and promote
MigrationOperator->>KubeDB: redirect writes and complete cleanup
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/postgres/remote-replica/migration-yamls/pg-mig-standalone.yaml`:
- Around line 7-8: The healthChecker section in pg-mig-standalone.yaml (the
post-cutover manifest) contains disableWriteCheck: true, which was appropriate
for the warm-replica phase but should not persist after promotion to a writable
instance. Remove the disableWriteCheck setting or set it to false in this
manifest to re-enable write health checks on the promoted database.
Additionally, update migration.md at Step 6: cutover, Point 3, to explicitly
instruct readers to re-enable the write health check after promotion and not
carry the setting forward when adapting the manifest, ensuring write failures
are not masked on the production instance.
In `@docs/guides/postgres/remote-replica/migration.md`:
- Around line 250-257: The command in step 4 is hardcoded to use the pg-mig-auth
secret name, which only applies to the migration path where authSecret was
removed. Add a clarifying note after the code block explaining that users
following the "keep authSecret" path (where credentials remain in
spec.authSecret like source-pg-auth) must substitute their actual authSecret
name in place of pg-mig-auth in the kubectl get secret command.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 6081318d-3f63-4805-ba91-ed996a4433c3
📒 Files selected for processing (4)
docs/guides/postgres/remote-replica/migration-yamls/pg-mig-standalone.yamldocs/guides/postgres/remote-replica/migration-yamls/pg-mig.yamldocs/guides/postgres/remote-replica/migration-yamls/source-appbinding.yamldocs/guides/postgres/remote-replica/migration.md
| **4. Redirect writes and confirm.** The database is migrated when a write is accepted with | ||
| the final credentials: | ||
|
|
||
| ```bash | ||
| PGPASSWORD=$(kubectl get secret pg-mig-auth -n demo -o jsonpath='{.data.password}' | base64 -d) | ||
| kubectl exec -n demo pg-mig-0 -c postgres -- env PGPASSWORD="$PGPASSWORD" \ | ||
| psql -h 127.0.0.1 -U postgres -d postgres -c "SELECT NOT pg_is_in_recovery();" | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Clarify the secret name for the "keep authSecret" cutover path.
This step's command reads kubectl get secret pg-mig-auth -n demo, but that secret only exists when authSecret was removed before promotion (the hardened branch, per the rule at Line 240-243). For the bare-source branch that keeps authSecret (username postgres), the credential secret remains the original one referenced in spec.authSecret (e.g., source-pg-auth), not pg-mig-auth.
Add a short note that readers on the "keep it" branch must substitute their own authSecret name in this command.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/migration.md` around lines 250 - 257, The
command in step 4 is hardcoded to use the pg-mig-auth secret name, which only
applies to the migration path where authSecret was removed. Add a clarifying
note after the code block explaining that users following the "keep authSecret"
path (where credentials remain in spec.authSecret like source-pg-auth) must
substitute their actual authSecret name in place of pg-mig-auth in the kubectl
get secret command.
|
Visit the preview URL for this PR (updated for commit 68bb403): https://kubedb-v2-hugo--pr1039-pg-migration-remote-vokn6a88.web.app (expires Sat, 15 Aug 2026 04:45:11 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: 0f29ae8ae0bd54a99bf2b223b6833be47acd5943 |
The previous Step 6 measured downtime but never proved the migration lossless: its numbers came from runs whose writers either kept writing after "stopping" (a killed kubectl exec does not kill the server-side loop) or never wrote at all. A migration that lost in-flight rows is not a successful migration. Step 6 is now gated: - stop application writes and VERIFY the stop: pg_current_wal_lsn() AND row counts frozen across two samples - freeze a source content fingerprint: order-independent, constant-memory (sum of hashtextextended over each table), so it works at any table size - after the replica reports the frozen LSN applied, the same fingerprint on the replica must be IDENTICAL, or the cutover is aborted with the replica left safely streaming - after promotion, the fingerprint scoped to pre-cutover data must reproduce the frozen value: zero rows lost, proven, not assumed Replica-side checks run over the local socket as the source's own application user -- the replication user cannot read the application's tables and the final postgres password does not exist until promotion. Executed twice against a hardened source (no postgres role, replication-only user): once while writing the runbook, once again afterwards following the published steps verbatim. Both runs zero loss, write gaps 52.22 s and 48.57 s from server-side row timestamps. Signed-off-by: Tamal Saha <tamal@appscode.com>
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/guides/postgres/remote-replica/migration.md (1)
330-334: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winRebuild affected indexes before clearing the collation version.
Move the
REINDEX INDEX .../REINDEX DATABASEcommands beforeUPDATE pg_database SET datcollversion = NULL. If Step 5 misses an index, clearingdatcollversionfirst leaves stale collation-dependent ordering records.REINDEX DATABASEalso only covers the current database, so enumerate affected databases when more than one copied database needs remediation.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/postgres/remote-replica/migration.md` around lines 330 - 334, Reorder the migration commands so the Step 5 collation-sensitive indexes are rebuilt with REINDEX INDEX, or each affected database is rebuilt with REINDEX DATABASE, before executing the datcollversion UPDATE. When multiple copied databases require remediation, enumerate and process each database rather than relying on REINDEX DATABASE for only the current database.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/postgres/remote-replica/migration.md`:
- Around line 240-244: Change the code fence containing the bare SELECT
statements to sql, keeping the existing SQL content and comments unchanged.
- Around line 251-254: Update the step 4 fingerprint query to hash each row with
explicit, unambiguous delimiters and NULL-preserving encoding for all relevant
fields before aggregation, then compare source and replica fingerprints and fail
promotion when they differ. Do not rely on the current count plus
boundary-colliding single digest as proof of zero loss.
- Around line 303-311: Clarify the Step 6 fingerprint validation around the
migrated database: provide explicit, step-specific queries for the payload and
writes tables that exclude post-cutover rows and use the defined timestamp
boundary, while accounting for origin and ts fields included in Step 2’s hash.
Ensure the queries produce values comparable with the frozen Step 2 fingerprint
without including the first post-migration write.
- Around line 262-265: Update the WAL catch-up check around the kubectl/psql
command to use bounded polling: repeatedly query until the result is exactly
`t`, explicitly handle `NULL` or non-matching output as not ready, and exit with
failure on query errors or when the timeout is reached. Preserve the existing
namespace, pod, container, database, and frozen-LSN values.
---
Outside diff comments:
In `@docs/guides/postgres/remote-replica/migration.md`:
- Around line 330-334: Reorder the migration commands so the Step 5
collation-sensitive indexes are rebuilt with REINDEX INDEX, or each affected
database is rebuilt with REINDEX DATABASE, before executing the datcollversion
UPDATE. When multiple copied databases require remediation, enumerate and
process each database rather than relying on REINDEX DATABASE for only the
current database.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 49b0f6b0-03a6-4074-82fa-5f6800573b90
📒 Files selected for processing (1)
docs/guides/postgres/remote-replica/migration.md
| SELECT (SELECT count(*) FROM payload) | ||
| ||'|'|| (SELECT coalesce(sum(hashtextextended(id::text||data, 0)), 0) FROM payload) | ||
| ||'|'|| (SELECT count(*) FROM writes) | ||
| ||'|'|| (SELECT coalesce(sum(hashtextextended(id::text||origin||ts::text, 0)), 0) FROM writes); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
git ls-files | rg '^docs/guides/postgres/remote-replica/migration\.md$|postgres/remote-replica|migration\.md$' || true
echo "== relevant lines =="
if [ -f docs/guides/postgres/remote-replica/migration.md ]; then
nl -ba docs/guides/postgres/remote-replica/migration.md | sed -n '220,275p'
fi
echo "== search fingerprint usages/definitions =="
rg -n "hashtext|fingerprint|SELECT \(SELECT count|sum\(hashtext|count\(\*\) FROM payload|writes" docs/guides/postgres/remote-replica/migration.md || true
echo "== postgres text docs (if available locally) =="
rg -n "null handling|ignores null|concatenation\s*\|\||sum.*null" /usr/share/doc/postgresql* 2>/dev/null || trueRepository: kubedb/docs
Length of output: 1623
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== relevant migration lines =="
awk 'NR>=220 && NR<=275 {printf "%6d\t%s\n", NR, $0}' docs/guides/postgres/remote-replica/migration.md
echo "== fingerprint-related search =="
grep -Rin -n "hashtext|fingerprint|SELECT \(SELECT count|sum\(hashtext|count\(\*\) FROM payload|writes" docs/guides/postgres/remote-replica/migration.md || true
echo "== PostgreSQL docs references in local filesystem =="
grep -Rin -n "null handling|ignores null|concatenation\s*\|\||sum.*null" /usr/share/doc/postgresql* 2>/dev/null || true
echo "== behavioral probe for documented inputs =="
python3 - <<'PY'
cases = [
("1", "23"),
("12", "3"),
("1", None),
("2", "3"),
]
inputs = []
for a, b in cases:
text_a = str(a)
text_b = "" if b is None else str(b)
inputs.append(text_a + text_b)
for i, (case, inp) in enumerate(zip(cases, inputs), 1):
print(f"{i}: {case!r} -> {inp!r}")
print("boundary_loss", len(set(inputs)) != len(inputs))
print("null_sum_ignored_inputs", inputs)
PYRepository: kubedb/docs
Length of output: 3563
Use a NULL-aware, boundary-preserving fingerprint before promotion.
id::text || data does not preserve boundaries, so (1, '23') and (12, '3') feed the same string to hashtextextended(...). sum ignores NULL aggregates, so NULL data values can also make the digest collapse different table contents. Require step 4 to fail when the source and replica digests differ instead of treating the current count plus single digest as an exact zero-loss proof.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/migration.md` around lines 251 - 254,
Update the step 4 fingerprint query to hash each row with explicit, unambiguous
delimiters and NULL-preserving encoding for all relevant fields before
aggregation, then compare source and replica fingerprints and fail promotion
when they differ. Do not rely on the current count plus boundary-colliding
single digest as proof of zero loss.
| ```bash | ||
| kubectl exec -n demo pg-mig-0 -c postgres -- psql -U admin -d postgres -tAc \ | ||
| "SELECT pg_wal_lsn_diff(pg_last_wal_replay_lsn(), '<frozen-lsn>') >= 0;" # wait for: t | ||
| ``` |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file context =="
sed -n '230,285p' docs/guides/postgres/remote-replica/migration.md || true
echo "== nearby WAL-replay commands =="
rg -n "pg_last_wal_replay_lsn|pg_wal_lsn_diff|wait|timeout|replica|migrate|migration" docs/guides/postgres/remote-replica/migration.md || true
echo "== postgres docs references in repository =="
rg -n "functions-admin|pg_last_wal_replay_lsn|pg_wal_lsn_diff" . --glob '!node_modules' --glob '!dist' --glob '!build' || trueRepository: kubedb/docs
Length of output: 9042
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
path = Path('docs/guides/postgres/remote-replica/migration.md')
text = path.read_text()
blocks = re.findall(r'```(bash|sql)([\s\S]*?)```', text)
cmd_block = next((b for _, b in blocks if 'pg_last_wal_replay_lsn' in b), None)
print('cmd_block_found=', cmd_block is not None)
print('cmd_block:')
print(cmd_block)
print('contains_kubectl=', 'kubectl' in (cmd_block or ''))
print('contains_kubectl_wait=', 'kubectl wait' in (cmd_block or ''))
print('contains_sleep=', any(s in (cmd_block or '') for s in ['sleep', 'kubectl wait', 'timeout']))
print('contains_case_if_loop=', any(s in (cmd_block or '') for s in ['case ', 'if ', 'while ', 'for ', 'until ']))
print('contains_psql_error_exit=', any(s in (cmd_block or '') for s in ['set -e', '|| ', '&& ', 'exit', 'err=']))
# Probe the exact SQL shape for NULL behavior from PostgreSQL docs text only if available in repo;
# otherwise use deterministic psql exit-code semantics only: a single psql run returns 0 only
# when command output matches the query and no parse/error occurred; otherwise 1.
print('single_psql_invocation=', (cmd_block or '').count('psql') == 1)
PYRepository: kubedb/docs
Length of output: 517
🌐 Web query:
PostgreSQL pg_last_wal_replay_lsn returns NULL before recovery
💡 Result:
The PostgreSQL function pg_last_wal_replay_lsn returns NULL when the server is not in a state of recovery [1][2]. Specifically, if a PostgreSQL server is started normally (i.e., it is a primary server and has not undergone recovery), the function returns NULL because there is no WAL replay activity occurring [1][2]. The function is designed to report the last write-ahead log (WAL) location that has been replayed during a recovery process, such as on a standby server [1][3][4]. When recovery is in progress, the function returns the LSN of the last replayed record; once recovery completes (e.g., if a standby is promoted), the value typically remains static at the location of the last applied WAL record [1][2]. Note that even after a server is promoted from a standby to a primary using pg_promote, the function may continue to return the last replayed LSN rather than NULL, as the state of the server has transitioned out of recovery, but the historical record of the last replayed LSN remains stored [5]. Users encountering unexpected NULL values should verify if the server is running in recovery mode by checking pg_is_in_recovery [5].
Citations:
- 1: https://access.crunchydata.com/documentation/postgresql14/14.23/functions-admin.html
- 2: https://www.postgresql.org/docs/current/functions-admin.html
- 3: https://runebook.dev/en/docs/postgresql/functions-admin/pg_last_wal_replay_lsn
- 4: https://pgpedia.info/p/pg_last_wal_replay_lsn.html
- 5: https://www.postgresql.org/message-id/19047-0494c16d796fea52%40postgresql.org
🌐 Web query:
PostgreSQL functions-admin pg_last_wal_replay_lsn NULL before replay starts
💡 Result:
In PostgreSQL, the function pg_last_wal_replay_lsn returns NULL in several specific scenarios [1]: 1. When the server has been started normally and is not in recovery (i.e., it is a primary server) [1][2]. 2. When the server is in recovery, but no WAL records have been replayed yet [1][3][4]. The function is designed to report the location of the last WAL record replayed during recovery [1][5]. If the server is a primary (not in recovery), it does not perform WAL replay, and thus the function returns NULL [1][2]. Similarly, on a standby server, if recovery has initiated but zero WAL records have been processed, the replay location pointer (recptr) remains zero, causing the function to return NULL [3][4]. It is important to note that once a standby is promoted to a primary (e.g., via pg_promote), the server is no longer in recovery, but pg_last_wal_replay_lsn will continue to return the last LSN that was replayed before promotion rather than switching to NULL [6]. This behavior persists because the function reports the last applied record from its time as a standby, even after the node becomes a primary [6]. If you are seeing NULL unexpectedly on a standby, ensure that the standby is actively in recovery and has successfully received and applied at least one WAL record [1][7]. For monitoring purposes on a primary server, use pg_current_wal_lsn instead [2].
Citations:
- 1: https://www.postgresql.org/docs/current/functions-admin.html
- 2: https://runebook.dev/en/docs/postgresql/functions-admin/pg_last_wal_replay_lsn
- 3: https://ryogrid.github.io/create_pg_super_document/p/pg_last_wal_replay_lsn.html
- 4: https://github.com/ryogrid/create_pg_super_document/blob/master/generated_docs/p/pg_last_wal_replay_lsn.md
- 5: https://pgpedia.info/p/pg_last_wal_replay_lsn.html
- 6: https://www.postgresql.org/message-id/19047-0494c16d796fea52%40postgresql.org
- 7: https://www.postgresql.org/message-id/MA1PR0101MB1687AEC7E3ACC54F999B6E44E0E70%40MA1PR0101MB1687.INDPRD01.PROD.OUTLOOK.COM
Add bounded polling to the WAL-catch-up check.
The current psql command runs only once. It does not wait for replay to start if pg_last_wal_replay_lsn() returns NULL, and it can let the migration proceed on a query error or stale result. Add a bounded loop that polls until output is t, and exits on timeout or query failure.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/migration.md` around lines 262 - 265,
Update the WAL catch-up check around the kubectl/psql command to use bounded
polling: repeatedly query until the result is exactly `t`, explicitly handle
`NULL` or non-matching output as not ready, and exit with failure on query
errors or when the timeout is reached. Preserve the existing namespace, pod,
container, database, and frozen-LSN values.
The port-5432 limitation is lifted: the operator injects PRIMARY_PORT from spec.clientConfig.service.port into the remote replica containers, the init scripts use it for pg_isready/psql/pg_basebackup/primary_conninfo, and the coordinator uses it for its source client, pg_rewind source-server string and recovery basebackup. Defaults to 5432 everywhere when unset. Verified live: a replica seeded and streaming from a source exposed only on port 5433, with port=5433 visible in primary_conninfo and the lag monitor sampling over the same port. Signed-off-by: Tamal Saha <tamal@appscode.com>
The collation-remedy content (datcollversion clearing, REINDEX of collation-sensitive indexes) is removed and replaced by a hard requirement: the source's OS/libc must match the chosen PostgresVersion image's, verified by comparing SELECT version() on both sides -- a mismatch is not supported. Re-ran the full migration against an Alpine source matching the Official Alpine image: with matching libc no collation warning appears at all, which is also documented as the tell-tale for a mismatch. Adds a "Verifying the cutover" section with the exact procedure and real outputs from a live run with a ~5 TPS writer: sentinel-file writer stop (killing a client does not stop server-side sessions), quiesce verified by a frozen WAL position across two samples, catch-up checked with >= against the frozen LSN, zero transaction loss proven by identical count/max-id/ordered-id fingerprint on both sides (2904|2904|9b409e93...), and downtime measured from server-side row timestamps: 37.63 s. Notes the normal sequence jump across promotion so it is not mistaken for loss. Also drops the explicit healthChecker.disableWriteCheck from the replica manifest -- the health checker already handles remote replicas on its own. Signed-off-by: Tamal Saha <tamal@appscode.com>
…quirement The note still described the collation warning as harmless-and-fixed-later, which contradicted the new hard requirement one section above. It is now the mismatch tell-tale pointing back to the Step 5 check. Signed-off-by: Tamal Saha <tamal@appscode.com>
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/postgres/remote-replica/migration.md`:
- Around line 211-214: Add the `text` or `console` language tag to every
output-only Markdown fence in the migration guide, including the PostgreSQL
version blocks and the additional locations noted in the review, so all fences
satisfy markdownlint MD040.
- Around line 324-328: Update the cutover verification wording in the section
around “Verifying the cutover: zero transaction loss and measured downtime” to
use “afterward” and “arrived,” while preserving the existing meaning and
transaction details.
- Around line 201-218: Revise the “Distribution / libc match” guidance so it
does not claim that version() or the platform triple proves runtime OS, libc
version, or collation-provider compatibility. Either add explicit source and
target checks for those properties, including the collation provider, or narrow
the documented compatibility requirement and warning interpretation to the
platform information these commands actually establish.
- Around line 354-358: Update the source snapshot query near “Snapshot the
source truth” to use the same row-set scope as the replica loss check, including
the origin='writer' predicate if that is the intended migration scope. Ensure
count, max(id), and the ordered id fingerprint all operate on that identical
filtered set, or remove the target-side filter and document that all writes are
being compared.
- Around line 348-352: Update the WAL sampling command in the migration quiesce
check to explicitly target the postgres database by adding the database option
to the existing psql invocation, while preserving the current user and query
arguments.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 1f97059a-aa57-4ed0-aef3-50a3a84ac3ac
📒 Files selected for processing (2)
docs/guides/postgres/remote-replica/migration-yamls/pg-mig.yamldocs/guides/postgres/remote-replica/migration.md
💤 Files with no reviewable changes (1)
- docs/guides/postgres/remote-replica/migration-yamls/pg-mig.yaml
| **Distribution / libc match** — the source and the KubeDB image must be built against the | ||
| same libc. Compare `version()` on both sides; the platform triple must match (here: | ||
| `x86_64-pc-linux-musl` on both). If they differ, stop and pick a matching `PostgresVersion` | ||
| — a mismatch is not supported: | ||
|
|
||
| ```bash | ||
| kubectl exec -n <source-ns> <source-pod> -- psql -U <user> -d postgres -tAc "SELECT version();" | ||
| kubectl exec -n demo pg-mig-0 -c postgres -- psql -U migrator -d postgres -tAc "SELECT version();" | ||
| ``` | ||
|
|
||
| ``` | ||
| PostgreSQL 17.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit | ||
| PostgreSQL 17.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit | ||
| ``` | ||
|
|
||
| With matching distributions the replica connects without any collation-version warning — | ||
| if you see `database "postgres" has no actual collation version, but a version was | ||
| recorded` on every connection, the libc differs and this migration path does not apply. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg 'docs/guides/postgres/remote-replica/migration\.md|PostgresVersion|PostgreSQL|postgres'
echo
echo "Target context:"
sed -n '180,250p' docs/guides/postgres/remote-replica/migration.md
echo
echo "Search for migration source/target compatibility mentions:"
rg -n "matching distributions|collation-version|database \"postgres\" has no actual collation version|x86_64-pc-linux-musl|PostgresVersion|version\\(\\)" docs/guides/postgres/remote-replica -SRepository: kubedb/docs
Length of output: 23651
🌐 Web query:
PostgreSQL SQL ALTER COLLATION collation-version mismatch operating system library changes
💡 Result:
In PostgreSQL, a collation version mismatch occurs when the version of the operating system's collation provider (such as glibc or ICU) used by the database differs from the version recorded in the system catalog when the collation was created [1][2]. This mismatch is typically triggered by operating system upgrades or changes to underlying library binaries [1][3]. Because PostgreSQL relies on these providers for consistent sort orders, a change in collation definition can lead to corruption in objects that depend on collation, such as B-tree indexes, constraints, materialized views, and partitions [1][3]. To resolve a collation version mismatch, you must follow these steps: 1. Rebuild Affected Objects: Before updating the version in the system catalog, you must manually rebuild all objects dependent on the affected collation to ensure they are consistent with the new sorting rules [1][2]. This is commonly done using operations like REINDEX, REFRESH MATERIALIZED VIEW, or, depending on the object type, VACUUM FULL, TRUNCATE, or CLUSTER [4]. 2. Refresh the Collation Version: Once the objects have been rebuilt and are consistent with the new provider version, you can eliminate the mismatch warning by updating the recorded version in the system catalog [1][2]. - For a specific collation: Use the command ALTER COLLATION REFRESH VERSION [1][5]. - For the database default collation: Use the command ALTER DATABASE REFRESH COLLATION VERSION [2][5]. Note that the REFRESH VERSION command only updates the system catalog to silence the warning; it does not verify that the actual database objects have been rebuilt [1][2]. Administrators are responsible for ensuring that all dependent objects are correctly updated before executing this command [1][4]. When using the GNU C library (glibc), PostgreSQL uses the C library's version as a proxy for the collation version, which can be imprecise as distribution maintainers sometimes back-port collation changes to older library releases [1][2]. For ICU-based collations, the version information is provided directly by the ICU library [6][2].
Citations:
- 1: https://www.postgresql.org/docs/19/sql-altercollation.html
- 2: https://www.postgresql.org/docs/current/sql-altercollation.html
- 3: https://wiki.postgresql.org/wiki/Locale_data_changes
- 4: https://www.postgresql.org/message-id/CA%2BhUKGJv6V94Px59hR9KpSTJoqSP64cRxmrdqhs7EFhVODOFrA%40mail.gmail.com
- 5: https://www.postgresql.org/docs/15/sql-altercollation.html
- 6: https://www.postgresql.org/docs/18/sql-altercollation.html
Make the compatibility gate prove the stated OS/libc requirement.
The gate relies on version() and the platform triple. These commands do not inspect the source and target runtime distributions, libc versions, or collation providers. Collation-version mismatches can result from operating-system or library changes and are not specific to libc/glibc versus musl. Add explicit checks on the source and target OS, libc version, and collation provider, or narrow the requirement to what these commands actually establish.
🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 211-211: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/migration.md` around lines 201 - 218,
Revise the “Distribution / libc match” guidance so it does not claim that
version() or the platform triple proves runtime OS, libc version, or
collation-provider compatibility. Either add explicit source and target checks
for those properties, including the collation provider, or narrow the documented
compatibility requirement and warning interpretation to the platform information
these commands actually establish.
| ``` | ||
| PostgreSQL 17.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit | ||
| PostgreSQL 17.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add language tags to output fences.
markdownlint-cli2 reports MD040 for these fences. Mark output-only blocks as text or console.
Also applies to: 360-362, 378-380, 390-392
🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 211-211: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/migration.md` around lines 211 - 214, Add
the `text` or `console` language tag to every output-only Markdown fence in the
migration guide, including the PostgreSQL version blocks and the additional
locations noted in the review, so all fences satisfy markdownlint MD040.
Source: Linters/SAST tools
| ## Verifying the cutover: zero transaction loss and measured downtime | ||
|
|
||
| Run a writer against the source during the warm phase and prove afterwards that every | ||
| committed transaction arrived and how long writes were unavailable. Everything below is | ||
| from a live run: a ~5 TPS writer, 2904 transactions committed before cutover. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Correct the wording at Line [326].
Use afterward and arrived:
-prove afterwards that every committed transaction arrive
+prove afterward that every committed transaction arrived📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ## Verifying the cutover: zero transaction loss and measured downtime | |
| Run a writer against the source during the warm phase and prove afterwards that every | |
| committed transaction arrived and how long writes were unavailable. Everything below is | |
| from a live run: a ~5 TPS writer, 2904 transactions committed before cutover. | |
| ## Verifying the cutover: zero transaction loss and measured downtime | |
| Run a writer against the source during the warm phase and prove afterward that every | |
| committed transaction arrived and how long writes were unavailable. Everything below is | |
| from a live run: a ~5 TPS writer, 2904 transactions committed before cutover. |
🧰 Tools
🪛 LanguageTool
[locale-violation] ~326-~326: In American English, ‘afterward’ is the preferred variant. ‘Afterwards’ is more commonly used in British English and other dialects.
Context: ... source during the warm phase and prove afterwards that every committed transaction arrive...
(AFTERWARDS_US)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/migration.md` around lines 324 - 328,
Update the cutover verification wording in the section around “Verifying the
cutover: zero transaction loss and measured downtime” to use “afterward” and
“arrived,” while preserving the existing meaning and transaction details.
Source: Linters/SAST tools
| ```bash | ||
| kubectl exec -n <source-ns> <source-pod> -- touch /tmp/stopw | ||
| # run twice, 2s apart; proceed only when both values are identical | ||
| kubectl exec -n <source-ns> <source-pod> -- psql -U <user> -tAc "SELECT pg_current_wal_lsn();" | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate file and surrounding commands:"
wc -l docs/guides/postgres/remote-replica/migration.md 2>/dev/null || true
sed -n '320,365p' docs/guides/postgres/remote-replica/migration.md 2>/dev/null || true
echo
echo "Search nearby postgres/psql commands in the same guide:"
rg -n 'psql|SELECT pg_current_wal_lsn|CREATE DATABASE|ALTER SYSTEM|SELECT datname' docs/guides/postgres/remote-replica/migration.md
echo
echo "Check whether postgresql.org psql docs are reachable:"
python3 - <<'PY'
import urllib.request
urls = [
"https://www.postgresql.org/docs/8.3/app-psql.html",
"https://www.postgresql.org/docs/current/app-psql.html",
]
for url in urls:
try:
with urllib.request.urlopen(url, timeout=20) as r:
print(url, r.status, r.url)
except Exception as e:
print(url, type(e).__name__, e)
PYRepository: kubedb/docs
Length of output: 4306
🌐 Web query:
postgresql documentation app psql default database PSQLDATABASE environment variable -d command-line option
💡 Result:
To specify the target database when using the PostgreSQL command-line client, psql, you can use either command-line options or environment variables [1]. Command-Line Option You can specify the database name directly using the -d flag [1][2]. Example: psql -d dbname If you do not use the -d flag, psql interprets the first non-option command-line argument as the database name [1]. Example: psql dbname Environment Variable You can set the PGDATABASE environment variable to define a default database [1][3]. If no database is specified via the command line, psql will use the value of PGDATABASE [1][3]. If neither the command-line option nor the PGDATABASE environment variable is provided, psql defaults to attempting to connect to a database with the same name as your current operating system user name [1][4]. Note: There is no environment variable named PSQLDATABASE; the standard variable used by libpq (the library underlying psql) is PGDATABASE [1][3].
Citations:
- 1: https://www.postgresql.org/docs/current/app-psql.html
- 2: https://hasura.io/blog/top-psql-commands-and-flags-you-need-to-know-postgresql
- 3: https://www.postgresql.org/docs/current/libpq-envars.html
- 4: https://www.ucolick.org/~de/PG/Man/psql.1.html
🌐 Web query:
PostgreSQL app psql default behavior when no database specified current OS user same name as operating system user name
💡 Result:
When you invoke psql without specifying a database name, it follows a deterministic default process based on your current operating system user [1][2]. The default behavior is as follows: 1. Determine Database User Name: If you do not provide a database user name using the -U option or the PGUSER environment variable, psql defaults to your current operating-system user name [1][3]. 2. Determine Database Name: Once the database user name has been established, psql uses that exact same name as the default target database [1][3]. In practice, if you are logged into your operating system as "username", running the command psql will attempt to connect to a database named "username" as the database user "username" [3]. It is important to note that this is merely a default selection mechanism [1]. If a database with that name does not exist, the connection attempt will fail with a fatal error [3]. This behavior does not imply that such a database is automatically created or required to exist for other PostgreSQL operations; rather, it is a convenience shortcut provided by the client application [1][3]. You can override these defaults at any time by explicitly specifying the database name as an argument or by using command-line flags and environment variables like PGDATABASE [1][4].
Citations:
- 1: https://www.postgresql.org/docs/current/app-psql.html
- 2: https://www.postgresql.org/docs/current/tutorial-accessdb.html
- 3: https://www.postgresql.org/message-id/20170323131232.GC16353%40depesz.com
- 4: https://www.postgresql.org/docs/19/app-psql.html
Specify the database for the WAL sample.
psql -U <user> does not target postgres; without -d/PGDATABASE, it falls back to the operating-system user-based default. Add -d postgres so the quiesce check uses the same database context as the rest of the migration steps.
Proposed fix
-kubectl exec -n <source-ns> <source-pod> -- psql -U <user> -tAc "SELECT pg_current_wal_lsn();"
+kubectl exec -n <source-ns> <source-pod> -- psql -U <user> -d postgres -tAc "SELECT pg_current_wal_lsn();"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ```bash | |
| kubectl exec -n <source-ns> <source-pod> -- touch /tmp/stopw | |
| # run twice, 2s apart; proceed only when both values are identical | |
| kubectl exec -n <source-ns> <source-pod> -- psql -U <user> -tAc "SELECT pg_current_wal_lsn();" | |
| ``` |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/migration.md` around lines 348 - 352,
Update the WAL sampling command in the migration quiesce check to explicitly
target the postgres database by adding the database option to the existing psql
invocation, while preserving the current user and query arguments.
| **Snapshot the source truth** (count, highest id, and a fingerprint over every id): | ||
|
|
||
| ```bash | ||
| kubectl exec -n <source-ns> <source-pod> -- psql -U <user> -d postgres -tAc "SELECT count(*)||'|'||max(id)||'|'||md5(string_agg(id::text,',' ORDER BY id)) FROM writes;" | ||
| ``` |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
fd -a 'migration.md' . | sed 's#^\./##'
echo '--- target file excerpt ---'
sed -n '330,390p' docs/guides/postgres/remote-replica/migration.md || true
echo '--- writes origin references ---'
rg -n "origin|writes|fingerprint|highest id|max\\(id\\)|md5\\(string_agg" docs/guides/postgres/remote-replica/migration.mdRepository: kubedb/docs
Length of output: 6124
Use the same row set for the loss check.
The source fingerprint hashes every row in writes, but the replica uses WHERE origin='writer'. This can reject a lossless migration when pre-existing or other-origin rows exist. Apply the same row-set predicate to both queries, or remove the target-side filter and state the intended scope.
Proposed fix
-SELECT count(*)||'|'||max(id)||'|'||md5(string_agg(id::text,',' ORDER BY id)) FROM writes;
+SELECT count(*)||'|'||max(id)||'|'||md5(string_agg(id::text,',' ORDER BY id))
+FROM writes
+WHERE origin = 'writer';📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| **Snapshot the source truth** (count, highest id, and a fingerprint over every id): | |
| ```bash | |
| kubectl exec -n <source-ns> <source-pod> -- psql -U <user> -d postgres -tAc "SELECT count(*)||'|'||max(id)||'|'||md5(string_agg(id::text,',' ORDER BY id)) FROM writes;" | |
| ``` | |
| **Snapshot the source truth** (count, highest id, and a fingerprint over every id): | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/migration.md` around lines 354 - 358,
Update the source snapshot query near “Snapshot the source truth” to use the
same row-set scope as the replica loss check, including the origin='writer'
predicate if that is the intended migration scope. Ensure count, max(id), and
the ordered id fingerprint all operate on that identical filtered set, or remove
the target-side filter and document that all writes are being compared.
… downtime as approximate The health checker's write check is valuable on a promoted primary and the remote-replica phase no longer needs the override. Present the measured write gap as a 40-70s range across repeated runs (a clean-room re-run measured 68.45s) rather than a single number. Signed-off-by: Tamal Saha <tamal@appscode.com>
1 participant
Adds
docs/guides/postgres/remote-replica/migration.md+ manifests: migrating a PostgreSQL not managed by KubeDB (bare StatefulSet, remote-DC VM, hardened install) into a KubeDB cluster via remote replica — seed, stream, verify zero lag, cut over.Tested, not written from theory
Both source flavours ran end to end on two live clusters before the guide was written:
postgres:17.4, superuser sharedmigratoruser, restrictedpg_hba, nopostgresrole (initdbasadmin)Ready, lag monitor 5s→300s backoff observedReadywith migrator creds as authSecret (source superuser never shared)postgres)pg-mig-auth, promotion re-keyed the copied catalogWhat the guide documents that is easy to get wrong
kubectl-dba remote-configcannot target a foreign source (it execs into source pods) — the hand-crafted AppBinding is shown in fullpg_hba.confis read out of the source's own rejection log (NAT decides what it sees); the seed retries forever, so the fix requires no restartLOGIN-ablepostgresrole must exist in the source catalog before cutover — promotion connects locally as postgres to re-key passwords; the one-line fix replicates through WAL and was demonstrated livepg_current_wal_lsn()frozen across two samples — killing a client does not kill server-side sessions (observed the hard way)pg_wal_lsn_diff(...) >= 0against the frozen LSN, not equality — the source still emits checkpoint WAL after quiescingREFRESH COLLATION VERSIONfails (invalid collation version change), thedatcollversion = NULLremedy, and the pre-cutover query that counts indexes needingREINDEXLeads with a limitations table: self-managed sources only (no RDS/Cloud SQL), matching major version, port 5432 fixed, extensions, tablespaces, slotless WAL retention (the seed itself is safe —
pg_basebackup -Xsuses a self-cleaning temporary slot).Depends on the in-flight remote-replica PRs (kubedb/postgres#908, kubedb/pg-coordinator#257 incl. the new lag monitor, kubedb/postgres-init-docker#62).
Summary by CodeRabbit