Populate /role_scripts/standby for remote replica coordinator - #62
Merged
Conversation
Signed-off-by: souravbiswassanto <saurov@appscode.com>
…imeline When a former standby is started via the primary role script (most importantly the remote-replica -> standalone-HA promotion), standby.signal is present and pg_ctl start brings postgres up in recovery. Previously start.sh removed standby.signal and then ran CREATE DATABASE / ALTER USER writes before the trailing pg_ctl promote; the writes fail under read-only recovery, so on the loop's retry postgres started directly as a primary on the EXISTING timeline and the trailing promote was a no-op. The new HA primary thus stayed on the same timeline as its old source cluster, which on failback forces a full pg_basebackup instead of pg_rewind (a whole-day op at multi-TB scale). Fix: as soon as postgres has started, if standby.signal is present, run pg_ctl promote (which ends recovery, increments the timeline, and clears standby.signal) and wait until pg_is_in_recovery() is false, BEFORE any write. The writes then run against the promoted primary on the new timeline. Scope: only affects a node started via the primary script with standby.signal present (the promotion case). A normal primary start (no standby.signal) and the live-standby fast-failover path (promoted by the coordinator via gRPC, not start.sh) are unchanged. Signed-off-by: souravbiswassanto <saurov@appscode.com>
…romotion The post-promote wait must not depend on connecting as the postgres superuser: if that role is missing/renamed the psql probe fails and the loop burns its full 120s timeout. Poll pg_controldata's cluster state (in archive recovery -> in production) instead, which needs no DB connection or role. Signed-off-by: souravbiswassanto <saurov@appscode.com>
Apply the same promote-before-writes fix already in PG17 to the other supported major versions (13,14,15,16,18) so the remote-replica -> standalone-HA timeline bump works regardless of PostgreSQL version. When standby.signal is present at startup (a former standby being promoted), pg_ctl promote runs before the CREATE DATABASE / ALTER USER writes, forking a new timeline so failback uses pg_rewind instead of a full pg_basebackup. Versions < 13 are out of scope. Signed-off-by: souravbiswassanto <saurov@appscode.com>
-X fetch collects the WAL only after the data copy finishes, so the source must retain every WAL segment generated during the entire basebackup. For a large database seeded over a WAN that window is hours to days, and a source without a generous wal_keep_size fails the seed at the very end with "requested WAL segment has already been removed". -Xs streams the WAL on a second connection concurrently with the copy, and pg_basebackup backs that stream with a temporary replication slot it creates and drops itself (default since PG 10). The seed therefore needs no WAL retention configuration on the source at all -- important when the source is a customer database we do not manage. Cost: one extra walsender for the duration of the seed. The coordinator's recovery-path basebackup (executePgBasebackup) already uses -Xs; this aligns the init-script seed with it. Signed-off-by: Tamal Saha <tamal@appscode.com>
All source-facing commands — pg_isready, the liveness psql probe, pg_basebackup, and primary_conninfo — assumed the source listens on 5432. Read PRIMARY_PORT (injected by the operator from the source AppBinding's spec.clientConfig.service.port) and default to 5432 when unset, so existing deployments and older operators are unaffected. Applied to all supported majors (13-18). Signed-off-by: Tamal Saha <tamal@appscode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
init_scripts/run.sh, theREMOTE_REPLICA=truebranch now also copies standby scripts into/role_scripts/standby/in addition to/run_scripts/role/Problem
The pg-coordinator's
AddRoleBasedScripts()restores postgres startup scripts after a crash by copying from/role_scripts/<raftRole>/(e.g./role_scripts/standby/). For HA clusters the init container populates this directory, but for remote replicas it only populated/run_scripts/role/and left/role_scripts/empty. This causedAddRoleBasedScripts()to silently fail, leaving postgres unable to restart after recovery.Test plan
AddRoleBasedScripts()and postgres restarts with the correct standby scripts