postgres: add zero-data-loss cross-cluster replication guide - #1038
postgres: add zero-data-loss cross-cluster replication guide#1038tamalsaha wants to merge 2 commits into
Conversation
Adds docs/guides/postgres/remote-replica/synchronous.md and its manifests: a
synchronous remote-replica setup where every acknowledged commit is already
flushed in the DR cluster.
The quorum is FIRST 2 ("pg-london-0","pg-singapore-0","pg-singapore-1") over a
2-replica primary, so whichever Singapore pod holds the primary role the two
selected standbys are always one local and one cross-cluster, with
commitLevel "On" requiring a flush on both.
Verified end to end on two clusters: identical content fingerprint on the DR
site after destroying the primary with deletionPolicy Delete, with
pg_is_in_recovery() still true -- no promotion involved.
Documents three things that are easy to get wrong:
- synchronous must be enabled only after the DR replica is streaming; with
numSyncReplicas 2 and one standby every commit blocks, including the writes
KubeDB performs during bootstrap
- commitLevel must be quoted, since bare On is a YAML 1.1 boolean and the
admission webhook rejects it
- re-running in a namespace that previously used a different CA fails with
"could not accept SSL connection: tlsv1 alert unknown ca", because
cert-manager reuses a still-valid leaf Secret instead of reissuing when the
CA Issuer's backing secret is replaced
Signed-off-by: Tamal Saha <tamal@appscode.com>
![kodiak-appscode[bot]](https://avatars.githubusercontent.com/in/92481?s=60&v=4){kind=small}
📝 WalkthroughWalkthroughAdds a complete guide and Kubernetes manifests for PostgreSQL synchronous replication between Singapore and London. The changes configure shared TLS, authentication, remote replicas, synchronous quorum, replication checks, zero-data-loss validation, operations, and cleanup. ChangesSynchronous PostgreSQL replication
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant kubectl-dba
participant pg-singapore
participant pg-london
Operator->>pg-singapore: Apply primary PostgreSQL resource
kubectl-dba->>pg-singapore: Generate remote-replica configuration
Operator->>pg-london: Apply London remote-replica resource
pg-singapore->>pg-london: Stream WAL
Operator->>pg-singapore: Configure synchronous standby quorum
pg-singapore->>pg-london: Require flushed-WAL acknowledgement
Operator->>pg-singapore: Delete the primary cluster and PVCs
pg-london-->>Operator: Report matching data while remaining in recovery
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
docs/guides/postgres/remote-replica/synchronous.md (1)
56-500: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd a language hint to output-only fenced code blocks.
Several fenced code blocks that show command output (Lines 56, 104, 116, 177, 245, 268, 293, 358, 420, 438, 474, 500) have no language specified, which markdownlint-cli2 flags under MD040. Add
textorconsoleto these blocks for lint compliance and consistent rendering.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/postgres/remote-replica/synchronous.md` around lines 56 - 500, Update the output-only fenced code blocks in this guide, including the blocks showing command results and query output, to specify an appropriate language hint such as text or console. Leave executable bash and YAML fences unchanged, and ensure every output-only fence is MD040-compliant.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@docs/guides/postgres/remote-replica/synchronous.md`:
- Around line 56-500: Update the output-only fenced code blocks in this guide,
including the blocks showing command results and query output, to specify an
appropriate language hint such as text or console. Leave executable bash and
YAML fences unchanged, and ensure every output-only fence is MD040-compliant.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 1cd29594-99b8-4893-beb2-454229fb8b04
📒 Files selected for processing (5)
docs/guides/postgres/remote-replica/synchronous-yamls/pg-issuer.yamldocs/guides/postgres/remote-replica/synchronous-yamls/pg-london.yamldocs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-sync.yamldocs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore.yamldocs/guides/postgres/remote-replica/synchronous.md
…lerance The two-replica layout has no slack: the primary has exactly two standbys and numSyncReplicas is 2, so losing either blocks writes until it returns. Recommend replicas: 3 with a four-name standbyNames list. It costs no extra pods -- replicas: 2 is even and needs a coordinator-only arbiter to break Raft ties, replicas: 3 is odd and needs none -- so both run three pods, except with three replicas the third holds data and can serve reads. Verified on two clusters: the fourth name sits at sync_state potential, and deleting the active local sync standby promotes it automatically with no operator action. Writes measured 0.09s with a standby down against 0.12s at full quorum, and the row was on the DR site before the commit returned. Also records what three replicas does not buy: if London is the standby that fails, FIRST 2 falls back to two local standbys and the zero-RPO guarantee is silently downgraded to single-cluster durability. That is inherent to FIRST N with a list longer than N. Documents both resolutions and adds an alerting note on sync_state, which is the only signal that cross-cluster durability was lost. Signed-off-by: Tamal Saha <tamal@appscode.com>
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-3.yaml`:
- Line 9: Update clientAuthMode from md5 to scram in
docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-3.yaml:9-9
and
docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-sync-3.yaml:9-9.
In both manifests, update spec.version from 17.4 to a patched PostgreSQL 17
release or later, and rotate the configured passwords so no role retains an MD5
verifier.
In `@docs/guides/postgres/remote-replica/synchronous.md`:
- Around line 544-551: Add the text language tag to both fenced output blocks in
the synchronous replication example, including the block around the replica
status table and the additional block referenced near lines 556–561, while
leaving their contents unchanged.
- Around line 553-554: Update the standby description around `pg-singapore-2` to
avoid equating `sync_state = potential` or `state = streaming` with readiness
for promotion or synchronous use. Add the relevant LSN/lag comparison
demonstrating that the standby is caught up before claiming it can be promoted
without operator action; otherwise, revise the wording to describe only its
candidate and WAL sender states.
- Around line 586-588: Update the synchronous replication alert guidance to
check for the presence of a connected `pg-london-0` row, requiring
`application_name = 'pg-london-0'`, `state = 'streaming'`, and `sync_state =
'sync'`, rather than relying only on a `sync_state` transition to `potential`.
Alert whenever no such row exists in `pg_stat_replication`, including when
London has disappeared entirely.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 3a216e30-7097-465b-9e65-e81e76399747
📒 Files selected for processing (3)
docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-3.yamldocs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-sync-3.yamldocs/guides/postgres/remote-replica/synchronous.md
| spec: | ||
| authSecret: | ||
| name: pg-singapore-auth | ||
| clientAuthMode: md5 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== target files =="
for f in docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-3.yaml docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-sync-3.yaml docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore.yaml; do
if [ -f "$f" ]; then
echo "--- $f ---"
sed -n '1,80p' "$f" | cat -n
else
echo "MISSING $f"
fi
done
echo "== nearby docs references =="
rg -n "clientAuthMode|pg-singapore|PostgresVersion|17\.4|17\.10|CVE-2026-6478|pg_authid|password_encryption" docs/guides/postgres/remote-replica docs -g '*.md' -g '*.yaml' || trueRepository: kubedb/docs
Length of output: 48157
🌐 Web query:
PostgreSQL 17.10 CVE-2026-6478 MD5 authentication timing channel fix
💡 Result:
CVE-2026-6478 is a security vulnerability in PostgreSQL involving a covert timing channel that could allow an attacker to recover user credentials during the authentication process when using MD5-hashed passwords [1][2]. The vulnerability was addressed in PostgreSQL 17.10 (released on May 14, 2026) and other contemporary releases (18.4, 16.14, 15.18, and 14.23) [1][3][2]. The fix involves replacing non-constant-time string comparison functions, such as memcmp or strcmp, with the constant-time function timingsafe_bcmp when checking passwords and hashes [4][5][3]. Key details regarding this vulnerability include: - Vulnerability Mechanism: The issue stems from the use of data-dependent string comparison functions, which create a covert timing channel that an attacker can exploit to deduce password hashes [1][2]. - Impact: This vulnerability specifically affects MD5-hashed passwords, which may still exist in databases that were upgraded from PostgreSQL 13 or earlier [1][2]. It does not affect SCRAM-SHA-256 passwords, which have been the default authentication method in supported PostgreSQL releases [1][2]. - Remediation: Users are advised to upgrade to the patched versions (17.10 or later) [2]. Additionally, users can mitigate the risk by forcing the use of SCRAM-SHA-256 for all accounts, setting the password_encryption parameter to scram-sha-256, and removing MD5 authentication entries from pg_hba.conf [2].
Citations:
- 1: https://www.postgresql.org/support/security/CVE-2026-6478/
- 2: https://app.opencve.io/cve/CVE-2026-6478
- 3: https://github.com/postgres/postgres/blob/REL_17_STABLE/doc/src/sgml/release-17.sgml
- 4: https://www.postgresql.org/docs/17/release-17-10.html
- 5: https://www.postgresql.org/docs/release/17.10/
🌐 Web query:
KubeDB clientAuthMode scram postgres md5
💡 Result:
In KubeDB, the clientAuthMode field in the Postgres Custom Resource (CR) defines the authentication method used for PostgreSQL client connections [1][2][3]. The supported values for this field are md5 and scram [1][2]. Key details regarding these authentication modes in KubeDB include: 1. Supported Values: You can configure clientAuthMode as either md5 or scram [1][2]. This setting dictates how the PostgreSQL server authenticates incoming connections [1]. 2. Operational Changes: Updating the clientAuthMode in your KubeDB configuration will typically trigger an OpsRequest, such as a restart of the database, to apply the new authentication settings [4]. 3. PostgreSQL Compatibility: While KubeDB provides these options, it is important to note how PostgreSQL itself handles these modes [5]. PostgreSQL's md5 authentication method is a legacy, less secure challenge-response mechanism [5]. If md5 is specified as an authentication method in the server's pg_hba.conf but the user's password is encrypted using SCRAM (e.g., scram-sha-256), PostgreSQL will automatically switch to using SCRAM-based authentication [5]. 4. Best Practices: For enhanced security, using scram is recommended over md5, as the MD5 hash algorithm is no longer considered secure against modern attack vectors [5]. When configuring KubeDB, you can specify this field directly in the Postgres spec, for example: apiVersion: kubedb.com/v1 kind: Postgres metadata: name: my-postgres spec: clientAuthMode: scram... If you are removing TLS configurations, KubeDB allows you to define the clientAuthMode as part of the PostgresOpsRequest to specify which authentication method should be active once TLS is disabled [1][2].
Citations:
- 1: https://kubedb.com/docs/v2025.6.30/guides/postgres/reconfigure-tls/reconfigure-tls/
- 2: https://kubedb.com/docs/v2025.4.30/guides/postgres/reconfigure-tls/reconfigure-tls/
- 3: https://kubedb.com/docs/v2025.4.30/guides/postgres/quickstart/quickstart/
- 4: https://appscode.com/blog/post/kubedb-v2026.4.27/
- 5: https://www.postgresql.org/docs/current/auth-password.html
🌐 Web query:
KubeDB docs clientAuthMode scram Postgres spec
💡 Result:
In KubeDB, the PostgreSQL client authentication mode is configured via the clientAuthMode field within the Postgres custom resource (CR) or when performing TLS-related operations via a PostgresOpsRequest [1][2][3]. SCRAM (Salted Challenge Response Authentication Mechanism) is a supported authentication mode in KubeDB [1][3]. Configuration Details: 1. Postgres Resource Specification: You can define the client authentication mode directly in your Postgres CR under the spec section: spec: clientAuthMode: scram 2. PostgresOpsRequest: When reconfiguring TLS settings (specifically when removing TLS), you can define the client authentication mode that should be applied once TLS is removed. The field supports scram as a valid value [1][3]: spec: type: ReconfigureTLS tls: clientAuthMode: scram remove: true According to KubeDB documentation, supported values for clientAuthMode generally include md5 and scram [1][3]. When TLS is enabled with certificate-based authentication, the mode is typically set to cert [1][3].
Citations:
- 1: https://kubedb.com/docs/v2025.6.30/guides/postgres/reconfigure-tls/reconfigure-tls/
- 2: https://kubedb.com/docs/v2025.4.30/guides/postgres/remote-replica/remotereplica/
- 3: https://kubedb.com/docs/v2025.4.30/guides/postgres/reconfigure-tls/reconfigure-tls/
Weak Cryptography (CWE-327): Use of a Broken or Risky Cryptographic Algorithm
Reachability: External
Remove MD5 authentication and update the PostgreSQL version.
clientAuthMode: md5 enables MD5-based password authentication, and the examples use version: "17.4", which is affected by CVE-2026-6478 when MD5 password verifiers exist. Use clientAuthMode: scram, update spec.version to a patched PostgreSQL 17 release or later, and rotate passwords so no role stores an MD5 verifier.
📍 Affects 2 files
docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-3.yaml#L9-L9(this comment)docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-sync-3.yaml#L9-L9
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-3.yaml` at
line 9, Update clientAuthMode from md5 to scram in
docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-3.yaml:9-9
and
docs/guides/postgres/remote-replica/synchronous-yamls/pg-singapore-sync-3.yaml:9-9.
In both manifests, update spec.version from 17.4 to a patched PostgreSQL 17
release or later, and rotate the configured passwords so no role retains an MD5
verifier.
| ``` | ||
| application_name | state | sync_state | sync_priority | ||
| ------------------+-----------+------------+--------------- | ||
| pg-london-0 | streaming | sync | 1 | ||
| pg-singapore-0 | streaming | sync | 2 | ||
| pg-singapore-2 | streaming | potential | 4 | ||
| (3 rows) | ||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add language tags to both fenced output blocks.
markdownlint reports MD040 for these fences. Mark both blocks as text.
Proposed fix
-```
+```text
...
-```
+```textAlso applies to: 556-561
🧰 Tools
🪛 markdownlint-cli2 (0.23.1)
[warning] 544-544: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/synchronous.md` around lines 544 - 551,
Add the text language tag to both fenced output blocks in the synchronous
replication example, including the block around the replica status table and the
additional block referenced near lines 556–561, while leaving their contents
unchanged.
Source: Linters/SAST tools
| `pg-singapore-2` is `potential` — connected and caught up, but not currently counted. Lose the | ||
| active local standby and PostgreSQL promotes the spare with no operator action: |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the documented section and nearby context.
if [ -f docs/guides/postgres/remote-replica/synchronous.md ]; then
echo "== file lines 520-570 =="
sed -n '520,570p' docs/guides/postgres/remote-replica/synchronous.md | nl -ba -v520
else
echo "target file not found"
fd -a 'synchronous\.md' .
fi
echo
echo "== search for state/sync_state context in repo =="
rg -n "sync_state|state = streaming|potential|caught up|replicat|lag" docs/guides/postgres/remote-replica/synchronous.md docs -S 2>/dev/null | head -200Repository: kubedb/docs
Length of output: 214
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Fetch authoritative PostgreSQL documentation for sync_standbys / sync_state semantics.
python3 - <<'PY'
from urllib.request import urlopen
url = "https://www.postgresql.org/docs/15/runtime-config-replication.html"
try:
text = urlopen(url, timeout=20).read().decode("utf-8", "ignore")
except Exception as exc:
print(f"FETCH_FAILED: {exc}")
raise SystemExit(0)
terms = ["sync_state", "potential", "standby", "synchronous", "caught up"]
lines = text.splitlines()
for i,line in enumerate(lines, 1):
if any(t in line.lower() for t in terms):
start=max(1,i-3); end=min(len(lines),i+3)
print(f"--- around line {i} ---")
for j in range(start,end+1):
print(f"{j}: {lines[j-1]}")
print()
PYRepository: kubedb/docs
Length of output: 298
🌐 Web query:
PostgreSQL synchronous_commit sync_state potential state streaming synchronous_standby_names documentation
💡 Result:
In PostgreSQL, the interaction between synchronous_commit, synchronous_standby_names, and the state of replication is managed to balance data durability with system availability [1][2]. 1. synchronous_standby_names This parameter defines which standby servers are eligible to be synchronous standbys and how many must acknowledge data before a transaction commit is allowed to proceed [1][3][4]. It supports two primary modes: FIRST num_sync (standby_name [,...]): Priority-based. Transactions wait for acknowledgement from the first num_sync standbys listed that are currently active [3][4]. ANY num_sync (standby_name [,...]): Quorum-based. Transactions wait for acknowledgement from any num_sync standbys in the list [3][4]. 2. Synchronous States and pg_stat_replication The primary server monitors standby status via the pg_stat_replication view [1][2]. For a standby to be considered an active synchronous standby, it must be connected and actively streaming data (indicated by a state of streaming in pg_stat_replication) [1][3][4]. Standbys that are not currently chosen as synchronous (e.g., lower priority in a FIRST configuration or not part of the quorum) are considered potential synchronous standbys [4]. 3. synchronous_commit This parameter determines the level of synchronization enforced on the primary during a transaction commit [5][6]: on: Commits wait for synchronous standbys to write the WAL record to durable storage [5][2]. remote_write: Commits wait for synchronous standbys to receive and write the WAL record to their file systems, but not necessarily to durable storage [5][6]. remote_apply: Commits wait for synchronous standbys to replay the WAL record so it is visible to queries [1][5][2]. local: Commits wait for the local flush to disk, effectively ignoring synchronous standby requirements even if configured [1][5][3]. off: Commits do not wait for replication, providing no synchronous durability guarantees [1][5][3]. If synchronous_standby_names is empty, synchronous replication is effectively disabled regardless of the synchronous_commit setting (except for local/off modes) [1][3][4]. Conversely, if synchronous_standby_names is non-empty, transaction commits will wait for the configured number of standbys to reply, provided synchronous_commit is set to a mode other than local or off [1][5][3].
Citations:
- 1: https://www.postgresql.org/docs/current/runtime-config-replication.html
- 2: https://www.postgresql.org/docs/19/warm-standby.html
- 3: https://postgresqlco.nf/doc/en/param/synchronous_standby_names/
- 4: https://www.postgresql.org/docs/19/runtime-config-replication.html
- 5: https://www.postgresql.org/docs/18/runtime-config-wal.html
- 6: https://www.postgresql.org/docs/16/warm-standby.html
🌐 Web query:
PostgreSQL 15 runtime-config-replication sync_state potential streaming lag synchronous standby standby_state streaming replication_status
💡 Result:
In PostgreSQL 15, the sync_state and associated streaming states in the pg_stat_replication view are critical for monitoring replication health and synchronous replication topology [1][2]. Streaming States (state column): The state column reflects the current operation mode of the WAL sender process [1]: - startup: The WAL sender is initializing. - catchup: The standby is currently streaming WAL records to catch up with the primary's current position [3][1]. A standby cannot become synchronous until it transitions out of this state [3]. - streaming: The standby has caught up and is streaming changes in real-time [3][1]. - backup: The WAL sender is currently sending a base backup to the standby [1]. - stopping: The WAL sender is in the process of shutting down [1]. Synchronous Replication States (sync_state column): The sync_state column indicates how a standby participates in synchronous replication [1]: - async: The standby is configured for asynchronous replication; it does not block primary commits [4][1]. - potential: The standby is currently asynchronous but is eligible to become a synchronous standby if a current synchronous standby fails [4][1]. - sync: The standby is active as a synchronous standby; primary transactions wait for this standby to confirm the flush of WAL records [4][1]. - quorum: The standby is part of a quorum-based synchronous replication group, acting as a candidate for confirmation [4][1]. Replication Lag and Synchronous Gating: Replication lag is measured via LSNs (Log Sequence Numbers) such as sent_lsn, write_lsn, flush_lsn, and replay_lsn [4][2]. For synchronous replication, the primary commit blocks based on the synchronous_commit setting [5]: - remote_write: Primary waits until the standby has written the WAL to its file system [5]. - on (default): Primary waits until the standby has flushed the WAL to durable storage [5]. - remote_apply: Primary waits until the standby has replayed the WAL, making changes visible to queries [5][6]. Monitoring lag for a sync standby should prioritize flush_lag (the time to reach disk) or replay_lag (the time to apply), depending on the synchronous_commit setting [4]. If a standby is in a catchup state, it is not yet synchronized, and its lag may be high as it processes backlogged WAL records [3].
Citations:
- 1: https://www.postgresql.org/docs/15/monitoring-stats.html
- 2: https://www.cybertec-postgresql.com/en/monitoring-replication-pg_stat_replication/
- 3: https://www.postgresql.org/docs/15/warm-standby.html
- 4: https://www.postgresscripts.com/post/monitor-postgresql-replication-lag-with-pg-stat-replication/
- 5: https://www.postgresql.org/docs/15/runtime-config-wal.html
- 6: https://www.postgresql.org/docs/15/runtime-config-replication.html
🌐 Web query:
site:postgresql.org/docs/15/monitoring-stats.html pg_stat_replication sync_state potential state catchup streaming
💡 Result:
In PostgreSQL, the sync_state column in the pg_stat_replication view indicates the synchronous replication status of a connected standby server [1]. The possible values for sync_state are: async: The standby server is operating asynchronously and is not considered for synchronous replication [1]. potential: The standby server is currently operating as an asynchronous standby, but it is configured in a way that allows it to potentially become a synchronous standby if one of the currently active synchronous standbys fails [1]. sync: The standby server is currently acting as a synchronous standby, meaning transactions on the primary will wait for confirmation from this server [1]. quorum: The standby server is considered a candidate for quorum-based synchronous replication [1]. Additionally, the state column in pg_stat_replication provides the status of the WAL sender process itself, which includes values such as catchup (the standby is currently catching up with the primary's WAL) and streaming (the standby has caught up and is actively streaming changes) [1].
Citations:
🌐 Web query:
site:postgresql.org/docs/15/runtime-config-replication.html synchronous_standby_names sync_state "potential" "catchup"
💡 Result:
In PostgreSQL, the synchronous_standby_names parameter defines a list of standby servers that can act as synchronous standbys [1]. When using priority-based synchronous replication (configured with the FIRST keyword), standbys listed in this parameter are evaluated based on their order [1]. Standbys that appear earlier in the list are granted higher priority and are considered active synchronous standbys if they are connected and streaming data in real-time [1]. Standbys that appear later in the list serve as potential synchronous standbys [1]. If a higher-priority standby disconnects, the system automatically promotes a potential synchronous standby from the list to replace it [1]. Regarding the terms catchup or sync_state, while they may appear in user-defined monitoring or specific extension views, the core PostgreSQL documentation for runtime configuration does not define these as standard values for synchronous replication state management; the system relies on the list order and the streaming connection status to determine which standbys are currently active [1].
Citations:
🏁 Script executed:
#!/bin/bash
set -euo pipefail
if [ -f docs/guides/postgres/remote-replica/synchronous.md ]; then
echo "== target lines 520-570 =="
awk '{printf "%6d\t%s\n", NR, $0}' docs/guides/postgres/remote-replica/synchronous.md | sed -n '520,570p'
else
echo "target file not found"
fi
echo
echo "== relevant local search =="
rg -n "sync_state|state = streaming|potential|caught up|caught up|replicat|lag|Lag|LSN|sent_lsn|write_lsn|flush_lsn|replay_lsn|catchup" docs/guides/postgres/remote-replica/synchronous.md docs -S || trueRepository: kubedb/docs
Length of output: 50368
🌐 Web query:
PostgreSQL pg_stat_replication sync_state potential synchronous_standby streaming "caught up" lag
💡 Result:
In PostgreSQL, the pg_stat_replication view provides real-time information about the status of standby servers [1]. The sync_state column indicates the current synchronous replication role of a standby, while various "lag" and "location" columns help determine if a standby has "caught up" [1][2][3]. Sync State Definitions The sync_state column describes whether a standby is currently acting as a synchronous standby for the primary [1][4]. Possible values include: - async: The standby is configured for asynchronous replication [1][2]. - sync: The standby is currently acting as a synchronous standby [1][2]. - potential: The standby is currently asynchronous, but it is a candidate to become synchronous if a currently active synchronous standby fails (relevant in priority-based replication) [1][2][5]. - quorum: The standby is a candidate to satisfy a quorum-based synchronous replication requirement [1][2]. Determining if a Standby has "Caught Up" A standby is considered "caught up" when its reported Write-Ahead Log (WAL) positions match the primary's current position [2][3]. You can monitor this by comparing the following columns in pg_stat_replication [2][3]: - sent_location: The last WAL location sent by the primary to this standby [2]. - write_location (or write_lsn): The last WAL location written to disk by the standby [2][3]. - flush_location (or flush_lsn): The last WAL location flushed to disk by the standby [2][3]. - replay_location (or replay_lsn): The last WAL location replayed into the database on the standby [2][3]. If all these locations are equal to the primary's current WAL position, the standby is fully caught up [2][3]. Understanding Lag Lag columns (write_lag, flush_lag, replay_lag) in pg_stat_replication represent the time elapsed since the WAL was generated on the primary until it reached a specific state (write, flush, or replay) on the standby [1][3]. - These times measure the potential commit delay that would be introduced if the standby were configured as a synchronous standby [1]. - If a standby is entirely caught up and there is no new WAL activity, these lag values will eventually show NULL [1]. - Note that lag times are not predictions of how long a currently lagging standby will take to catch up; they reflect the delay of transactions already processed [1]. In the context of synchronous replication, a standby must have caught up with the primary to move from a "potential" state to an active "sync" state [6]. If a standby is lagging behind, it does not contribute to quorum-based replication requirements until it successfully catches up [7].
Citations:
- 1: https://www.postgresql.org/docs/current/monitoring-stats.html
- 2: https://www.claudiokuenzler.com/blog/723/how-to-monitor-postgresql-replication
- 3: https://sqlconjuror.com/postgresql-monitoring-replication/
- 4: https://api.pgxn.org/src/pgsql_tweaks/pgsql_tweaks-1.0.5/sql/view_monitoring_replication.sql
- 5: https://dba.stackexchange.com/questions/316794/sync-state-pg-stat-replication
- 6: https://github.com/postgres/postgres/blob/207cb2ab/src/backend/replication/syncrep.c
- 7: https://www.postgresql.org/message-id/CAE9k0PkhdZCXrpCPRdezb0-Y_iUMgHWfUCBzNJR5reV8XaoRDg%40mail.gmail.com
Do not equate potential with caught up.
sync_state = potential only reports that the standby is a candidate for future synchronous use. state = streaming only reports the WAL sender mode. If this text describes a standby actually ready to be promoted or used as synchronous, add the LSN/lag comparison that supports that.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/synchronous.md` around lines 553 - 554,
Update the standby description around `pg-singapore-2` to avoid equating
`sync_state = potential` or `state = streaming` with readiness for promotion or
synchronous use. Add the relevant LSN/lag comparison demonstrating that the
standby is caught up before claiming it can be promoted without operator action;
otherwise, revise the wording to describe only its candidate and WAL sender
states.
| - **Alert on the sync set, not just on replication.** `sync_state` dropping from `sync` to | ||
| `potential` for `pg-london-0` is the signal that cross-cluster durability has been lost, and | ||
| nothing else in the system will complain about it. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
git ls-files | grep -F 'docs/guides/postgres/remote-replica/synchronous.md' || true
echo "== adjacent lines =="
if [ -f docs/guides/postgres/remote-replica/synchronous.md ]; then
sed -n '560,605p' docs/guides/postgres/remote-replica/synchronous.md | nl -ba -v560
fi
echo "== search sync_state/replica alert examples =="
rg -n "sync_state|pg_stat_replication|Alert on|pg-london-0|potential|streaming" docs/guides/postgres/remote-replica/synchronous.md docs || true
echo "== PostgreSQL docs references in repo =="
rg -n "postgresql.org/docs/current/monitoring-stats" . || trueRepository: kubedb/docs
Length of output: 279
🌐 Web query:
PostgreSQL current monitoring-stats pg_stat_replication sync_state synchronous_commit synchronous_standby_names documentation
💡 Result:
In PostgreSQL, the pg_stat_replication view is the primary tool for monitoring streaming replication, providing visibility into the state of connections between the primary and its standby servers [1][2]. Key components related to synchronous replication monitoring include: sync_state in pg_stat_replication The sync_state column indicates the current role of a standby server within the synchronous replication configuration [3]. Possible values include: - sync: The standby is currently operating as a synchronous standby, meaning transaction commits on the primary wait for confirmation from this server [3][4]. - potential: The standby is configured for synchronous replication but is not currently acting as a primary synchronous standby. It serves as a backup and would be promoted to sync if a current synchronous standby fails [3][4]. - async: The standby is operating in asynchronous mode [3][4]. - quorum: The standby is participating in quorum-based synchronous replication [3][4]. synchronous_standby_names This configuration parameter specifies the list of standby servers that can serve as synchronous standbys [5][6]. It dictates which standbys are eligible for synchronous replication based on the defined strategy (e.g., FIRST or ANY) [5][6]. When synchronous replication is enabled, the primary server waits for confirmation from these standbys based on the configured commit level [5]. synchronous_commit This parameter determines the level of synchronization required for a transaction to be considered successfully committed [7][5]. It interacts directly with synchronous_standby_names: - If synchronous_standby_names is empty, synchronous replication is disabled, and this parameter primarily controls local flushing [7][5]. - When synchronous_standby_names is populated, synchronous_commit controls how the primary waits for standby confirmation [7][5]. Modes include: - remote_apply: Waits until the standby has received, written, and applied the transaction (making it visible to queries on the standby) [7]. - on: Waits until the standby has received and flushed the transaction to durable storage [7]. - remote_write: Waits until the standby has received and written the transaction to its file system (not necessarily to durable storage) [7]. Monitoring Strategy To monitor effectively, ensure that standbys listed in synchronous_standby_names show a state of streaming in pg_stat_replication [5][6]. Discrepancies between flush_lsn and replay_lsn in this view can help identify if a standby is receiving data but lagging in applying it, which is particularly relevant when using the remote_apply commit mode [1].
Citations:
- 1: https://www.cybertec-postgresql.com/en/monitoring-replication-pg_stat_replication/
- 2: https://www.postgresql.org/docs/current/monitoring-stats.html
- 3: https://www.interdb.jp/pg/pgsql11/03.html
- 4: https://www.claudiokuenzler.com/blog/723/how-to-monitor-postgresql-replication
- 5: https://www.postgresql.org/docs/current/runtime-config-replication.html
- 6: https://postgresqlco.nf/doc/en/param/synchronous_standby_names/
- 7: https://www.postgresql.org/docs/18/runtime-config-wal.html
🌐 Web query:
PostgreSQL synchronous_commit pg_stat_replication sync_state when standby disconnected potential sync
💡 Result:
In PostgreSQL, the behavior of synchronous replication when a standby disconnects depends on the primary's configuration and whether other standbys are available to take over [1]. When a standby configured as a synchronous standby disconnects, the primary server's WAL sender for that standby will terminate [1]. If synchronous_standby_names is configured to allow for potential synchronous standbys, the primary will immediately promote a standby with a sync_state of 'potential' to 'sync' to maintain the required number of synchronous replicas [1][2]. If no other standby can satisfy the synchronous_standby_names configuration, transactions on the primary that require synchronous commit (based on synchronous_commit being set to on, remote_apply, or remote_write) will block indefinitely until a standby becomes available or the synchronous_standby_names configuration is updated [3][4][5]. Key details regarding pg_stat_replication and sync_state: 1. Sync State Definitions: - sync: The standby is currently acting as a synchronous replica; the primary waits for acknowledgments from it [6][7]. - potential: The standby is not currently acting as a synchronous replica but is eligible to become one if a current 'sync' standby disconnects [6][7]. - async: The standby is operating asynchronously; the primary does not wait for acknowledgments from it [6][7]. 2. Important Behaviors: - Blocking: If synchronous_commit is set to any value other than off or local (e.g., on, remote_apply, remote_write), the primary will stall if it cannot receive the required acknowledgments from the configured synchronous standby(s) [5][8]. This is intended behavior to guarantee durability, but it can lead to application outages if not managed correctly [3][9][10]. - Disconnection: When a standby is disconnected, its row in pg_stat_replication will eventually disappear once the connection is cleaned up [1]. - Local Commit: Even when using synchronous replication, a transaction may commit locally on the primary while waiting for the standby acknowledgment. If a user cancels a waiting transaction (e.g., via CTRL+C), the primary may log a warning that the transaction was committed locally but not yet replicated to the standby [9][10]. To avoid blocking the primary during standby maintenance or failure, administrators often monitor pg_stat_replication to identify the sync_state and can temporarily set synchronous_commit to local or off for specific sessions if they need to perform maintenance while a standby is down [3][5][11].
Citations:
- 1: https://github.com/postgres/postgres/blob/92268b35d04c2de416279f187d12f264afa22614/src/backend/replication/syncrep.c
- 2: https://www.postgresql.org/message-id/1299450975.1696.5974.camel%40ebony
- 3: https://runebook.dev/en/docs/postgresql/runtime-config-replication/GUC-SYNCHRONOUS-STANDBY-NAMES
- 4: https://www.postgresql.org/docs/current/runtime-config-replication.html
- 5: https://www.postgresql.org/docs/18/runtime-config-wal.html
- 6: https://www.postgresscripts.com/post/monitor-postgresql-replication-lag-with-pg-stat-replication/
- 7: https://www.claudiokuenzler.com/blog/723/how-to-monitor-postgresql-replication
- 8: https://www.postgresql.org/docs/17/runtime-config-wal.html
- 9: https://www.postgresql.org/message-id/cac4b9df-92c6-77aa-687b-18b86cb13728@stratox.cz
- 10: https://www.postgresql.org/message-id/84cc0332fadf4f4bb1fff7a169db8ecd58f7911c.camel%40cybertec.at
- 11: https://www.postgresql.org/message-id/42B0E1C07CB24F4889ED8A345AF8EFC2%40maumau
🌐 Web query:
site:postgresql.org/docs/current/monitoring-stats.html pg_stat_replication sync_state standby disconnected
💡 Result:
In the PostgreSQL pg_stat_replication view, there is no sync_state value of disconnected [1]. The sync_state column indicates the synchronous replication status of a standby server and is restricted to the following valid values [1]: async: The standby server is operating asynchronously [1]. potential: The standby server is currently asynchronous but is eligible to become a synchronous standby if a current synchronous standby fails [1]. sync: The standby server is currently acting as a synchronous standby [1]. quorum: The standby server is considered a candidate for quorum-based synchronous replication [1]. If a standby server disconnects from the primary, the row corresponding to that WAL sender process in pg_stat_replication will typically disappear from the view, as the view only displays statistics for currently connected standby servers [1].
Citations:
🌐 Web query:
site:postgresql.org/docs/current/monitoring-stats.html pg_stat_replication sync_state standby disconnected
💡 Result:
In PostgreSQL, the sync_state column in the pg_stat_replication view does not include a disconnected value [1]. The official documentation defines the possible values for sync_state as async, potential, sync, and quorum [1]. The pg_stat_replication view only contains rows for currently connected standby servers; if a standby server disconnects, its corresponding row is removed from the view [1]. Therefore, you will not see a status of disconnected for a standby within this view, as the view only provides information about active WAL sender processes [1].
Citations:
Alert when London is absent, not only when its state changes.
A disconnected pg-london-0 can disappear from pg_stat_replication, so sync_state may not drop from sync to potential. Alert when no connected row exists with application_name = 'pg-london-0', state = 'streaming', and sync_state = 'sync'.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/synchronous.md` around lines 586 - 588,
Update the synchronous replication alert guidance to check for the presence of a
connected `pg-london-0` row, requiring `application_name = 'pg-london-0'`,
`state = 'streaming'`, and `sync_state = 'sync'`, rather than relying only on a
`sync_state` transition to `potential`. Alert whenever no such row exists in
`pg_stat_replication`, including when London has disappeared entirely.
|
Visit the preview URL for this PR (updated for commit e46d9df): https://kubedb-v2-hugo--pr1038-pg-sync-cross-cluste-01n3r91v.web.app (expires Tue, 11 Aug 2026 12:46:45 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: 0f29ae8ae0bd54a99bf2b223b6833be47acd5943 |
1 participant
Adds
docs/guides/postgres/remote-replica/synchronous.mdplus manifests — a synchronous remote-replica setup where every acknowledged commit is already flushed in the DR cluster.The quorum
With a 2-replica primary, exactly one Singapore pod is primary, so the two selected standbys are always one local + one cross-cluster regardless of which pod holds the role:
pg-singapore-0pg-london-0+pg-singapore-1pg-singapore-1pg-london-0+pg-singapore-0Verified end to end
Run on two clusters against the guide's own manifests, from a clean namespace:
Then the primary was destroyed with
deletionPolicy: Delete(PVCs wiped) and the DR site compared by content fingerprint:Identical, with no promotion — the data was durable in the DR cluster at commit time, not recovered afterwards.
Three traps the guide documents
numSyncReplicas: 2and one standby, every commit blocks — including the writes KubeDB performs during bootstrap, so the database never becomesReady. The guide deploys asynchronous first and switches in a later step.commitLevelmust be quoted. BareOnis a YAML 1.1 boolean and the webhook rejects it withjson: cannot unmarshal bool into Go struct field ...commitLevel.could not accept SSL connection: tlsv1 alert unknown ca. cert-manager reuses a still-valid leaf Secret rather than reissuing when the CA Issuer's backing secret is replaced, so deleting theCertificateobjects is not enough — theSecrets must go too. Hit while testing; includes the symptom, the fix, and anopensslone-liner to compare CA serials.Auth secrets are created from a shell variable via
kubectl create secret, so no password is committed to the repo.Summary by CodeRabbit