postgres/remote-replica: TLS SAN guidance, custom port, new CLI flags - #1040
postgres/remote-replica: TLS SAN guidance, custom port, new CLI flags#1040tamalsaha wants to merge 4 commits into
Conversation
Driven by a customer report: their replica looped on "Attempting pg_isready on primary" with SOURCE_SSL_MODE=verify-full while a non-TLS connection worked. Root cause is not a network or code problem: verify-full checks the dialed hostname against the server certificate's SANs, and the certificate is issued for in-cluster names only, so any external/load-balancer address fails the handshake. Adds a "TLS across clusters" section: verify-ca is the recommended mode between clusters; verify-full is supported guidance with the exact ReconfigureTLS ops request that adds the external hostname SAN to the server certificate (reissue and rotation handled by the ops manager, no manual restarts); and the pg_isready-loop symptom is named so the next person can recognize it. Updates the remote-config example for the new flags: -d host:port / --port (written into the AppBinding and honored end to end by the replica), --replica-name (emits a ready-to-apply replica manifest sized from the source spec), --auth-secret, plus notes on the fixed output path and the -y flag being a plain confirmation skip. Signed-off-by: Tamal Saha <tamal@appscode.com>
![kodiak-appscode[bot]](https://avatars.githubusercontent.com/in/92481?s=60&v=4){kind=small}
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe remote replica documentation now covers configuration generation, TLS behavior and reconfiguration, coordinator operations, lag monitoring, timeline recovery, standby services, Prometheus scraping, and Grafana dashboards. ChangesRemote Replica Documentation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/postgres/remote-replica/remotereplica.md`:
- Around line 261-264: Update the remote replica documentation so the endpoint
used by the connection command matches the certificate SAN: change the `-d`
example to dial `pg-singapore.example.com:5432`, or add the dialed IP as an IP
SAN in the certificate request. Preserve the existing `server` alias and
`dnsNames` configuration.
- Around line 218-224: Remove the later manual creation steps for pg-london-auth
and pg-london from the documented sequence, including the hand-written manifest
and apply commands around the resource-creation sections. Keep the
--replica-name generated manifest flow as the sole resource-creation path, so
applying the generated file does not duplicate resources or override its
authentication configuration.
- Around line 241-243: Update the TLS troubleshooting guidance around
“Attempting pg_isready on primary” so it does not imply SAN mismatch is the
primary explanation. State that successful non-TLS connectivity only confirms
network reachability, and direct readers to verify CA data, client
certificate/key, TLS policy, and KubeDB/PostgreSQL logs for other TLS failures.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b382e8b0-0331-44d2-8f79-253abf534139
📒 Files selected for processing (1)
docs/guides/postgres/remote-replica/remotereplica.md
| - `--replica-name` additionally emits a ready-to-apply remote replica `Postgres` manifest, | ||
| sized from the source's spec (version, replicas, storage, resources) with the | ||
| `remoteReplica` stanza and auth secret filled in. Treat it as a starting point — a | ||
| secondary site is often sized differently on purpose. | ||
| - `--auth-secret <name>` overrides the generated auth secret's name. | ||
| - The output file is always written to the **current directory** as | ||
| `<dbname>-remote-config.yaml`; `-y` only skips the confirmation prompt. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Remove the duplicate pg-london creation path.
--replica-name is documented to emit a ready-to-apply Postgres manifest, and Line 210 applies the generated file. The later steps still create pg-london from the hand-written manifest at Lines 302-337 and create pg-london-auth at Lines 283-300. Following the documented sequence can hit AlreadyExists or apply a different auth configuration. Remove the later manual resources, or state that users must not apply the generated Postgres manifest.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/remotereplica.md` around lines 218 - 224,
Remove the later manual creation steps for pg-london-auth and pg-london from the
documented sequence, including the hand-written manifest and apply commands
around the resource-creation sections. Keep the --replica-name generated
manifest flow as the sole resource-creation path, so applying the generated file
does not duplicate resources or override its authentication configuration.
| handshake unless that address was added to the certificate. The replica then loops | ||
| forever with `Attempting pg_isready on primary` while a plain (non-TLS) connection works | ||
| — that symptom almost always means a SAN mismatch, not a network problem. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the reviewed guide around the quoted lines.
if [ -f docs/guides/postgres/remote-replica/remotereplica.md ]; then
echo "== file exists =="
wc -l docs/guides/postgres/remote-replica/remotereplica.md
sed -n '210,260p' docs/guides/postgres/remote-replica/remotereplica.md | cat -n
else
echo "file not found"
fi
echo
echo "== nearby pg_isready / TLS / cert mentions =="
rg -n "pg_isready|SAN|certificate|TLS|certificate-authority-data|client certificate|operator logs|connection" docs/guides/postgres/remote-replica/remotereplica.md docs -g '*.md' || trueRepository: kubedb/docs
Length of output: 50368
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target file summary =="
wc -l docs/guides/postgres/remote-replica/remotereplica.md || true
echo
echo "== target file around quoted text =="
sed -n '220,250p' docs/guides/postgres/remote-replica/remotereplica.md | cat -n
echo
echo "== postgres remote-replica references to TLS/cert/pg_isready in target doc =="
rg -n "TLS|certificate|pg_isready|certificate-authority-data|ca.crt|replica|operator logs|logs" docs/guides/postgres/remote-replica/remotereplica.md || true
echo
echo "== focused source docs around postgres pg_isready troubleshooting =="
rg -n -C 3 "Attempting pg_isready on primary|pg_isready on primary|SAN|certificate-authority-data|TLS" docs/guides/postgres -g '*.md' || trueRepository: kubedb/docs
Length of output: 50369
Don’t rule out other TLS failures around this symptom.
A successful non-TLS pg_isready only shows network reachability. Wrong CA data, missing client certificate/key, or TLS policy errors can also leave the replica stuck retrying Attempting pg_isready on primary; add a note to check TLS configuration and KubeDB/PostgreSQL logs instead of saying it “almost always means a SAN mismatch”.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/remotereplica.md` around lines 241 - 243,
Update the TLS troubleshooting guidance around “Attempting pg_isready on
primary” so it does not imply SAN mismatch is the primary explanation. State
that successful non-TLS connectivity only confirms network reachability, and
direct readers to verify CA data, client certificate/key, TLS policy, and
KubeDB/PostgreSQL logs for other TLS failures.
| - alias: server | ||
| dnsNames: | ||
| - pg-singapore.example.com # the address the replica dials | ||
| apply: Always |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Use the same endpoint in -d and the certificate SAN.
Line 207 dials 172.104.37.147, but this request adds only the DNS SAN pg-singapore.example.com. verify-full checks the endpoint used by the connection, so the documented IP command still fails TLS verification. Use pg-singapore.example.com:5432 in -d, or document a certificate request that adds the dialed IP as an IP SAN.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/remotereplica.md` around lines 261 - 264,
Update the remote replica documentation so the endpoint used by the connection
command matches the certificate SAN: change the `-d` example to dial
`pg-singapore.example.com:5432`, or add the dialed IP as an IP SAN in the
certificate request. Preserve the existing `server` alias and `dnsNames`
configuration.
|
Visit the preview URL for this PR (updated for commit 9a4c31d): https://kubedb-v2-hugo--pr1040-pg-remote-replica-do-n07bp6mh.web.app (expires Tue, 18 Aug 2026 09:45:11 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: 0f29ae8ae0bd54a99bf2b223b6833be47acd5943 |
…ndby Service Remote replica pods now run 2/2 with a pg-coordinator sidecar (no Raft) that self-heals on source timeline changes, keeps the standby role label truthful, and logs replication lag. A <name>-standby Service exists at any replica count and is the correct entry point for read traffic; the primary-selecting <name> Service has no endpoints while the database is a replica. All statements re-verified against the release stack in a full setup/failover/failback pass. Signed-off-by: Tamal Saha <tamal@appscode.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/guides/postgres/remote-replica/remotereplica.md (1)
202-225: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winUse a portable path for the generated file.
The workflow documents output in the current directory, but Line 277 uses an author-specific absolute path. Use
./pg-singapore-remote-config.yamlor the exact path emitted byremote-config.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/guides/postgres/remote-replica/remotereplica.md` around lines 202 - 225, Update the remote replica example around the kubectl apply command to use a portable generated-file path, such as ./pg-singapore-remote-config.yaml, matching the current-directory output described by remote-config; remove the author-specific absolute path while preserving the documented filename.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs/guides/postgres/remote-replica/remotereplica.md`:
- Around line 202-225: Update the remote replica example around the kubectl
apply command to use a portable generated-file path, such as
./pg-singapore-remote-config.yaml, matching the current-directory output
described by remote-config; remove the author-specific absolute path while
preserving the documented filename.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 9210be1e-5d28-4d0d-b52f-9f64423e2ac5
📒 Files selected for processing (1)
docs/guides/postgres/remote-replica/remotereplica.md
Documents the coordinator's DR metrics on the raft-metrics port, the spec.monitor + OnDelete pod-roll step, the NetworkPolicy scrape-allow needed on netpol-enabled clusters, ConfigMap-based dashboard provisioning (API imports do not survive a persistence-less Grafana restart), and how to read the dashboard. Written from a live setup on a public-IP two-cluster pair. Signed-off-by: Tamal Saha <tamal@appscode.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/guides/postgres/remote-replica/monitoring.md`:
- Line 13: Update the introductory link in the monitoring guide to replace the
generic “here” text with descriptive text identifying the KubeDB documentation
overview, while preserving the existing destination URL.
- Around line 138-152: The dashboard setup instructions at
docs/guides/postgres/remote-replica/monitoring.md lines 138-152 must use a valid
dashboard source containing postgres_remote_replica_dashboard.json; update the
referenced opnpulse/dashboards revision or link/provide checkout steps for a
source where the artifact exists, and ensure the kubectl command is runnable.
The related reference at docs/guides/postgres/remote-replica/monitoring.md lines
45-49 requires the same source correction; no separate behavior change is
needed.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 76702a11-f34c-4a8f-a132-5745e5b45117
📒 Files selected for processing (1)
docs/guides/postgres/remote-replica/monitoring.md
| section_menu_id: guides | ||
| --- | ||
|
|
||
| > New to KubeDB? Please start [here](/docs/README.md). |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use descriptive link text.
Replace [here] with text that identifies the destination, such as [the KubeDB documentation overview]. This improves link context and resolves MD059.
🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 13-13: Link text should be descriptive
(MD059, descriptive-link-text)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/monitoring.md` at line 13, Update the
introductory link in the monitoring guide to replace the generic “here” text
with descriptive text identifying the KubeDB documentation overview, while
preserving the existing destination URL.
Source: Linters/SAST tools
| The **KubeDB / Postgres / Remote Replica** dashboard | ||
| ([opnpulse/dashboards, postgres folder](https://github.com/opnpulse/dashboards/tree/master/postgres)) | ||
| has three rows: *DR Protection Status* (streaming, source reachable, RPO in bytes, lag | ||
| data age, recoveries in 24h), *Replication Lag* (byte lag from the coordinator; apply | ||
| lag in seconds from the exporter — the latter also grows while the source is idle, read | ||
| them together), and *Self-Healing & Replica Health*. | ||
|
|
||
| Provision it as a ConfigMap so it survives Grafana restarts (kube-prometheus-stack's | ||
| Grafana has no persistence — dashboards imported through the UI or API are lost on pod | ||
| restart; the sidecar re-provisions labeled ConfigMaps): | ||
|
|
||
| ```bash | ||
| kubectl create configmap pg-remote-replica-dashboard -n monitoring \ | ||
| --from-file=postgres_remote_replica_dashboard.json | ||
| kubectl label configmap pg-remote-replica-dashboard -n monitoring grafana_dashboard=1 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target file around affected lines =="
if [ -f docs/guides/postgres/remote-replica/monitoring.md ]; then
sed -n '35,60p;130,160p' docs/guides/postgres/remote-replica/monitoring.md | cat -n
else
echo "target file not found"
fi
echo
echo "== dashboard json references in target file =="
rg -n "postgres_remote_replica_dashboard|external|dashboard|Grafana|7\.5|7\.4|github.com/opnpulse/dashboards" docs/guides/postgres/remote-replica/monitoring.md || true
echo
echo "== check public file listing and README version for referenced path =="
url="https://api.github.com/repos/opnpulse/dashboards/git/trees/master?recursive=1"
python3 - <<'PY'
import json, urllib.request
from urllib.request import Request, HTTPError
url="https://api.github.com/repos/opnpulse/dashboards/git/trees/master?recursive=1"
req=Request(url, headers={'User-Agent':'CodeRabbit'})
try:
with urllib.request.urlopen(req, timeout=20) as r:
data=json.load(r)
except Exception as e:
print("GITHUB_TREE_ERROR:", type(e).__name__, e)
raise SystemExit(0)
target="postgres_remote_replica_dashboard.json"
matches=[]
paths=[]
for item in data.get("tree", []):
paths.append(item["path"])
if item["path"] == target:
matches.append(item)
print("master_contains_target_file:", len(matches)>0, matches[:5])
postgres_files=sorted(p for p in paths if p.startswith("postgres/") and p.endswith(".json"))
print("postgres_json_files=", postgres_files)
# find README line possibly mentioning versions
readme="".join(data.get("tree", [])[0]) if any(p=="README.md" for p in paths) else None
</PY
python3 - <<'PY'
import urllib.request
url="https://raw.githubusercontent.com/opnpulse/dashboards/master/README.md"
print("README fetch:", end=" ")
try:
text=urllib.request.urlopen(url, timeout=20).read().decode()
print("ok")
for i,line in enumerate(text.splitlines(),1):
if "Grafana" in line or "7.4" in line or "7.5" in line:
print(f"{i}: {line[:200]}")
except Exception as e:
print("error:", e)
PYRepository: kubedb/docs
Length of output: 4426
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import urllib.request
import json
url = "https://api.github.com/repos/opnpulse/dashboards/git/trees/master?recursive=1"
req = urllib.request.Request(url, headers={"User-Agent": "CodeRabbit"})
try:
with urllib.request.urlopen(req, timeout=20) as r:
data = json.load(r)
except Exception as e:
print("GITHUB_TREE_ERROR:", type(e).__name__, e)
raise SystemExit(0)
target = "postgres_remote_replica_dashboard.json"
paths = [item["path"] for item in data.get("tree", [])]
print("master_contains_target_file:", target in paths)
print("matched_files:")
for item in data.get("tree", []):
if item["path"] == target:
print(item)
print("\npostgres_json_files:")
for p in sorted(p for p in paths if p.startswith("postgres/") and p.endswith(".json")):
print(p)
readme_content = None
for item in data.get("tree", []):
if item["path"] == "README.md" and item["type"] == "blob":
with urllib.request.urlopen(item["url"], timeout=20) as r:
readme_content = r.read().decode(errors="replace")
break
print("\nGrafana/version mentions in README.md:")
if readme_content:
for i, line in enumerate(readme_content.splitlines(), 1):
if "Grafana" in line or "7.4" in line or "7.5" in line:
print(f"{i}: {line}")
else:
print("README.md not found")
PYRepository: kubedb/docs
Length of output: 312
🌐 Web query:
github.com opnpulse dashboards master postgres postgres_remote_replica_dashboard.json
💡 Result:
The requested file, postgres_remote_replica_dashboard.json, does not appear to exist within the public repositories of the OpenPulse (opnpulse) GitHub organization [1]. While OpenPulse is an OpenTelemetry-based observability project maintained by AppsCode [1][2], public repositories associated with it, such as opnpulse/installer [3] and opnpulse/alerts [4], focus on infrastructure components, Helm charts [5][6], and runbooks [7], rather than providing a library of specific Grafana dashboard JSON files for PostgreSQL. It is possible the file you are looking for originated in a different repository, such as the AppsCode grafana-dashboards repository, which does contain various PostgreSQL-related dashboards [8][9][10], or it may be a custom or non-public file. For PostgreSQL observability with Prometheus and Grafana, the standard community source for such dashboards is the prometheus-community/postgres_exporter repository [11], which provides officially maintained dashboard templates.
Citations:
- 1: https://github.com/opnpulse
- 2: https://quantumopsschool.com/blog/openpulse/
- 3: https://github.com/opnpulse/installer
- 4: https://github.com/ops-center/alerts
- 5: appscode/charts@7763a23
- 6: appscode/charts@dca022b
- 7: https://github.com/ops-center/runbooks
- 8: https://github.com/appscode/grafana-dashboards/blob/master/postgres/postgres_databases_dashboard.json
- 9: https://github.com/appscode/grafana-dashboards/blob/master/postgres/postgres_summary_dashboard.json
- 10: https://github.com/appscode/grafana-dashboards/blob/master/postgres/postgres_pods_dashboard.json
- 11: https://github.com/prometheus-community/postgres_exporter/blob/master/postgres_mixin/dashboards/postgres-overview.json
Use a dashboard source that contains postgres_remote_replica_dashboard.json.
opnpulse/dashboards@master/postgres does not include the referenced remote-replica JSON file, so the kubectl create configmap --from-file=postgres_remote_replica_dashboard.json command cannot run as written. Patch this step to include/checkout the artifact from a valid revision, or link a source where the file exists.
📍 Affects 1 file
docs/guides/postgres/remote-replica/monitoring.md#L138-L152(this comment)docs/guides/postgres/remote-replica/monitoring.md#L45-L49
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/guides/postgres/remote-replica/monitoring.md` around lines 138 - 152,
The dashboard setup instructions at
docs/guides/postgres/remote-replica/monitoring.md lines 138-152 must use a valid
dashboard source containing postgres_remote_replica_dashboard.json; update the
referenced opnpulse/dashboards revision or link/provide checkout steps for a
source where the artifact exists, and ensure the kubectl command is runnable.
The related reference at docs/guides/postgres/remote-replica/monitoring.md lines
45-49 requires the same source correction; no separate behavior change is
needed.
…ree clusters Whether KubeDB creates its NetworkPolicies is an install-time chart choice; on a cluster without them the scrape-allow policy becomes the only policy selecting the database pods and denies operator health checks, sticking the CR in Provisioning. Signed-off-by: Tamal Saha <tamal@appscode.com>
1 participant
Driven by a customer report (replica looping on
Attempting pg_isready on primarywithverify-fullwhile non-TLS worked).TLS across clusters — new section
verify-cais the official recommendation between clusters: verifies against the pinned private CA without hostname checking, which is correct when the same database is reached via different names inside and outside its cluster.verify-fullas guidance: requires the dialed (external/LB) hostname in the server cert SANs. Includes the exactReconfigureTLSops request that addsdnsNamesto the server certificate on a live database — reissue/rotation handled by ops-manager.pg_isreadyloop + non-TLS works = SAN mismatch, not networking) so it's recognizable.remote-config example refreshed
-d host:port/--port— written into the AppBinding'sservice.port, honored end to end by the replica (seed, streaming, monitor, recovery)--replica-name— emits a ready-to-apply replicaPostgresmanifest sized from the source spec--auth-secret, fixed output path,-ysemantics, CLI version noteAll flags exist on kubedb/cli#832 and were verified live: one command against a TLS source exposed on port 5434, one
kubectl applyon the remote cluster, replica Ready and streaming over 5434.Targets the upcoming release that includes the remote-replica PR stack (kubedb/postgres#908, kubedb/pg-coordinator#257, kubedb/postgres-init-docker#62, kubedb/cli#832).
Summary by CodeRabbit