K8SPG-949: rancher support

[valmiranogueira]

CHANGE DESCRIPTION

Description:

Rancher and Monitoring Improvements

• Update cert-manager installation and verification as it's a dependency used by Rancher.
• Update monitoring tests to use curl commands through the LoadBalancer, since cluster IPs in Rancher are not externally accessible.
• Add Huge Pages configuration to Kubernetes nodes using a DaemonSet (made it generic), updated DOKS to only update additional node.
• Fix the Chaos Mesh deployment by configuring the appropriate container runtime, resolving container crashes.
• Chaos Mesh bandwidth limitations are used for the standby-streaming test, while standby-backrest is using Chaos Mesh iptables commands in order to block access to S3 (this is to avoid to use sidecar containers cause in Openshift operator would require privileged service account - K8SPG-1013)

Additional Improvements

• Removed the dependency on vars.sh during the image build process. This improves build performance and avoids delays caused by timing out kubectl commands.
• Consolidated all templates under the conf directory at both the test-specific and generic levels, providing a more consistent configuration structure.
• Improved and centralized the platform detection logic to reduce duplication and simplify maintenance.
• Standardized shell function declarations by consistently using the function keyword.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PG version?
• Does the change support oldest and newest supported Kubernetes version?

[github-actions]

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

shfmt

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/functions

Lines 2210 to 2211 in 51ab90d

	CSR_REQUEST="$(base64 <${tmp_dir}/server.csr | tr -d '\n')" \
	CSR_SIGNER="${csr_signer}" \

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/functions

Line 2213 in 51ab90d

'.metadata.name = strenv(CSR_NAME) |

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/functions

Line 2216 in 51ab90d

"${TESTS_CONFIG_DIR}/vault-csr.yaml" >${tmp_dir}/csr.yaml

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/functions

Line 2224 in 51ab90d

if [[ "${PLATFORM}" == "openshift" ]]; then

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/functions

Line 2241 in 51ab90d

if [[ "${PLATFORM}" == "openshift" ]]; then

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/functions

Line 2254 in 51ab90d

if [[ "${protocol}" == "https" ]]; then

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/functions

Line 2270 in 51ab90d

if [[ "${PLATFORM}" == "openshift" ]]; then

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/vars.sh

Line 35 in 51ab90d

if [[ -n "${PLATFORM:-}" ]]; then

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/vars.sh

Line 59 in 51ab90d

if [[ "${xtrace_enabled}" == true ]]; then

---

[shfmt] _{reported by reviewdog 🐶}

percona-postgresql-operator/e2e-tests/vars.sh

Line 64 in 51ab90d

if [[ -z "${PLATFORM:-}" ]]; then

[Copilot]

Pull request overview

This PR updates the E2E test harness to better support Rancher (RKE2) and other platforms by centralizing platform detection, shifting monitoring checks to use LoadBalancer access, and replacing OS-dependent iptables test logic with NetworkPolicy/Chaos Mesh based approaches.

Changes:

• Centralized Kubernetes platform detection via PLATFORM and updated tests/scripts to use it consistently.
• Improved Rancher/OpenShift compatibility by replacing iptables-based network manipulation with NetworkPolicy and Chaos Mesh bandwidth chaos, plus runtime-aware Chaos Mesh deployment.
• Added reusable config/templates (Vault Helm values, CSR templates, monitoring QAN payload, hugepages manifests, snapshot/storage classes) and removed vars.sh dependency from the image build script.

Reviewed changes

Copilot reviewed 41 out of 42 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
e2e-tests/vars.sh	Adds detect_k8s_platform() and exports PLATFORM for consistent platform detection.
e2e-tests/tests/upgrade-minor/01-create-cluster.yaml	Switches OpenShift detection to PLATFORM == openshift.
e2e-tests/tests/standby-streaming/99-cleanup.yaml	Ensures Chaos Mesh is torn down during cleanup.
e2e-tests/tests/standby-streaming/05-standby-remove-lockdown.yaml	Removes iptables cleanup; deletes Chaos Mesh resources instead.
e2e-tests/tests/standby-streaming/05-assert.yaml	Reworks standby lag assertions to poll status via kubectl get ... -o json + jq.
e2e-tests/tests/standby-streaming/04-assert.yaml	Reworks lag-detected assertions to polling via jq.
e2e-tests/tests/standby-streaming/03-standby-lockdown.yaml	Replaces iptables-based throttling with Chaos Mesh network chaos helper.
e2e-tests/tests/standby-streaming/03-assert.yaml	Adds an assert step to wait for Chaos Mesh injection.
e2e-tests/tests/standby-streaming/02-create-standby-cluster.yaml	Removes the netshoot sidecar previously used for iptables.
e2e-tests/tests/standby-streaming/00-deploy-operator.yaml	Increases timeout and deploys Chaos Mesh for the test.
e2e-tests/tests/standby-pgbackrest/12-read-from-each-pod-standby.yaml	Minor formatting/line update in loop termination.
e2e-tests/tests/standby-pgbackrest/05-standby-remove-lockdown.yaml	Removes iptables cleanup; deletes NetworkPolicies.
e2e-tests/tests/standby-pgbackrest/05-assert.yaml	Reworks standby-ready assertions to polling via jq.
e2e-tests/tests/standby-pgbackrest/04-assert.yaml	Reworks lag-detected assertions to polling via jq.
e2e-tests/tests/standby-pgbackrest/03-standby-lockdown.yaml	Replaces iptables S3 blocking with a NetworkPolicy helper.
e2e-tests/tests/standby-pgbackrest/02-create-standby-cluster.yaml	Removes the netshoot sidecar previously used for iptables.
e2e-tests/tests/monitoring/conf/qan20-payload.json	Adds a static base QAN payload template used by monitoring tests.
e2e-tests/tests/monitoring/08-check-mountpoint-expose.yaml	Uses execute_curl (in-cluster curl) to query via LoadBalancer address.
e2e-tests/tests/monitoring/04-update-pmm-server-token.yaml	Increases timeout to reduce flakiness.
e2e-tests/tests/monitoring/01-assert.yaml	Adds a command to wait for LoadBalancer ingress before resource assertions.
e2e-tests/tests/k8s-upgrade/00-create-k8s-cluster.yaml	Updates supported upgrade platforms naming and validates current platform.
e2e-tests/tests/init-deploy/05-assert.yaml	Reworks secret assertions to tolerate operator-owned secrets via ownerReferences.
e2e-tests/tests/huge-pages/conf/huge-pages-rancher.yaml	Adds Rancher-specific hugepages cluster manifest.
e2e-tests/tests/huge-pages/conf/huge-pages-openshift-machine-config.yaml	Adds OpenShift MachineConfig template for hugepages sysctl.
e2e-tests/tests/huge-pages/conf/huge-pages-doks.yaml	Adds DOKS-specific hugepages cluster manifest.
e2e-tests/tests/huge-pages/conf/huge-pages-daemonset.yaml	Adds a generic privileged DaemonSet to configure hugepages on labeled nodes.
e2e-tests/tests/huge-pages/01-create-cluster.yaml	Uses PLATFORM and updated hugepages enablement flow.
e2e-tests/tests/huge-pages/00-deploy-operator.yaml	Increases timeout for operator deploy step.
e2e-tests/tests/demand-backup-offline-snapshot/conf/volume-snapshot-class-rancher.yaml	Adds Rancher/Longhorn VolumeSnapshotClass manifest.
e2e-tests/tests/demand-backup-offline-snapshot/conf/storage-class-ebs-gp3.yaml	Adds reusable EBS gp3 StorageClass manifest.
e2e-tests/tests/demand-backup-offline-snapshot/11-create-cluster-from-snapshot.yaml	Switches to using PLATFORM for templating.
e2e-tests/tests/demand-backup-offline-snapshot/01-create-cluster.yaml	Uses a helper to select snapshot class based on platform.
e2e-tests/tests/demand-backup-offline-snapshot/00-deploy-operator.yaml	Uses PLATFORM and deploys snapshot class via helper.
e2e-tests/release_versions	Adds Rancher/RKE2 version pins.
e2e-tests/functions	Centralizes platform usage, adds in-cluster curl helper, runtime-aware Chaos Mesh deploy, NetworkPolicy/Chaos helpers, refactors hugepages + vault templating.
e2e-tests/conf/vault-values-tls.yaml	Adds Helm values template for TLS-enabled Vault installs.
e2e-tests/conf/vault-values-no-tls.yaml	Adds Helm values template for non-TLS Vault installs.
e2e-tests/conf/vault-csr.yaml	Adds CSR manifest template for Vault TLS bootstrap.
e2e-tests/conf/vault-csr.conf	Adds OpenSSL CSR config template with placeholders.
e2e-tests/conf/network-policy-block-external.yml	Adds a NetworkPolicy template used to block external egress in tests.
e2e-tests/conf/chaos-network-delay.yml	Adds a Chaos Mesh NetworkChaos template for bandwidth limitation.
e2e-tests/build	Removes sourcing vars.sh and derives IMAGE tag from the git branch name.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 43 changed files in this pull request and generated no new comments.

Suppressed comments (7)

e2e-tests/tests/standby-streaming/05-assert.yaml:10

• Problem: The script defines timeout/elapsed but never updates or enforces them, so the loop can run until KUTTL kills the step.
  Why it matters: This can turn real failures into long timeouts and makes the failure mode less deterministic.
  Fix: Increment elapsed and fail explicitly once the timeout is reached.
  e2e-tests/tests/standby-streaming/04-assert.yaml:9
• Problem: This until loop has no explicit timeout, so it can run until the overall KUTTL step timeout rather than failing at the intended ~420s bound.
  Why it matters: When the condition can’t be met, this increases runtime and makes failures harder to interpret.
  Fix: Add an elapsed/timeout guard (or wrap the loop in timeout 420s ...) and exit non-zero when exceeded.
  e2e-tests/tests/standby-pgbackrest/04-assert.yaml:10
• Problem: This until loop has no explicit timeout, so it can run until the overall KUTTL step timeout rather than failing at the intended ~420s bound.
  Why it matters: When the condition can’t be met, this increases runtime and makes failures harder to interpret.
  Fix: Add an elapsed/timeout guard (or wrap the loop in timeout 420s ...) and exit non-zero when exceeded.

      set -o errexit
      interval=5

      until
        json=$(kubectl get pg standby-cluster -n "${NAMESPACE}" -o json) &&

e2e-tests/tests/standby-pgbackrest/05-assert.yaml:11

• Problem: This until loop has no explicit timeout, so it can run until the overall KUTTL step timeout.
  Why it matters: When the condition can’t be met, the test will fail as a long timeout instead of a controlled assertion failure.
  Fix: Add an elapsed/timeout guard and exit non-zero when exceeded.

      set -o errexit
      interval=5

      until
        json=$(kubectl get pg standby-cluster -n "${NAMESPACE}" -o json) &&
        state=$(printf '%s' "$json" | jq -r '.status.state // empty') &&

e2e-tests/functions:684

• Problem: execute_curl only waits up to 60s for the curl pod to complete and doesn’t handle kubectl run failures without aborting under set -e.
  Why it matters: Image pulls or scheduling can easily exceed 60s in CI, making monitoring tests flaky even when the endpoint works.
  Fix: Make the wait duration configurable/increase it, and wrap kubectl run in an if ! ...; then block so failures are handled predictably.

	kubectl run "$name" \
		-n "$NAMESPACE" \
		--restart=Never \
		--image="$image" \
		--command -- curl -sS "$@" >/dev/null

e2e-tests/functions:528

• Problem: get_psql_user_host is defined twice in the same file.
  Why it matters: Duplicate function definitions make the script harder to maintain and can hide accidental differences (the later definition silently overrides the earlier one).
  Fix: Remove the duplicate definition and keep a single get_psql_user_host implementation.

function get_psql_user_host() {
	local secret_name=${1}

	kubectl -n ${NAMESPACE} get "secret/${secret_name}" --template='{{.data.host | base64decode }}'
}

e2e-tests/functions:1046

• Problem: This function calls sed directly instead of using the repository’s portable $sed wrapper.
  Why it matters: vars.sh defines sed=$(which gsed || which sed) to handle environments where GNU sed is required; bypassing it can break the script.
  Fix: Use $sed here, consistent with other usages in this file (e.g., e2e-tests/functions:106).

	sed \
		-e "s#test-cluster#${cluster}#g" \
		"$file" | kubectl apply --namespace "${ns}" -f -

[egegunes]

should we rename this file to huge-pages-rancher?

[valmiranogueira]

Not really, it's used for doks and aks

[egegunes]

hmm, then should we change NODE_SERVICE env or is that not used on doks and aks?

[JNKPercona]

Test Name	Result	Time
backup-enable-disable	passed	00:00:00
builtin-extensions	passed	00:00:00
custom-extensions	passed	00:00:00
cert-manager-tls	passed	00:00:00
cert-management-policy	passed	00:00:00
custom-envs	passed	00:00:00
custom-tls	passed	00:00:00
database-init-sql	passed	00:00:00
demand-backup	passed	00:00:00
demand-backup-offline-snapshot	passed	00:00:00
dynamic-configuration	passed	00:00:00
extra-volumes	passed	00:00:00
finalizers	passed	00:00:00
init-deploy	passed	00:00:00
huge-pages	passed	00:00:00
major-upgrade-14-to-15	passed	00:00:00
major-upgrade-15-to-16	passed	00:00:00
major-upgrade-16-to-17	passed	00:00:00
major-upgrade-17-to-18	passed	00:00:00
ldap	passed	00:00:00
ldap-tls	passed	00:00:00
logcollection	passed	00:00:00
monitoring	passed	00:00:00
one-pod	passed	00:00:00
repo-host-autogrow	passed	00:00:00
operator-self-healing	passed	00:00:00
pgbouncer	passed	00:00:00
pg-tde	passed	00:00:00
pg-tde-wal-encrypt	failure	00:14:00
pitr	passed	00:00:00
scaling	passed	00:00:00
scheduled-backup	passed	00:00:00
self-healing	passed	00:00:00
sidecars	passed	00:00:00
standby-pgbackrest	passed	00:00:00
standby-streaming	passed	00:00:00
start-from-backup	passed	00:00:00
tablespaces	passed	00:00:00
telemetry-transfer	passed	00:00:00
upgrade-consistency	passed	00:00:00
upgrade-minor	passed	00:00:00
users	passed	00:00:00
migration-from-crunchy-standby	passed	00:00:00
migration-from-crunchy-pv	passed	00:00:00
migration-from-crunchy-backup-restore	passed	00:00:00

Summary	Value
Tests Run	45/45
Job Duration	01:00:11
Total Test Time	00:14:00

commit: 2952ed0
image: perconalab/percona-postgresql-operator:PR-1737-2952ed0a6