feat(ci): add disconnected OCP smoke test for Helm and Operator by zdrapela · Pull Request #5008 · redhat-developer/rhdh

zdrapela · 2026-06-24T07:49:10Z

https://redhat.atlassian.net/browse/RHIDP-13974

Add end-to-end disconnected (air-gapped) CI smoke tests for RHDH deployed via Helm and Operator on OCP.

What it does

Mirrors RHDH container images, Helm chart, operator, and dynamic plugins to an isolated mirror registry on a disconnected OCP cluster, deploys RHDH, and runs a Playwright smoke test (guest login → homepage).

Architecture

Shared library (`lib/disconnected.sh`)

require_env / setup_auth — validate env vars, configure container-tools auth
build_imageset_config — generate oc-mirror ImageSetConfiguration from chart values
run_oc_mirror — mirror images via oc-mirror --v2
patch_idms — add cross-registry entries to the generated IDMS
fetch_script — download scripts from rhdh-operator

Helm handler (`jobs/ocp-disconnected-helm.sh`)

Chart pull (GA from charts.openshift.io, CI from oci://quay.io/rhdh/chart)
Image mirroring via oc-mirror (hub, PostgreSQL, lightspeed, RAG content)
IDMS/ITMS generation and patching with cross-registry mirrors
Plugin mirroring via mirror-plugins.sh from rhdh-operator
Namespace setup: registries.conf ConfigMap, mirror CA ConfigMap, registry auth Secret
Helm post-renderer to inject disconnected volumes without array clobber
Minimal values file (values_disconnected-smoke.yaml) — guest auth only

Operator handler (`jobs/ocp-disconnected-operator.sh`)

Operator mirroring + installation via prepare-restricted-environment.sh
Plugin mirroring, namespace setup, Backstage CR with registries.conf mount

Post-renderer (`resources/disconnected/helm-post-renderer.sh`)

Patches the rendered Deployment to add:

registries.conf — redirects plugin pulls to mirror registry
policy.json — permissive signature policy for mirrored images
Mirror registry CA cert at /etc/containers/certs.d/<registry>/ca.crt

Issues encountered and fixed

Helm "array clobber" (`extraVolumes` replaced by override file)

Static helm-overrides.yaml defined 4 extraVolumes but chart 1.11 has 7. Helm replaces (not merges) arrays from -f files. Fix: use --post-renderer to append volumes to the rendered manifests.

Missing ConfigMaps (`app-config-rhdh`, `dynamic-plugins-config`)

values_showcase.yaml references ConfigMaps created by apply_yaml_files() which the disconnected handler skips. Fix: use minimal values_disconnected-smoke.yaml with only guest auth.

TLS failure (`x509: certificate signed by unknown authority`)

Init container's skopeo can't verify mirror registry's self-signed CA. Fix: mount CA cert at /etc/containers/certs.d/<registry>/ca.crt (standard container-tools per-registry path).

Auth failure (`authentication required`)

Init container has no credentials for the mirror registry. Fix: create ${RELEASE_NAME}-dynamic-plugins-registry-auth Secret (chart's built-in optional mount).

Signature verification failure (`A signature was required, but no signature exists`)

RHDH image's policy.json requires Red Hat GPG signatures for registry.access.redhat.com. Mirrored images lack signatures (server unreachable). Fix: mount permissive policy.json in init container.

`REGISTRY_AUTH_FILE` conflict with oc-mirror

The distribution/distribution library (used by oc-mirror) interprets REGISTRY_AUTH_FILE as storage driver config, causing panics. Fix: unset before oc-mirror, restore after.

Chart digest encoding (`repo@sha256` + `tag`)

Chart encodes PostgreSQL digest refs as repository: "repo@sha256" + tag: "<hash>". Fix: PG_SEPARATOR normalizes this for IDMS fields.

Companion PR

feat(rhdh): align disconnected CATALOG_INDEX_IMAGE as single Gangway override openshift/release#81128 — CATALOG_INDEX_IMAGE as single Gangway override parameter

openshift-ci · 2026-06-24T07:49:14Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

codecov · 2026-06-24T07:51:02Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.69%. Comparing base (0599876) to head (c1046fd).
⚠️ Report is 31 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5008      +/-   ##
==========================================
+ Coverage   55.39%   63.69%   +8.30%     
==========================================
  Files         122      123       +1     
  Lines        2365     2424      +59     
  Branches      544      572      +28     
==========================================
+ Hits         1310     1544     +234     
+ Misses       1049      878     -171     
+ Partials        6        2       -4

Flag	Coverage Δ
rhdh	`63.69% <ø> (+8.30%)`	⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0599876...c1046fd. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-06-24T07:56:19Z

The container image build workflow finished with status: cancelled.

github-actions · 2026-06-24T08:24:29Z

Image was built and published successfully. It is available at:

github-actions · 2026-06-24T14:16:18Z

The container image build workflow finished with status: cancelled.

zdrapela · 2026-06-24T14:38:34Z

/test e2e-ocp-disconnected-helm-nightly

zdrapela · 2026-06-24T14:38:38Z

/test e2e-ocp-disconnected-operator-nightly

github-actions · 2026-06-24T14:49:27Z

Image was built and published successfully. It is available at:

zdrapela · 2026-06-25T08:32:29Z

/test e2e-ocp-disconnected-helm-nightly

github-actions · 2026-06-25T08:34:07Z

The container image build workflow finished with status: failure.

github-actions · 2026-06-25T09:16:16Z

Image was built and published successfully. It is available at:

zdrapela · 2026-06-25T11:21:26Z

/test e2e-ocp-disconnected-helm-nightly

github-actions · 2026-06-25T13:38:22Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

github-actions · 2026-06-25T13:46:33Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

github-actions · 2026-06-25T13:52:38Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

zdrapela · 2026-06-26T07:33:56Z

/test e2e-ocp-disconnected-helm-nightly

zdrapela · 2026-06-26T07:34:01Z

/test e2e-ocp-disconnected-operator-nightly

github-actions · 2026-06-26T07:38:41Z

The container image build workflow finished with status: cancelled.

zdrapela · 2026-06-26T07:39:14Z

/test e2e-ocp-disconnected-helm-nightly

Helm: After applying IDMS/ITMS, the MachineConfigOperator triggers a rolling update of cluster nodes (drain, reboot). Deploying Helm immediately causes pods to be evicted mid-startup — backstage connects to PG before it's ready and never recovers (503 forever). Fix: wait for all MachineConfigPools to reach Updated=True before deploying. Operator: The c/storage library's getRootlessStorageOpts() ignores the 'graphroot' field in storage.conf for rootless users and defaults to $HOME/.local/share/containers/storage (where chown fails in the CI user namespace). The only field that overrides this is 'rootless_storage_path'. Also set XDG_DATA_HOME so the fallback computation also points to our tmpdir. Move 'podman system reset' before writing configs since some podman versions delete $HOME/.config/containers/ during reset. Assisted-by: OpenCode

The nested-podman entrypoint detects /dev/fuse + fuse-overlayfs and creates an overlay storage config at /home/user/.config/containers/ storage.conf with graphroot=/tmp/graphroot. The VFS driver we were overriding to triggers chown on storage init which fails in the pod's user namespace. Root cause: commands.sh sets HOME=/tmp, so podman no longer finds the entrypoint's config at /home/user/.config/containers/. Our handler then wrote a custom VFS config, but VFS calls chown() on the graphroot directory tree during initialization, and chown is forbidden in the user namespace created by nested_podman: true (hostUsers: false). Fix: run prepare-restricted-environment.sh with HOME=/home/user so podman uses the entrypoint's overlay+fuse-overlayfs config naturally. This matches how other teams (e.g., MCO) use the nested-podman image in CI — no custom storage config needed. Removed: _CONTAINERS_USERNS_CONFIGURED, CONTAINERS_STORAGE_CONF, XDG_DATA_HOME, podman system reset, custom storage.conf. Kept: BUILDAH_ISOLATION=chroot (avoids nested userns for builds). Assisted-by: OpenCode

The entrypoint's storage config creation silently fails because /home/user is not writable by uid 1000 at entrypoint time — the passwd entry that grants gid 0 is created after the mkdir step. All previous attempts used the VFS driver, which calls chown() on the graphroot during storage initialization — forbidden in the pod's user namespace (hostUsers: false). The overlay driver with fuse-overlayfs does NOT chown the graphroot. Fix: write the same overlay+fuse-overlayfs config the entrypoint would have created (driver=overlay, graphroot=/tmp/graphroot, mount_program=/usr/bin/fuse-overlayfs) at the runtime HOME (/tmp, set by commands.sh). Both /dev/fuse and fuse-overlayfs are available in the CI pod (confirmed by previous run's debug output). Assisted-by: OpenCode

Both storage drivers fail in the CI pod's user namespace: - overlay+fuse-overlayfs needs newuidmap which fails (file capabilities from build time are not effective inside the pod's userns) - VFS doesn't need newuidmap but chowns the graphroot directory tree during storage initialization, which is forbidden in the userns Fix: use VFS with _CONTAINERS_USERNS_CONFIGURED=1 (skip newuidmap) and pre-create the graphroot directory tree (vfs/dir, vfs-images, vfs-layers, runroot/vfs, runroot/libpod) so that the store init's MkdirAllAndChown finds existing dirs owned by uid 1000 and skips the chown. BUILDAH_ISOLATION=chroot avoids nested userns for builds. ignore_chown_errors covers layer-level chown operations. Write the config at the runtime HOME (/tmp, set by commands.sh) at ${HOME}/.config/containers/storage.conf — the standard rootless path. Assisted-by: OpenCode

… Podman Ubuntu Podman cannot initialize storage under hostUsers:false. Switch prepare-restricted-environment.sh to --use-oc-mirror true so catalog mirroring no longer requires podman build, and stop wiring the plugin catalog index as the OLM --index-image. Co-authored-by: Cursor <cursoragent@cursor.com>

OCP 4.21 auto-detects OLM v1, but prepare-restricted-environment.sh then fails applying a missing clusterCatalog.yaml after oc-mirror. Use --olm-version v0 (CatalogSource/Subscription) and map RELEASE_VERSION next to '*' for filter-versions. Co-authored-by: Cursor <cursoragent@cursor.com>

The full rhdh-start.yaml references ConfigMaps/Secrets from the normal apply_yaml_files() flow. Use a guest-auth-only CR and force NAME_SPACE=showcase-disconnected so env_variables.sh cannot override it.

Avoid hardcoding redhat-operator-index:v4.21 so the job matches the live cluster major.minor (same oc version pattern as container-init.sh).

Temporarily fetch prepare-restricted-environment.sh from rhdh-operator PR redhat-developer#3259 so oc-mirror + OLM v1 can skip the missing clusterCatalog.yaml on main, and drop the --olm-version v0 workaround.

Extract duplicated Helm/Operator post-mirror steps into disconnected.sh so both handlers call the same helpers without changing deploy behavior.

Use the cluster major.minor already exported by container-init.sh instead of re-querying oc version in the disconnected operator handler.

Merge mirror credentials into the cluster pull-secret, attach a real reg-pull-secret to the installer SA, and wait on ClusterExtension with status dumps when the Backstage CRD never appears.

ClusterCatalog was stuck not Serving with x509 unknown authority. Add the mirror registry CA to image.config additionalTrustedCA and wait for MCP before running prepare.

install-dynamic-plugins failed with x509 on the mirrored catalog index because the operator CR only mounted registries.conf. Match Helm: mount mirror CA, policy.json, and registry auth for the init container.

CI mikefarah/yq rejected --arg when patching Backstage extraFiles. Interpolate mirror CA path and auth secret name via the shell instead.

Point prepare-restricted-environment.sh at refs/pull/3259/head so CI picks up the newest revision of the OLM v1 oc-mirror fix automatically.

Operator reconcile failed because registries.conf and policy.json both referenced rhdh-plugin-mirror-conf, creating duplicate volume names. Mount policy.json from a separate ConfigMap instead.

Operator subPath mounts to .config/containers failed (Not a directory). Mount auth.json under /tmp and set REGISTRY_AUTH_FILE for the init container.

…ator smoke Operator smoke was CrashLooping on plugin-catalog-index (not mirrored) and still starting lightspeed sidecars. Clear CATALOG_INDEX_IMAGE and set flavours: [] so guest→homepage smoke needs neither.

Stop patching the smoke CR with quay.io/rhdh-community/rhdh from IMAGE_* defaults. That ref is not in prepare IDMS; CSV registry.redhat.io hub is.

Empty CATALOG_INDEX_IMAGE skipped dynamic-plugins.default.yaml, so the home-page plugin never installed and guest login hit a / 404. Use the index already mirrored by mirror_plugins instead.

plugin-catalog-index:next enables a local homepage dist path that GA rhdh-hub-rhel9 lacks. Match Helm by skopeo-copying the CI hub to the mirror and pointing the smoke CR at it (flavours remain empty).

Remove leftover ConfigMap wiring from a conflicting edit; homepage comes from catalog defaults on the mirrored CI hub image.

Operator getInitContainer overwrites CR init image patches with RELATED_IMAGE_backstage (CSV GA). Point RELATED_IMAGE at the mirrored CI hub and post-patch the Deployment so homepage plugins resolve.

rhdh-operator#3259 is merged on main, so drop the temporary pull/3259 pin.

zdrapela · 2026-07-27T10:49:57Z

/test e2e-ocp-disconnected-operator-nightly

github-actions · 2026-07-27T10:50:22Z

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

sonarqubecloud · 2026-07-27T10:50:31Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

openshift-ci · 2026-07-27T12:49:08Z

@zdrapela: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ocp-disconnected-operator-nightly	`09442eb`	link	false	`/test e2e-ocp-disconnected-operator-nightly`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci Bot added the do-not-merge/work-in-progress label Jun 24, 2026

zdrapela force-pushed the rhdh.3 branch from 74fdb84 to a50ed35 Compare June 24, 2026 14:14

zdrapela force-pushed the rhdh.3 branch from f963573 to 09f8971 Compare June 25, 2026 08:45

zdrapela force-pushed the rhdh.3 branch from 09f8971 to 11c3e51 Compare June 25, 2026 13:37

zdrapela force-pushed the rhdh.3 branch from 11c3e51 to 0759110 Compare June 25, 2026 13:45

zdrapela force-pushed the rhdh.3 branch from 0759110 to f7f7ec3 Compare June 25, 2026 13:52

zdrapela force-pushed the rhdh.3 branch from f7f7ec3 to 29b5277 Compare June 26, 2026 07:33

zdrapela changed the title ~~feat(ci): add disconnected CI smoke tests for Helm deployment~~ feat(ci): add disconnected OCP smoke test for Helm and Operator Jun 26, 2026

zdrapela mentioned this pull request Jun 26, 2026

feat(rhdh): align disconnected CATALOG_INDEX_IMAGE as single Gangway override openshift/release#81128

Open

zdrapela force-pushed the rhdh.3 branch from 29b5277 to 95b9364 Compare June 26, 2026 07:38

zdrapela force-pushed the rhdh.3 branch from 95b9364 to 16819c2 Compare June 26, 2026 07:39

zdrapela and others added 25 commits July 27, 2026 11:41

fix(disconnected): use minimal Backstage CR for operator smoke

5b47e6a

The full rhdh-start.yaml references ConfigMaps/Secrets from the normal apply_yaml_files() flow. Use a guest-auth-only CR and force NAME_SPACE=showcase-disconnected so env_variables.sh cannot override it.

fix(disconnected): derive operator index image from cluster OCP version

2fa96d6

Avoid hardcoding redhat-operator-index:v4.21 so the job matches the live cluster major.minor (same oc version pattern as container-init.sh).

fix(disconnected): pin prepare script for OLM v1 oc-mirror path

9954c09

Temporarily fetch prepare-restricted-environment.sh from rhdh-operator PR redhat-developer#3259 so oc-mirror + OLM v1 can skip the missing clusterCatalog.yaml on main, and drop the --olm-version v0 workaround.

refactor(disconnected): share MCP, plugin-mirror, and auth helpers

7929a6d

Extract duplicated Helm/Operator post-mirror steps into disconnected.sh so both handlers call the same helpers without changing deploy behavior.

fix(disconnected): reuse CONTAINER_PLATFORM_VERSION for OLM index

cc90671

Use the cluster major.minor already exported by container-init.sh instead of re-querying oc version in the disconnected operator handler.

fix(disconnected): help OLM v1 pull mirrored operator images

404b479

Merge mirror credentials into the cluster pull-secret, attach a real reg-pull-secret to the installer SA, and wait on ClusterExtension with status dumps when the Backstage CRD never appears.

fix(disconnected): trust mirror CA for OLM v1 catalogd pulls

c8b4a50

ClusterCatalog was stuck not Serving with x509 unknown authority. Add the mirror registry CA to image.config additionalTrustedCA and wait for MCP before running prepare.

fix(disconnected): mount mirror CA for operator plugin installs

6c9b4de

install-dynamic-plugins failed with x509 on the mirrored catalog index because the operator CR only mounted registries.conf. Match Helm: mount mirror CA, policy.json, and registry auth for the init container.

fix(disconnected): avoid unsupported yq --arg in operator CR render

901d536

CI mikefarah/yq rejected --arg when patching Backstage extraFiles. Interpolate mirror CA path and auth secret name via the shell instead.

fix(disconnected): always fetch prepare script from operator PR head

0becc3c

Point prepare-restricted-environment.sh at refs/pull/3259/head so CI picks up the newest revision of the OLM v1 oc-mirror fix automatically.

fix(disconnected): avoid duplicate extraFiles ConfigMap volumes

295c6d4

Operator reconcile failed because registries.conf and policy.json both referenced rhdh-plugin-mirror-conf, creating duplicate volume names. Mount policy.json from a separate ConfigMap instead.

fix(disconnected): mount registry auth where the image allows it

b0cb855

Operator subPath mounts to .config/containers failed (Not a directory). Mount auth.json under /tmp and set REGISTRY_AUTH_FILE for the init container.

fix(disconnected): disable lightspeed and skip catalog index for oper…

083b4b5

…ator smoke Operator smoke was CrashLooping on plugin-catalog-index (not mirrored) and still starting lightspeed sidecars. Clear CATALOG_INDEX_IMAGE and set flavours: [] so guest→homepage smoke needs neither.

fix(disconnected): use operator default hub image for smoke

d8300ca

Stop patching the smoke CR with quay.io/rhdh-community/rhdh from IMAGE_* defaults. That ref is not in prepare IDMS; CSV registry.redhat.io hub is.

fix(disconnected): point operator smoke at mirrored catalog index

7134448

Empty CATALOG_INDEX_IMAGE skipped dynamic-plugins.default.yaml, so the home-page plugin never installed and guest login hit a / 404. Use the index already mirrored by mirror_plugins instead.

fix(disconnected): mirror CI hub for operator catalog defaults

41f186c

plugin-catalog-index:next enables a local homepage dist path that GA rhdh-hub-rhel9 lacks. Match Helm by skopeo-copying the CI hub to the mirror and pointing the smoke CR at it (flavours remain empty).

fix(disconnected): drop broken dynamic-plugins ConfigMap path

72f9376

Remove leftover ConfigMap wiring from a conflicting edit; homepage comes from catalog defaults on the mirrored CI hub image.

fix(disconnected): force mirrored hub on install-dynamic-plugins

8f57e62

Operator getInitContainer overwrites CR init image patches with RELATED_IMAGE_backstage (CSV GA). Point RELATED_IMAGE at the mirrored CI hub and post-patch the Deployment so homepage plugins resolve.

chore(ci): fetch prepare-restricted from RELEASE_BRANCH_NAME

09442eb

rhdh-operator#3259 is merged on main, so drop the temporary pull/3259 pin.

zdrapela force-pushed the rhdh.3 branch from 53e4e49 to 09442eb Compare July 27, 2026 10:49

Uh oh!

Conversation

zdrapela commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it does

Architecture

Shared library (lib/disconnected.sh)

Helm handler (jobs/ocp-disconnected-helm.sh)

Operator handler (jobs/ocp-disconnected-operator.sh)

Post-renderer (resources/disconnected/helm-post-renderer.sh)

Issues encountered and fixed

Helm "array clobber" (extraVolumes replaced by override file)

Missing ConfigMaps (app-config-rhdh, dynamic-plugins-config)

TLS failure (x509: certificate signed by unknown authority)

Auth failure (authentication required)

Signature verification failure (A signature was required, but no signature exists)

REGISTRY_AUTH_FILE conflict with oc-mirror

Chart digest encoding (repo@sha256 + tag)

Companion PR

Uh oh!

openshift-ci Bot commented Jun 24, 2026

Uh oh!

codecov Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

zdrapela commented Jun 24, 2026

Uh oh!

zdrapela commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

zdrapela commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

zdrapela commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

zdrapela commented Jun 26, 2026

Uh oh!

zdrapela commented Jun 26, 2026

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

zdrapela commented Jun 26, 2026

Uh oh!

zdrapela commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

sonarqubecloud Bot commented Jul 27, 2026

Quality Gate passed

Uh oh!

openshift-ci Bot commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

zdrapela commented Jun 24, 2026 •

edited

Loading

Shared library (`lib/disconnected.sh`)

Helm handler (`jobs/ocp-disconnected-helm.sh`)

Operator handler (`jobs/ocp-disconnected-operator.sh`)

Post-renderer (`resources/disconnected/helm-post-renderer.sh`)

Helm "array clobber" (`extraVolumes` replaced by override file)

Missing ConfigMaps (`app-config-rhdh`, `dynamic-plugins-config`)

TLS failure (`x509: certificate signed by unknown authority`)

Auth failure (`authentication required`)

Signature verification failure (`A signature was required, but no signature exists`)

`REGISTRY_AUTH_FILE` conflict with oc-mirror

Chart digest encoding (`repo@sha256` + `tag`)

codecov Bot commented Jun 24, 2026 •

edited

Loading