CNTRLPLANE-4009: feat(aro-hcp): add e2e-aro-hcp-presubmit job with CPO override and nightly support

[bryan-cox]

Summary

• Adds a new e2e-aro-hcp-presubmit CI job that tests both the HyperShift Operator (HO) and Control Plane Operator (CPO) from a hypershift PR against an ARO-HCP environment using nightly OCP release images
• Makes CPO push conditional in the images-push step — only runs when HYPERSHIFT_CPO_IMAGE is provided via a job-level dependency override (existing e2e-aro-hcp HO-only job is unaffected)
• Adds ARO_HCP_E2E_SUITE, ARO_HCP_OPENSHIFT_CHANNEL_GROUP, and ARO_HCP_OPENSHIFT_VERSION_ID env vars to the test step
• Sets ARO_HCP_E2E_SUITE=hypershift-presubmit/parallel in the presubmit job so it runs the correct test suite

Why

CNTRLPLANE-4009 / CNTRLPLANE-3980

CPO is not backwards compatible — it must run alongside matching control plane components from the same OCP development stream. Using a released stable version with CPO from main would crash. The solution uses nightly OCP release images (already discovered by Cluster Service from multi.ocp.releases.ci.openshift.org/graph, cached via ACR pull-through) so CPO from the PR gets a compatible release image.

The new job:

• Pushes HO to SVC ACR (same as existing job)
• Pushes CPO to OCP ACR (HCP namespaces only have OCP ACR pull creds)
• Sets ARO_HCP_E2E_SUITE=hypershift-presubmit/parallel to run the presubmit-specific test suite
• Sets ARO_HCP_OPENSHIFT_CHANNEL_GROUP=nightly and ARO_HCP_OPENSHIFT_VERSION_ID=5.0 so the test framework resolves the latest accepted 5.0 nightly
• CPO override flows via ARM experimental tag → admission → Cosmos → CS dispatch → HostedCluster annotation

Companion PR: Azure/ARO-HCP#6456 (test, framework, and Makefile changes)

Test plan

• Verify e2e-aro-hcp (HO-only) job is unaffected — no CPO push, no nightly env vars, same default suite
• Verify e2e-aro-hcp-presubmit job pushes both HO and CPO images
• Verify presubmit job runs hypershift-presubmit/parallel suite (not rp-api-compat-all/parallel)
• Verify nightly version resolution works with ARO_HCP_OPENSHIFT_CHANNEL_GROUP=nightly and ARO_HCP_OPENSHIFT_VERSION_ID=5.0
• Verify CPO override env file is sourced by test step

🤖 Generated with Claude Code

Summary by CodeRabbit

Adds the e2e-aro-hcp-presubmit CI job for HyperShift pull requests. The job tests HyperShift Operator and Control Plane Operator changes against ARO-HCP with nightly OpenShift 5.0 release images.

When HYPERSHIFT_CPO_IMAGE is set, CI pushes the Control Plane Operator image to the OCP ACR and passes its digest to the test workflow. CI continues to push the HyperShift Operator image to the SVC ACR.

The existing HO-only e2e-aro-hcp job remains unchanged.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: f65ff4e3-6a6e-4978-9bc0-8a9ee5849a52

📥 Commits

Reviewing files that changed from the base of the PR and between bbd3c97 and cf24bfa.

📒 Files selected for processing (5)

• ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
• ci-operator/step-registry/aro-hcp/hypershift-images-push/aro-hcp-hypershift-images-push-commands.sh
• ci-operator/step-registry/aro-hcp/hypershift-images-push/aro-hcp-hypershift-images-push-ref.yaml
• ci-operator/step-registry/aro-hcp/test/local/aro-hcp-test-local-commands.sh
• ci-operator/step-registry/aro-hcp/test/local/aro-hcp-test-local-ref.yaml

🚧 Files skipped from review as they are similar to previous changes (4)

• ci-operator/step-registry/aro-hcp/test/local/aro-hcp-test-local-commands.sh
• ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
• ci-operator/step-registry/aro-hcp/hypershift-images-push/aro-hcp-hypershift-images-push-commands.sh
• ci-operator/step-registry/aro-hcp/test/local/aro-hcp-test-local-ref.yaml

---

Walkthrough

The change adds an optional ARO HCP nightly presubmit job, supports optional CPO image publishing, passes the CPO digest to local tests, and adds configurable OpenShift channel and version settings.

Changes

ARO HCP CPO presubmit

Layer / File(s)	Summary
Publish and record the CPO image ci-operator/step-registry/aro-hcp/hypershift-images-push/*	When enabled, the push step authenticates to the OCP ACR, mirrors the CPO image, resolves its digest, and writes the CPO override environment file.
Consume the CPO override in local tests ci-operator/step-registry/aro-hcp/test/local/*	The local test step optionally sources the CPO override. New settings select the e2e suite, OpenShift channel group, and nightly minor version.
Configure the nightly presubmit job ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml	An optional e2e-aro-hcp-presubmit job uses the ARO HCP development profile, nightly OpenShift settings, CPO image publishing, and the existing workflow.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PushStep as hypershift-images-push
  participant OCPACR as OCP ACR
  participant SharedDir as SHARED_DIR
  participant LocalTest as aro-hcp-test-local
  PushStep->>OCPACR: Authenticate and mirror HYPERSHIFT_CPO_IMAGE
  OCPACR-->>PushStep: Return CPO image digest
  PushStep->>SharedDir: Write CPO_IMAGE_OVERRIDE
  LocalTest->>SharedDir: Source hypershift-cpo-override.env
  LocalTest->>LocalTest: Apply suite and OpenShift version settings

Loading

Possibly related PRs

• openshift/release#82403: Both changes add ARO HCP e2e configurations that use the aro-hcp-dev profile and aro-hcp-hypershift-e2e workflow.

Suggested reviewers: enxebre, deads2k

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	Added logs print OCP_ACR_URL and full CPO image references at lines 33, 76, and 141-147, exposing internal *.azurecr.io hostnames.	Redact registry hostnames from logs. Report only operation status and non-sensitive repository or digest identifiers.

✅ Passed checks (14 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the new e2e-aro-hcp-presubmit job, CPO override support, and nightly OCP support.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The PR changes only CI configuration and shell steps; no Ginkgo title APIs or dynamic test-title expressions were added. New suite and job identifiers are static.
Test Structure And Quality	✅ Passed	The commit changes only CI YAML and shell step wiring; it adds no Ginkgo test code or It blocks, so the listed test-quality requirements are not applicable.
Microshift Test Compatibility	✅ Passed	The PR changes only CI YAML and shell/ref files; it adds no Ginkgo tests or It/Describe/Context/When declarations, so the MicroShift test check is not applicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The PR changes only CI YAML and shell files. It adds no Ginkgo e2e tests or test source paths, so SNO compatibility checks do not apply.
Topology-Aware Scheduling Compatibility	✅ Passed	The patch modifies only CI configuration, step references, and shell scripts. No deployment manifests, operator code, controllers, or topology scheduling constraints were added or changed.
Ote Binary Stdout Contract	✅ Passed	The PR changes only YAML and Bash CI orchestration. It adds no OTE/Go binary entry points or stdout logging; shell echo/cat output is CI logging, not openshift-tests JSON.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The PR adds only CI configuration and shell/YAML steps; it adds no Ginkgo test declarations, hardcoded IPs, or public-endpoint test code.
No-Weak-Crypto	✅ Passed	The five changed files and added lines contain no MD5, SHA-1, DES, 3DES, RC4, Blowfish, ECB, custom crypto, or secret/token comparisons.
Container-Privileges	✅ Passed	The added job, workflow references, step metadata, and scripts contain no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or root settings.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bryan-cox
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift/hypershift/OWNERS [bryan-cox]
• ci-operator/step-registry/aro-hcp/hypershift-images-push/OWNERS [bryan-cox]
• ci-operator/step-registry/aro-hcp/test/local/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[openshift-ci-robot]

@bryan-cox: This pull request references CNTRLPLANE-4009 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

• Adds a new e2e-aro-hcp-presubmit CI job that tests both the HyperShift Operator (HO) and Control Plane Operator (CPO) from a hypershift PR against an ARO-HCP environment using nightly OCP release images
• Makes CPO push conditional in the images-push step — only runs when HYPERSHIFT_CPO_IMAGE is provided via a job-level dependency override (existing e2e-aro-hcp HO-only job is unaffected)
• Adds ARO_HCP_E2E_SUITE, ARO_HCP_OPENSHIFT_CHANNEL_GROUP, and ARO_HCP_OPENSHIFT_VERSION_ID env vars to the test step
• Sets ARO_HCP_E2E_SUITE=hypershift-presubmit/parallel in the presubmit job so it runs the correct test suite

Why

CNTRLPLANE-4009 / CNTRLPLANE-3980

CPO is not backwards compatible — it must run alongside matching control plane components from the same OCP development stream. Using a released stable version with CPO from main would crash. The solution uses nightly OCP release images (already discovered by Cluster Service from multi.ocp.releases.ci.openshift.org/graph, cached via ACR pull-through) so CPO from the PR gets a compatible release image.

The new job:

• Pushes HO to SVC ACR (same as existing job)
• Pushes CPO to OCP ACR (HCP namespaces only have OCP ACR pull creds)
• Sets ARO_HCP_E2E_SUITE=hypershift-presubmit/parallel to run the presubmit-specific test suite
• Sets ARO_HCP_OPENSHIFT_CHANNEL_GROUP=nightly and ARO_HCP_OPENSHIFT_VERSION_ID=5.0 so the test framework resolves the latest accepted 5.0 nightly
• CPO override flows via ARM experimental tag → admission → Cosmos → CS dispatch → HostedCluster annotation

Companion PR: Azure/ARO-HCP#6456 (test, framework, and Makefile changes)

Test plan

• Verify e2e-aro-hcp (HO-only) job is unaffected — no CPO push, no nightly env vars, same default suite
• Verify e2e-aro-hcp-presubmit job pushes both HO and CPO images
• Verify presubmit job runs hypershift-presubmit/parallel suite (not rp-api-compat-all/parallel)
• Verify nightly version resolution works with ARO_HCP_OPENSHIFT_CHANNEL_GROUP=nightly and ARO_HCP_OPENSHIFT_VERSION_ID=5.0
• Verify CPO override env file is sourced by test step

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@bryan-cox: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-hypershift-main-e2e-aro-hcp	openshift/hypershift	presubmit	Registry content changed
pull-ci-openshift-hypershift-release-5.1-e2e-aro-hcp	openshift/hypershift	presubmit	Registry content changed
pull-ci-openshift-hypershift-release-5.0-e2e-aro-hcp	openshift/hypershift	presubmit	Registry content changed
pull-ci-openshift-hypershift-release-4.23-e2e-aro-hcp	openshift/hypershift	presubmit	Registry content changed
pull-ci-Azure-ARO-HCP-main-e2e-parallel	Azure/ARO-HCP	presubmit	Registry content changed
pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel	Azure/ARO-HCP	presubmit	Registry content changed
periodic-ci-openshift-hypershift-release-5.1-periodics-e2e-aro-hcp	N/A	periodic	Registry content changed
periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aro-hcp	N/A	periodic	Registry content changed

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[openshift-ci]

@bryan-cox: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/generated-config	cf24bfa	link	true	/test generated-config

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.