Fin pr duckdb check 1 - #222
Draft
stutiibm wants to merge 47 commits into
Draft
Conversation
Refactor installation commands to remove unnecessary sudo usage and streamline the installation of dependencies for Python versions. Update CVE scan handling to improve clarity and functionality.
Refactor read_buildinfo.sh to improve readability and structure. Added a new function to read the 'Tested on' value from build scripts and updated the handling of build_script to support both string and array formats.
Added logic to determine version based on changed .sh files.
…able.sh variable.sh always exports BUILD_SCRIPT and TESTED_ON for the first/backward- compat script (UBI8), clobbering the correct per-job values set in the GHA env: block before build_wheels.sh runs. This caused all wheel jobs (ubi9_*, ubi10_*) to pull the UBI8 image and run the UBI8 build script. Fix: in every wheel Run build_wheels.sh step, capture THIS_SCRIPT and THIS_TESTED_ON from the GHA env before sourcing variable.sh, then re-export them as BUILD_SCRIPT and TESTED_ON after the source calls. Applied to all 11 wheel_build_ubiN_pyXXX jobs.
…ariable.sh All 12 wheel_build_ubiN_pyXXX jobs (ubi8: py310-313, ubi9: py310-314, ubi10: py312-314) now capture the GHA job-level BUILD_SCRIPT and TESTED_ON before sourcing variable.sh (which overwrites them with the UBI8 backward- compat values), then restore them after. This ensures each wheel job uses the correct UBI image and build script for its assigned UBI version.
- Add script_ubi8, script_ubi9, script_ubi10 as named build_info
job outputs, each a JSON object {script, tested_on} or empty string
- Add Emit per-UBI outputs step that reads SCRIPT_UBI8/9/10 from
variable.sh and emits them as workflow outputs
- Add build_ubi8 and build_ubi10 jobs that run only when their
respective scripts exist; build_ubi9 runs as before
- Add version-block best-match detection: count how many changed
scripts each build_info.json version block contains and pick the
highest-count block, preventing shared scripts from routing to
the wrong alphabetically-earlier block
- Add safety-net in emit step: fill any empty UBI slot directly from
the changed script's own # Tested on header, guaranteeing every
changed script in the PR is tested
- Fix BUILD_SCRIPT/TESTED_ON overwrite bug in all build_ubiN and all
12 wheel_build_ubiN_pyXXX jobs: variable.sh always exports the
backward-compat UBI8 script, clobbering the GHA job-level env vars
before the build runs; fixed by capturing THIS_SCRIPT/THIS_TESTED_ON
before source variable.sh and re-exporting after
- Fix scanner-env.sh writer: replace indented heredoc with echo and
printf %q to preserve CHANGED_FILES newlines correctly
- Remove wheel_build_ubi8_py314 (Python 3.14 not supported on UBI8)
- Add wheel_build_ubi10_py312, wheel_build_ubi10_py313,
wheel_build_ubi10_py314 jobs
- Add script_ubi8, script_ubi9, script_ubi10 outputs to build_info job - Add Emit per-UBI outputs step - Add build_ubi8 and build_ubi10 jobs - Add wheel_build_ubi10_py311 through wheel_build_ubi10_py314 jobs - Remove wheel_build_ubi8_py314 (Python 3.14 not supported on UBI8) - Remove branches: master restriction from workflow_dispatch trigger
- Add read_tested_on() helper to extract and normalise the '# Tested on' header value from any build script file (uppercase, collapse separators) - Add BUILD_SCRIPTS_JSON array construction supporting both string and list values for the build_script field in build_info.json - Add SCRIPT_UBI8, SCRIPT_UBI9, SCRIPT_UBI10 bucketing by parsing the UBI major version from each script's # Tested on header - Support build_script as a JSON array in version blocks (not just string) - Export SCRIPT_UBI8, SCRIPT_UBI9, SCRIPT_UBI10 in variable.sh so downstream jobs can consume the correct per-UBI script
- Replace hard-coded UBI8/UBI9/UBI10 if/elif chain with a resolve_docker_image() function that parses UBI major version from the TESTED_ON variable using sed (no grep -P which is absent on ppc64le runners) - Supports all TESTED_ON formats: UBI:8.10, UBI 9.3, UBI:10.0, ubi10.0, UBI : 10.0 etc. - Resolves correct registry image for UBI8, UBI9 and UBI10
- Replace hard-coded UBI8/UBI9/UBI10 if/elif chain with the same resolve_docker_image() function used in build_package.sh - Parses UBI major version from TESTED_ON using sed only - Resolves correct registry image for UBI8, UBI9 and UBI10
- Pass GRYPE_BIN environment variable into the Docker container so generalized_wheel_scanner.py can locate the pre-cached grype binary mounted at /home/tester/scan-tools-bin/grype - Pass ENABLE_CVE_SCAN environment variable so PR builds can set it to false and skip the CVE scan step inside the container
- Select GCC toolset version based on UBI major version: gcc-toolset-15 for UBI10+, gcc-toolset-13 for UBI8/UBI9 - On UBI10, SCL (Software Collections) was dropped so no enable script is sourced; PATH is updated directly instead
- Locate grype using GRYPE_BIN environment variable (set by
build_wheels.py to the pre-cached binary path) with fallback
to shutil.which('grype') for local/manual runs
- Previously relied solely on shutil.which which fails inside the
container because grype is not on the container PATH
Updated COS configuration to handle missing credentials gracefully in non-currency builds.
- Remove runtime grype download (wget + tar + install on every run) - Use $GRYPE_BIN environment variable pointing to the pre-cached binary from the workflow's install_scan_tools artifact - Fail fast with a clear error if GRYPE_BIN is not set
- Remove runtime grype download (wget + tar + install on every run) - Use $GRYPE_BIN environment variable pointing to the pre-cached binary from the workflow's install_scan_tools artifact - Fail fast with a clear error if GRYPE_BIN is not set
- Change GITHUB_BUILD_SCRIPT_BASE_OWNER from stutiibm back to ppc64le so PR validation fetches file lists from the correct upstream repo
Removed wheel build jobs for Python 3.10 and 3.13 in UBI8.
Ubi10 pipeline changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
set -eoption enabled and observe success ?