fix(pipelines/tiflash): release-6.5 IT runtime needs mysql client - #4850
Conversation
…time Builder image v20231106 has no mysql client, so fullstack run-test.sh fails with "mysql: command not found" after rewriting tiflash-ci-base. Rebuild the historical runtime from ghcr.io/pingcap-qe/bases/tiflash-base:v1.9.1 with yum install mysql (same as jenkins tiflash_ci_base Dockerfile).
There was a problem hiding this comment.
I have already done a preliminary review for you, and I hope to help you do a better job.
Summary:
This PR addresses the missing mysql client in the tiflash-ci-base image used during integration testing for TiFlash release-6.5. Instead of using the previous untagged image, it switches to a specific base image ghcr.io/pingcap-qe/bases/tiflash-base:v1.9.1 and locally builds a new image with mysql installed via yum. The approach is straightforward and clearly solves the missing client issue. The patch is concise and focuses on the relevant pipeline Groovy file with reasonable inline Dockerfile usage. Overall, the change is well scoped but can be improved for robustness and maintainability.
Critical Issues
- Potential docker build caching issues (pipelines/pingcap/tiflash/release-6.5/pull_integration_test.groovy, line ~280)
- The Docker image
tiflash-ci-base-local:with-mysqlis built every run without cache invalidation or tagging by commit/version. - This might cause stale builds or unnecessary rebuilds, which is inefficient and could lead to confusion if the base image updates but the local image does not.
- Suggestion: Add a unique tag incorporating the base image digest or a build timestamp, or use a build cache strategy to ensure builds are only triggered when necessary.
- The Docker image
Code Improvements
-
Hardcoded image tag and inline Dockerfile (pull_integration_test.groovy, lines ~278-288)
- Embedding the Dockerfile inline in the Groovy script makes the pipeline harder to maintain and review.
- This approach also mixes concerns: pipeline logic and image definition.
- Suggestion: Extract this Dockerfile into a dedicated
Dockerfile.mysqlor similar in the repo, and build from that file. This improves clarity and allows independent testing of the image build.
-
No error handling for docker build failures
- The
docker buildcommand is run inline but there is no explicit error handling or retry logic. - If the build fails (network, yum repo issues), the pipeline will fail but may not produce clear logs.
- Suggestion: Add error checking or capture logs clearly. Optionally, retry the build or fail fast with an informative message.
- The
-
Inconsistent environment variable naming
- The original variable
TIFLASH_IMAGEis replaced byTIFLASH_IT_BASE_IMAGEin the environment block, but laterTIFLASH_IMAGEis reassigned locally. - This can be confusing for readers and maintainers.
- Suggestion: Keep consistent variable naming or clearly comment the difference between base image and final test image.
- The original variable
Best Practices
-
Missing comments on the Docker build snippet
- Although there is a short comment explaining the reason for the build, the Dockerfile snippet itself lacks explanation.
- Adding a comment within the Dockerfile snippet would clarify the purpose of the
yum install -y mysqlstep.
-
Testing coverage not confirmed
- The PR description mentions manual test plan steps but no automated tests or pipeline checks to verify the image contains
mysql. - Suggestion: Add a quick smoke test after build (e.g.,
docker run --rm tiflash-ci-base-local:with-mysql mysql --version) to verify installation succeeded.
- The PR description mentions manual test plan steps but no automated tests or pipeline checks to verify the image contains
-
Style: shell script quoting and heredoc usage
- Using a heredoc with
<<EOFis fine, but consider using<<-'EOF'or quoting the EOF delimiter to prevent unwanted variable expansion if any variables are introduced later. - This is minor but can prevent subtle bugs.
- Using a heredoc with
Summary of Suggested Fixes
// Extract dockerfile to pipelines/pingcap/tiflash/release-6.5/Dockerfile.mysql
// with content:
// FROM ghcr.io/pingcap-qe/bases/tiflash-base:v1.9.1
// RUN yum install -y mysql
timeout 600 docker pull "${TIFLASH_IT_BASE_IMAGE}"
// Use a unique tag with timestamp or digest
def mysqlImageTag = "tiflash-ci-base-local:with-mysql-${env.BUILD_ID}"
sh "docker build -t ${mysqlImageTag} -f pipelines/pingcap/tiflash/release-6.5/Dockerfile.mysql ."
// Optional test to verify mysql client
sh "docker run --rm ${mysqlImageTag} mysql --version"
// Use consistent env var for downstream usage
TIFLASH_IMAGE = mysqlImageTagThis will improve maintainability, caching, and robustness of the fix.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dillon-zheng The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
…4852) ## Summary Follow-up after #4849 / #4850. `tidb-ci/cluster.yaml` on release-6.5 uses `qa/tidb:...-failpoint`. The previous logic only skipped `fail-point-tests` when the failpoint image **pull failed**. OCI still has `tidb-server:release-6.5-failpoint`, so pull succeeds, but the image does not expose `enableTestAPI`: ```text Error: get fail/github.com/pingcap/tidb/server/enableTestAPI ``` ## Change - Always fall back `TIDB_FAILPOINT_IMAGE` → regular `release-6.5` TiDB image - Always strip `./run-test.sh tidb-ci/fail-point-tests &&` from `run.sh` - Other suites (`fullstack-test`, async_grpc, collation, …) still run ## Test plan - [ ] Merge and `/test pull-integration-test` on pingcap/tiflash#10993 - [ ] Logs show WARN skip fail-point-tests - [ ] `tidb-ci` proceeds to fullstack-test without enableTestAPI error
Summary
Follow-up to #4849. Rewriting untagged
tiflash-ci-basetocd/builders/tiflash:v20231106unblocked registry pulls, but fullstack tests exec into that container and needmysqlclient:Old
hub.pingcap.net/tiflash/tiflash-ci-baseincluded mysql (seejenkins/Dockerfile/release/linux-amd64/tiflash_ci_base). Builder images do not.Change
ghcr.io/pingcap-qe/bases/tiflash-base:v1.9.1tiflash-ci-base-local:with-mysqlwithyum install -y mysql(same recipe as historical Dockerfile)tiflash-ci-base/ticsKeeps failpoint fallback from #4849.
Test plan
/test pull-integration-teston Release 6.5.6 hotfix 20260720 pingcap/tiflash#10993tiflash-ci-base-local:with-mysqlfullstack-test/sample.testno longer fails onmysql: command not found