feat(ci): add dynamic region selection and minor CI stability fixes - #538
feat(ci): add dynamic region selection and minor CI stability fixes#538alizaidis wants to merge 3 commits into
Conversation
18ee383 to
daebf8e
Compare
daebf8e to
f46d84d
Compare
| "${ACP_REPO_DIR}/platforms/gke/base/use-cases/inference-ref-arch/kubernetes-manifests/online-inference-gpu/vllm-native-cache-offloading/single-tier/configure_vllm.sh" | ||
|
|
||
|
|
||
| CRD_DIR="${ACP_REPO_DIR}/test/ci-cd/crds" |
There was a problem hiding this comment.
sorry, I am not clear why we want to add the crds locally here?
|
|
||
| # Check if region supports required GPU & TPU v6e machine types (ct6e-standard-4t OR a2-highgpu-1g / g2-standard-4) | ||
| if [ -n "${PROJECT_ID_TO_CHECK}" ]; then | ||
| mt_check=$(gcloud compute machine-types list --filter="zone ~ ${region} AND name=(ct6e-standard-4t OR a2-highgpu-1g OR g2-standard-4)" --project="${PROJECT_ID_TO_CHECK}" --format="value(name)" 2>/dev/null | head -n 1 || true) |
There was a problem hiding this comment.
We need to check for all the machine types being created as part of custom compute classes otherwise this code section https://github.com/GoogleCloudPlatform/accelerated-platforms/blob/main/platforms/gke/base/core/custom_compute_class/main.tf#L98. will fail the CI pipeline with these errors:
│ auto provisioned zones, last zone check error: Machine type a2-ultragpu-1g
--
| │ not available in us-east1-d zone Crd is not healthy.
| │ gpu-h100-80gb-high-x1: Machine type a3-highgpu-1g is not available in any
| │ of auto provisioned zones, last zone check error: Machine type
| │ a3-highgpu-1g not available in us-east1-d zone Crd is not healthy.
| │ gpu-h100-80gb-high-x2: Machine type a3-highgpu-2g is not available in any
| │ of auto provisioned zones, last zone check error: Machine type
| │ a3-highgpu-2g not available in us-east1-d zone Crd is not healthy.
| │ gpu-h100-80gb-high-x4: Machine type a3-highgpu-4g is not available in any
| │ of auto provisioned zones, last zone check error: Machine type
| │ a3-highgpu-4g not available in us-east1-d zone Crd is not healthy.
| │ gpu-h100-80gb-high-x8: Machine type a3-highgpu-8g is not available in any
| │ of auto provisioned zones, last zone check error: Machine type
| │ a3-highgpu-8g not available in us-east1-d zone Crd is not healthy.
| │ gpu-h100-80gb-mega-x8: Machine type a3-megagpu-8g is not available in any
| │ of auto provisioned zones, last zone check error: Machine type
| │ a3-megagpu-8g not available in us-east1-d zone Crd is not healthy.
| │ gpu-h200-141gb-ultra-x8: Machine type a3-ultragpu-8g is not available in
| │ any of auto provisioned zones, last zone check error: Machine type
| │ a3-ultragpu-8g not available in us-east1-d zone Crd is not healthy.
| │ gpu-h200-141gb-ultra-x8-co: Machine type a3-ultragpu-8g is not available in
| │ any of auto provisioned zones, last zone check error: Machine type
An example of failed pipeline : https://pantheon.corp.google.com/cloud-build/builds;region=us-central1/92944804-05b4-4192-b2e6-7813c5471e92;step=2?project=accelerated-platforms
Ideally, CANDIDATE_REGIONS in this script should only have those regions that we know supports all the machine types covered in accelerated platforms. But unfortunately, no region other than us-central1 supports all the machine types that we use here.
If my understanding is correct no CI test other than ComfyUI deploys any AI workload that would need machine availability, they just deploy the platform including CCC. ComfyUI deploys the workload but on L4 only. We will need to bypass the CCC check if we want to run the CI in a region other than us-central1 as none of the other regions have full coverage of the machines
| EOT | ||
|
|
||
| # Run preflight region check to select optimal region for CI run | ||
| DYNAMIC_CI_REGION="$("${ACP_REPO_DIR}/test/ci-cd/scripts/cloudbuild/select_best_ci_region.sh" "${PROJECT_ID:-}" | tail -n 1)" |
There was a problem hiding this comment.
$PROJECT_ID will be accelerated-platforms in the cloudbuild env so the script will check regions and cpu limits and usage in accelerated-platforms and not the project where the platform will be created(platform_default_project_id).
63e6f66 to
9f45a9a
Compare
9f45a9a to
f25c281
Compare
2 participants
PR: Dynamic CI Region Selection and minor CI stability fixes
Overview
This PR addresses the CI failures we've been experiencing due to
ZONE_RESOURCE_POOL_EXHAUSTEDerrors inus-central1. By dynamically shuffling our candidate deployment regions and validating their quota capacity for our baseline requirements, we statistically spread our CI load and prevent immediate cluster / system node pool creation failures.Additionally, this PR addresses critical review feedback by fixing the sequencing of project bootstrapping, bypassing region-specific Custom Compute Class health checks, and bulletproofing our token injection scripts.
Key Changes
1. Dynamic Region Shuffling (
select_best_ci_region.sh)shuf -ecommand. We now randomize the array of candidate regions on every single run, which inherently acts as a retry mechanism across regions when rerunning failed jobs.n4-standard-4Offerings Check: Added a strict precondition check. Candidate regions are now actively verified to ensure they offer then4-standard-4instance type required by our system node pools before being considered.2. Ephemeral Project Sequencing (
configure_build_environment.sh&create_project.sh)create_project.shto occur before region selection. This allows the script to pass$NEW_PROJECT_IDinto the selection logic, guaranteeing that quotas are evaluated against the actual ephemeral CI project rather than a generic parent.create_project.shwhere the Compute Engine API was being enabled before the project was attached to a billing account, which was silently failing the region selection script.3. Custom Compute Class Health Checks Bypass
export TF_VAR_cluster_check_custom_compute_classes_healthy="false"into the generatedbuild.envpayload.a3-megagpu-8g) that only exist in select zones, GKE's auto-provisioner flags the CRDs as unhealthy in alternative regions (likeeurope-west4). Bypassing this health check in CI prevents the pipeline from failing on non-central regions.4. Robust Hugging Face Token Population
populate_huggingface_token_secrets.shacross all use-cases (inference, training, and RL).secretmanager.googleapis.comAPI is enabled and explicitly checks if the secret container exists (creating it if necessary) before attempting to add the token payload version. This prevents crashes during secret population on fresh CI projects.5. Code Maintenance & Documentation
validate_kustomize.shregarding theLOCAL_CRDS_FLAGto clarify thatkubectl validateinherently requires local schemas for CRDs like Prometheus ServiceMonitor because there is no live cluster during the manifest generation phase to query for the CRD.