test(ci): batched rpm and deb delivery through a single modify task [do not merge]#6306
Draft
kduret wants to merge 51 commits into
Draft
test(ci): batched rpm and deb delivery through a single modify task [do not merge]#6306kduret wants to merge 51 commits into
kduret wants to merge 51 commits into
Conversation
TEMP test branch (do not merge). In plugins.yml: - disable the Artifactory deliver-packages job (if: false) - force the Pulp deliver to stability: unstable, drop the stability gates (a PR runs with canary stability) and decouple the twin from test-plugins - add a marker file in the os::linux::snmp plugin packaging so get-plugins packages something to deliver All changes tagged TEMP(test-pulp-unstable). (cherry picked from commit 9174df1)
TEMP(test-pulp-unstable): reproduce the volume of the failed develop run (~30 packages per distrib) so the batched task wait is validated under a realistic load.
TEMP(test-pulp-unstable): unit-tests and test-plugins bring nothing to the batched-delivery validation and would consume up to 25 runners for the 31 marked plugins; fatpacker/package tolerate the skipped dependency.
TEMP(test-pulp-unstable): replace the 31 packaging markers with a touch of the common packaging template (changes_common=true), so get-plugins lists all 675 plugins and the batched Pulp delivery is exercised at full nightly scale (~675 packages per distrib across the 8 parallel deliver jobs).
… into MON-200796-plugins-test-batch-wait
TEMP(test-pulp-unstable) experiment: upload the rpm packages as unassociated content (repository-less create tasks parallelize across the pulp workers instead of serializing on the repository lock), resolve the created content hrefs (sha256 fallback for job re-runs), then add the whole batch to the repository with a single modify task. Two repository-serialized tasks per delivery (modify + publication) instead of one per package. Requires the reconciled rpm/packages access policy (delivery-tooling#209).
… into MON-200796-plugins-test-batch-modify
TEMP(test-pulp-unstable): same scope as the sibling batch-wait test for a direct comparison (675 packages per distrib).
…e [skip ci] The previous restriction hit the first matching matrix, which is the disabled artifactory job's; anchor on the twin's job name this time. [skip ci]: no point running until delivery-tooling#209 is merged (the orphan upload is rejected by the default access policy).
This reverts commit cf52ee2.
Faster to decompress at install time than xz, supported by rpm >= 4.14 (el8 onwards).
One flaky HEAD out of hundreds (content-app/S3 hiccup) must not fail the whole verification.
The testing repositories keep every delivered version (and the shared deb repository holds every suite), so the module listing exceeds a single page; the fail-loud truncation guard becomes an actual pagination.
…single modify) The packages already exist as content units in the shared repository, so promoting is only a matter of stable suite associations: one legacy re-upload per architecture (creates the stable ReleaseComponent and ReleaseArchitecture), synchronous PackageReleaseComponent creates for the rest, one modify, one publication - instead of one repository-serialized upload task and one file download per package.
- the modify/add_content_units bodies exceed the argv limit past a few thousand hrefs: pass them with curl -d @file - the stable guards and index fetches now retry transient 5xx (404 stays a valid not-in-stable answer), and the upload retry budget grows to 5 attempts: the stack under full-matrix load returns occasional 502s
- the parallel workers now refresh the inherited OIDC token on entry: under server slowdowns the parent's every-40-spawns cadence can span the token validity, and the first guard call of a worker then dies on 401 - the promote sweeps only the LATEST version of each package: the testing repositories keep every delivered build now, and promoting the whole accumulated history to stable is never intended
Transient ingress 504s under full-matrix load; a replayed association that already applied answers 400 and falls back to the existing lookup.
… into MON-200796-plugins-test-batch-modify
A full-matrix delivery (8 distributions uploading, associating and polling at once) saturates the pulp database: postgres dropped connections and the api pods crash-looped. Four concurrent legs keep the throughput without tipping the stack over.
The per-package pulp-cli lookups in the rpm stable guard died silently on a transient 502 (killing the upload worker without a message), and a 502 on the existence check would even have bypassed the guard. One retried api call, fail-closed on unreachability - and two python cli startups per package saved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment: batched delivery through a single
modifytask, rpm and deb, on top of the full-scale test setup (675 plugins, Pulp unstable only, Artifactory and test jobs disabled). Validated at scale: delivery phase ~5 min (rpm) and ~8 min (deb) for 675 packages, against 43 and 76 min with per-package repository tasks.Method
rpm (deliver-rpm.sh, TEMP)
modifytask, then publish: 2 repository-serialized tasks per delivery instead of ~675.deb (deliver-deb.sh, TEMP) — a repository-less deb upload ignores
distribution/component, and the release_components api is not listable by the ci user, so:modifyadding packages + associations, then a structured publication.Hardening picked up along the way (ported to the 4 feature branches)
create_publication: publication started in--backgroundand awaited with re-authenticating polling (pulp-cli's built-in wait dies on the ~5 min OIDC token when publishing a large repository).One-off maintenance carried on this branch (TEMP)
apt-pluginsrepository (14 042 → 890 packages), develop deliveries preserved.Prerequisites (all merged)
delivery-tooling#209, #213 (rpm policy), #214 (deb policies), #216/#217 (bounded repository versions + daily orphan cleanup), #219 (retention set at creation).
Comparison target
The sibling test PR #6305 measured the same full-scale delivery with per-package repository tasks (batched waits). Refs: MON-200796