perf(ci): batch the pulp upload task waits#6304
Open
kduret wants to merge 12 commits into
Open
Conversation
Waiting for each upload task before sending the next upload pays the pulp task-queue latency once per package; under concurrent deliveries the queue grows and a single slow task aborts the delivery (~10 min wait_task cap) after minutes of sequential waits. Collect the task hrefs during the upload loop and await them as a batch before publishing (new wait_tasks helper); the batch guard aborts only when no task completes for ~10 min, so a draining queue is not mistaken for a hang. Refs MON-200796
kduret
requested review from
Tpo76 and
mushroomempires
and removed request for
a team
July 13, 2026 11:36
The GitHub Actions OIDC token expires ~5 minutes after issuance, so any delivery longer than that started failing mid-loop with HTTP 401 (full plugins delivery: ~5400 uploads across 8 jobs died at exactly +5 min), and task polls with a stale token read as 'did not complete in time'. Refresh the token before it goes stale from the api helpers (upload attempts and task-wait sweeps), propagate it to the following steps and rewrite the pulp-cli config so the pulp commands keep working; setup-pulp-cli exports the issuance time and audience the refresh needs. Refs MON-200796
pulp_upload runs in a command substitution, so its internal token refresh only lives in the subshell: the parent shell kept the stale token, killing the rpm stable-guardrail curl with a 401 at +5 min and forcing every post-expiry deb upload to re-mint a token and rewrite the pulp-cli config (seconds per package). Refresh once per loop iteration from the parent shell instead — the guardrail and the inherited subshell token stay fresh, and the per-package refresh overhead disappears. Refs MON-200796
Large deliveries can wait minutes to hours on the server task queue with no output. Emit a progress heartbeat from wait_tasks, at most every 30 seconds (time-based: a single sweep over a large batch can take minutes on its own), counting the completed tasks over the batch total. Refs MON-200796
… E2BIG) Two failures surfaced by a 675-package delivery once the task queue drains: - refresh_pulp_token echoed the ::add-mask:: workflow command on stdout; when the refresh triggers inside pulp_upload's command substitution the command line is captured into the task href, which then never resolves (the batch wait stalls on a single corrupted href). Emit it on stderr. - manifest_write passed the packages array as an --argjson argument, which exceeds the kernel argv size limit with hundreds of packages (jq: Argument list too long, exit 126, right after a successful publish). Slurp the entries from stdin instead. Refs MON-200796
The published-metadata resolvers piped the index body into an awk that exits on the first match, so the printf writer took a SIGPIPE and bash printed one "printf: write error: Broken pipe" diagnostic per verified package. Read the index from a temporary file instead - same early exit, no pipe writer. Refs: MON-200796
A single limit=1000 page silently truncates once the repository holds more module packages than the page size (the shared apt-plugins repository is past 5000), flagging delivered packages as missing. Refs: MON-200796
List the module packages newest first and stop as soon as every expected package has been seen: the shared plugins repository holds 10k+ module packages, so an unbounded offset walk is slow, eventually fails server-side, and the expected packages sit in the tail with the default ordering. The unexpected-extras report becomes count-based for the same reason. Refs: MON-200796
grep -q exits on the first match, the printf writer takes a SIGPIPE, and pipefail turns the pipeline status into 141: every package that IS present gets flagged missing once the listing outgrows the pipe buffer (the shared plugins repositories are past 12k module packages). Write the listing to a file and let grep read the file - no writer left to kill. Refs: MON-200796
…ng wait) pulp-cli reads its token once at startup, so its built-in wait dies with "Authentication failed for tasks_read" as soon as the publication of a large repository outlives the OIDC token validity (~5 minutes). Start the publication in the background and poll it with wait_task, which re-authenticates on every attempt. Refs: MON-200796
The retry window only covers publication propagation: once a round reads the published metadata and resolves nothing new while some packages already resolved, the remaining ones are not part of the publication at all (e.g. evicted by the retention policy) - report right away instead of burning the whole 5-minute window, and log the per-round progress. Refs: MON-200796
One flaky HEAD out of hundreds (content-app/S3 hiccup) must not fail the whole verification. Refs: MON-200796
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Speed up and harden the Pulp delivery: the upload loop no longer waits for each pulp task before sending the next upload.
On a develop centreon-plugins run, every deliver job (8 in parallel, ~35 packages each) timed out: each package upload waited for its pulp task to complete (~5s median, up to 55s as the server task queue grew), and once the queue saturated a single task exceeded the ~10 min
wait_taskcap, aborting the delivery before publication. Pulp serializes the tasks of a repository server-side, so this per-package wait pays the queue latency N times and brings no ordering benefit.Changes
deliver-rpm.sh/deliver-deb.sh: collect the upload task hrefs during the loop and await them as a batch right before publishing. The uploads (HTTP POSTs) now pipeline with the server-side task processing.api.sh: newwait_taskshelper — polls the pending tasks in sweeps and aborts only when no task completes for ~10 min, so a long-but-draining queue under concurrent deliveries is not mistaken for a hang (the per-taskwait_taskstays for the promote scripts). Covered by a local mocked-curl test: progressive completion, failed-task abort, empty batch, stall guard.Applied identically to centreon, centreon-collect, centreon-modules and centreon-plugins (shared pulp files kept byte-identical).
Refs: MON-200796