Let a cleanup run clear the whole backlog - #279
Merged
Conversation
A live run took one batch of 10,000 and stopped, so the nightly job could never clear the ~1.97M backlog: 197 nights at one batch each. It now keeps taking batches until one deletes nothing, and the batch is just the candidate query's page size. drain/1 was the same loop reachable only by hand, so it goes. What bounds a run is now the job's timeout rather than the batch. Jobs can declare one and Bob.Job.DockerCleanup takes 23 hours, long enough to keep working until the next night's run, which the queue dedups away while this one is still running. Bob.Runner arms its kill timer from it, and the maintenance sweep pre-filters on the shortest timeout and then checks each job against its own, so a cleanup is not requeued out from under itself at three hours. Bob.Job.timeout/1 calls Code.ensure_loaded? before function_exported?, which answers false for a module that has not been loaded and would have quietly given every job the default.
Docker Hub rate limits deletes per source IP, at 600 per 62s window, and the limiter paces us under it so we never see a 429. Two nodes therefore have two budgets, but pointing both at the whole candidate set just makes them race down the same unordered list, deleting each other's tags and getting 404s. Measured: two nodes on the same set moved 426 tags/min against 398 for one. :repos scopes a run, so one node can take hexpm/elixir-amd64 and the other hexpm/elixir-arm64. The repos are within 1% of each other in size, so the split is even and the backlog drains in about half the time. The list is intersected with the known per-arch repos rather than trusted, so no caller can point the deleter at a manifest repo.
Several said what the code is not, or described the design that was replaced, which only reads as an explanation to someone who saw the previous version.
ericmj
added a commit
that referenced
this pull request
Jul 30, 2026
* Clean up the erlang per-arch repos too The erlang per-arch repos were left out because DockerChecker.expected_elixir_tags/0 reads those rows to decide which Elixir images to build. It ranks them by version rather than by date, so the newest tag of an old OTP line stays in use however old it gets and a plain age cutoff would delete it. Keeping the erlang tags whose os_version the build matrix currently targets is enough to protect that, and costs 420 of the 71,061 stale erlang tags. The os_versions are re-read per batch so a base image released mid-run pulls the tags built against it back under protection, and the per-chunk re-check applies the same rule. Docker Hub meters the API per account, not per source IP. A pod deleting at 27 requests/minute from one node was pushed to 56 429s in 92 requests by a second node bursting from a different IP, with x-ratelimit-remaining reaching -534 on both. Scoping a repo per node cannot buy throughput. * Drop the :repos option It existed to point one node at each per-arch repo, which the account budget makes pointless. Nothing passes it, so the per-arch repo list stops being a parameter and the filter that kept a caller from aiming the deleter at a manifest repo goes with it. The dry run no longer claims a scheduled run stops at one batch; it has cleared the whole backlog since #279. * Fail closed when the build matrix cannot be read `x <> ALL('{}')` is true, so an empty os_version list protected no erlang tag rather than every one of them. `builds()` is empty on a database whose base_image_tags have never been populated, and it drops a line whose regex matches nothing, so the input that means "the matrix could not be read" was the input that deleted the most. It now holds the erlang repos back entirely. A tag whose metadata carries no os_version is spelled out as kept rather than left to `NULL <> ALL(...)`. The os_versions are pinned for the run like the cutoff. Re-reading them per batch removed protection as readily as it added it: a base image released mid-run rotates the os_version it replaces out of the matrix, and a run lasting days would then delete the erlang tags on it before the checker had built their replacements. An elixir request now reserves the erlang image it builds FROM. Expiring a request only sets its state, leaving any build already enqueued, so a job could outlive the protection its base image had and fail on a missing FROM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A live cleanup run took one batch of 10,000 and stopped, so the nightly job could never clear the ~1.97M backlog: 197 nights at one batch each. It now keeps taking batches until one deletes nothing, and the batch is just the candidate query's page size.
drain/1was that same loop reachable only by hand, so it goes.What bounds a run is now the job's timeout rather than the batch. Jobs can declare one, and
Bob.Job.DockerCleanuptakes 23 hours, long enough to keep working until the next night's run, which the queue dedups away while this one is still running.Bob.Runnerarms its kill timer from it, and the maintenance sweep pre-filters on the shortest timeout before checking each job against its own, so a cleanup is not requeued out from under itself at three hours and left running twice.Bob.Job.timeout/1callsCode.ensure_loaded?beforefunction_exported?, which answers false for a module that has not been loaded yet and would have quietly handed every job the default.:reposscopes a run to some of the per-arch repos. Docker Hub rate limits deletes per source IP, returning the key it used inx-ratelimit-ip, at 600 per 62 second window. The limiter paces us under that, which is why a 429 never appears in the logs. Each node has its own external IP and so its own budget, but pointing both at the whole candidate set only makes them race down the same unordered list deleting each other's tags: two nodes measured 426 tags/min against 398 for one. Scoping one node tohexpm/elixir-amd64and the other tohexpm/elixir-arm64splits the work evenly, the two being within 1% of each other in size, and roughly halves the drain.The repo list is intersected with the known per-arch repos rather than trusted, so no caller can point the deleter at a manifest repo and start removing the images people pull.
Measured on prod during the backlog drain: one node sustains about 348 tags/min, so the remaining work is roughly four days unpartitioned and two days split across both.