Build a manifest only when it adds an arch or a push - #283
Merged
Conversation
ericmj
force-pushed
the
manifest-arch-floor
branch
from
July 31, 2026 01:21
cceeef7 to
3c6fd1c
Compare
The per-arch tags a manifest was built from are deleted once they are 30 days old, and the manifest repos are never pruned, so from then on any rebuild has fewer sources than the manifest already spans. Nothing triggers one while the manifest row still lists every arch, since manifest_mismatches/3 holds as the per-arch rows shrink under it. A reconcile that reads a short arch set off Docker Hub's tag API breaks that, and Docker Hub does: it reports 1.20.2-erlang-29.0.4-debian-trixie-20260713-slim as arm64 only while the registry has both platforms. The rebuild that follows would then drop whichever arch cleanup had already taken. get_archs/3 tells an unreadable response from a 404, but a per-arch tag cleanup deleted is a genuine 404, so that guard does not cover this. Anything that would lose an arch is now left alone. Membership alone is not enough to decide the rest. The same tag can be re-pushed under a new digest — BuildDocker* pushes before it reports the tag, so a job killed in between leaves Docker Hub ahead of the database and the retry pushes again — and the manifest has to follow it. So it is also built when a source is newer than it, and left alone only when it already spans these arches and postdates all of them. Those runs published the same manifest again and moved its last_updated, which is what the tag list shows as the build date, so a tag untouched for months read as freshly built. A tag with no manifest yet still takes whatever is there, which is how it gets one while its second arch is still building.
ericmj
force-pushed
the
manifest-arch-floor
branch
from
July 31, 2026 02:56
3c6fd1c to
6412974
Compare
ericmj
marked this pull request as ready for review
July 31, 2026 02:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The per-arch tags a manifest was built from are deleted once they are 30 days old, and the manifest repos are never pruned, so from that point on any rebuild has fewer sources than the manifest already spans.
Normally nothing triggers a rebuild:
manifest_mismatches/3compares the manifest row against the per-arch rows, andm.archs @> p.archskeeps holding as the per-arch rows shrink under it. What breaks that is a reconcile recording a short arch set, and Docker Hub's tag API produces those — it currently reports1.20.2-erlang-29.0.4-debian-trixie-20260713-slimas arm64-only while the registry has both platforms. The rebuild that follows drops whichever arch cleanup had already taken, and the manifest users pull becomes single-arch.#282 taught
get_archs/3to tell an unreadable response from a 404, but a per-arch tag that cleanup deleted is a genuine 404, so that guard does not cover this case. Anything that would lose an arch is now left alone, including sets that only overlap, so a rebuild can never swap one arch for another.Membership alone is not enough to decide the rest, because the same tag can be re-pushed under a new digest.
BuildDockerElixir.run/5andBuildDockerErlangpush before reporting the tag, so a job killed in between leaves Docker Hub ahead of the database and the retry pushes a fresh digest under the same name. A rule that skipped whenever the arch set was unchanged would leave the manifest pointing at the old digests. So a manifest is also built when a source is newer than it, and left alone only when it already spans these arches and postdates all of them.That last part also stops the pointless rebuild: those runs published the same manifest again and moved its
last_updated, which is what the tag list renders as the build date, so a tag untouched for months read as freshly built.A tag with no manifest yet still takes whatever is there, which is how it gets one while its second arch is still building.
Costs one extra Docker Hub read per run, on a job that only fires on a mismatch or after a build.
Exposure grows with the backlog drain: the condition is "at least one per-arch tag already deleted", which becomes true for essentially every tag as it completes. There were 0 manifest mismatches and no republishes when this was written, so nothing is degrading right now.
hexpm/elixir:1.16-erlang-26.2.5.15-alpine-3.21.5, degraded this way on 2026-07-30, still has both per-arch tags, so it grows the set from[amd64]to[amd64, arm64]and republishes correctly.