Skip to content

Build a manifest only when it adds an arch or a push - #283

Merged
ericmj merged 1 commit into
mainfrom
manifest-arch-floor
Jul 31, 2026
Merged

Build a manifest only when it adds an arch or a push#283
ericmj merged 1 commit into
mainfrom
manifest-arch-floor

Conversation

@ericmj

@ericmj ericmj commented Jul 31, 2026

Copy link
Copy Markdown
Member

The per-arch tags a manifest was built from are deleted once they are 30 days old, and the manifest repos are never pruned, so from that point on any rebuild has fewer sources than the manifest already spans.

Normally nothing triggers a rebuild: manifest_mismatches/3 compares the manifest row against the per-arch rows, and m.archs @> p.archs keeps holding as the per-arch rows shrink under it. What breaks that is a reconcile recording a short arch set, and Docker Hub's tag API produces those — it currently reports 1.20.2-erlang-29.0.4-debian-trixie-20260713-slim as arm64-only while the registry has both platforms. The rebuild that follows drops whichever arch cleanup had already taken, and the manifest users pull becomes single-arch.

#282 taught get_archs/3 to tell an unreadable response from a 404, but a per-arch tag that cleanup deleted is a genuine 404, so that guard does not cover this case. Anything that would lose an arch is now left alone, including sets that only overlap, so a rebuild can never swap one arch for another.

Membership alone is not enough to decide the rest, because the same tag can be re-pushed under a new digest. BuildDockerElixir.run/5 and BuildDockerErlang push before reporting the tag, so a job killed in between leaves Docker Hub ahead of the database and the retry pushes a fresh digest under the same name. A rule that skipped whenever the arch set was unchanged would leave the manifest pointing at the old digests. So a manifest is also built when a source is newer than it, and left alone only when it already spans these arches and postdates all of them.

That last part also stops the pointless rebuild: those runs published the same manifest again and moved its last_updated, which is what the tag list renders as the build date, so a tag untouched for months read as freshly built.

A tag with no manifest yet still takes whatever is there, which is how it gets one while its second arch is still building.

Costs one extra Docker Hub read per run, on a job that only fires on a mismatch or after a build.

Exposure grows with the backlog drain: the condition is "at least one per-arch tag already deleted", which becomes true for essentially every tag as it completes. There were 0 manifest mismatches and no republishes when this was written, so nothing is degrading right now.

hexpm/elixir:1.16-erlang-26.2.5.15-alpine-3.21.5, degraded this way on 2026-07-30, still has both per-arch tags, so it grows the set from [amd64] to [amd64, arm64] and republishes correctly.

@ericmj
ericmj force-pushed the manifest-arch-floor branch from cceeef7 to 3c6fd1c Compare July 31, 2026 01:21
@ericmj ericmj changed the title Build a manifest from every arch or leave it alone Build a manifest only when it adds an arch Jul 31, 2026
The per-arch tags a manifest was built from are deleted once they are 30
days old, and the manifest repos are never pruned, so from then on any
rebuild has fewer sources than the manifest already spans. Nothing
triggers one while the manifest row still lists every arch, since
manifest_mismatches/3 holds as the per-arch rows shrink under it. A
reconcile that reads a short arch set off Docker Hub's tag API breaks
that, and Docker Hub does: it reports
1.20.2-erlang-29.0.4-debian-trixie-20260713-slim as arm64 only while the
registry has both platforms. The rebuild that follows would then drop
whichever arch cleanup had already taken.

get_archs/3 tells an unreadable response from a 404, but a per-arch tag
cleanup deleted is a genuine 404, so that guard does not cover this.
Anything that would lose an arch is now left alone.

Membership alone is not enough to decide the rest. The same tag can be
re-pushed under a new digest — BuildDocker* pushes before it reports the
tag, so a job killed in between leaves Docker Hub ahead of the database
and the retry pushes again — and the manifest has to follow it. So it is
also built when a source is newer than it, and left alone only when it
already spans these arches and postdates all of them. Those runs
published the same manifest again and moved its last_updated, which is
what the tag list shows as the build date, so a tag untouched for months
read as freshly built.

A tag with no manifest yet still takes whatever is there, which is how
it gets one while its second arch is still building.
@ericmj
ericmj force-pushed the manifest-arch-floor branch from 3c6fd1c to 6412974 Compare July 31, 2026 02:56
@ericmj ericmj changed the title Build a manifest only when it adds an arch Build a manifest only when it adds an arch or a push Jul 31, 2026
@ericmj
ericmj marked this pull request as ready for review July 31, 2026 02:58
@ericmj
ericmj merged commit 17c581f into main Jul 31, 2026
4 checks passed
@ericmj
ericmj deleted the manifest-arch-floor branch July 31, 2026 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant