Container base audit: what moving off Alpine does and does not buy

[HuggeK]

Audit of the core container's base image, prompted by the review on #714 arguing
that the official image cannot resolve mDNS .local names because "musl has no
NSS plugin mechanism, so the usual nss-mdns fix is not available the way it is
on glibc."

The conclusion in that review is correct — .local genuinely does not resolve.
The diagnosis is not, and it matters, because it points at the wrong fix.

1. The base image was never the variable

Go routes a .local lookup to libc only when cgo is available.
net/conf.go:395 opens if canUseCgo {, and the mDNS case sits inside it at
408-409:

case hostname != "" && stringslite.HasPrefix(src.source, "mdns"):
    if stringsHasSuffixFold(hostname, ".local") {
        return hostLookupCgo, dnsConf
    }

mustUseGoResolver begins if !cgoAvailable { return true }. Every FTW build
sets CGO_ENABLED=0, so canUseCgo is false and that branch is dead code. An
mdns4_minimal entry in nsswitch.conf falls through to hostLookupFilesDNS —
/etc/hosts, then a unicast DNS query to whatever resolv.conf points at.

This is true on every base image and every libc. Installing nss-mdns on a
Debian base would not have changed anything.

What NSS-based mDNS would actually require

All five of:

1. a glibc base;
2. CGO_ENABLED=1 — forfeits the static binary;
3. dynamic linking — glibc NSS resolves modules with dlopen, which a
   statically linked glibc binary cannot do. This also breaks the
   --platform=$BUILDPLATFORM cross-compile in Dockerfile and make verify-all;
4. libnss-mdns installed;
5. a reachable /run/avahi-daemon/socket — not present in the container, absent
   on a generic Docker host, and impossible under Docker Desktop.

Cost: CGO plus dynamic linking plus a new host dependency. Benefit: zero on any
host without avahi.

Our own Pi image confirms the host side: it runs NetworkManager (dns=default,
router nameservers straight into resolv.conf) plus avahi-daemon, with
dhcpcd masked and no systemd-resolved. Avahi serves glibc host programs
through NSS; it does not answer DNS queries. With network_mode: host the
container inherits that resolv.conf, so zap.local becomes a unicast query to
a router that will not answer it.

This is a live path, not a #714 hypothetical — the shipped
config.example.yaml binds devices by .local name (zap.local,
sid-os.local), and docs/sourceful-zap.md documents reaching the device at
http://zap.local/api/system.

Fix: #728 adds go/internal/mdnsresolve, forward .local resolution in Go,
wired into all six dial sites. Works on any base, any libc, no container change.

2. Measured comparison

Base layers via docker manifest inspect (arm64, compressed); full images built
locally (amd64, uncompressed).

	alpine:3.22	debian:trixie-slim	distroless/static	scratch
Base layer, compressed	4.12 MB	30.14 MB	0.70 MB	0
Full core image, uncompressed	52.5 MB	133 MB (measured, trixie)	~17 MB est.	~16 MB est.
Solves the mDNS motivation	n/a	no	no	no
arm64 runtime layer under QEMU	trivial (apk add ×2)	apt/dpkg emulated	none	none
Files to change	0	~6	~8 incl. the update path	~8 + own certs/zoneinfo/passwd
RUN in final stage	fine	fine	impossible (no shell)	impossible
On-site docker exec debugging	busybox shell	best	none	none
CVE surface	small	largest	smallest	smallest

2b. The alternative route, measured — #746

#746 implements the other option end to end: one alpine:3.22 base for core
and the updater, with .local resolved natively in Go and no avahi anywhere.
It is an alternative to #731 + #728, not a companion — one route or the other.

Building both in one session produced two findings that change section 2's
conclusions.

Alpine can consolidate only two of the three images

Measured on python:3.12-alpine:

pip install --only-binary=:all: cvxpy==1.9.2
  → Could not find a version that satisfies the requirement cvxpy==1.9.2
    (from versions: 0.4.10)

CVXPY publishes no musllinux wheels at all — 0.4.10 is the newest in
existence.

Package	musl wheel
numpy, scipy, highspy, scs	✅
cvxpy, clarabel, osqp, ecos	❌

Source builds would mean compiling a Rust solver (clarabel) and two C++ solvers
on every release, arm64 under emulation included. Attempted with a full
toolchain installed: it fails in sparsediffpy's build backend in 70 s, before
reaching any of them.

This is the same constraint that put core on Debian originally, and the comment
ca37bef deleted — "the CVXPY and HiGHS ARM64 wheels target glibc" — was
right about the direction even though highspy has since gained musl wheels.

The Debian route's real cost is 4.6 MB, not 83 MB

python:3.12-slim-trixie is built from debian:trixie-slim. Both report
first layer f2ec4de84f559f5c7be4, so under #731 the optimizer shares its base
with core and the updater; under #746 it shares nothing with them.

	#746 (alpine)	#731 (trixie)
Distinct bases in the deployment	2	1
libcs	2	1
Core image	52.5 MB	133 MB
Updater	130 MB	203 MB
Shared base	8.29 MB	78.6 MB

Counting unique bytes with O for the optimizer image:

• build: one alpine base for core and updater, with .local resolved in Go #746 → 52.5 + 130 − 8.29 + O = 174.2 + O
• build: run one base across the whole stack (debian:trixie-slim) #731 → 133 + 203 − 78.6 + (O − 78.6) = 257.4 + O − 78.6

Difference: +4.6 MB for #731 when the optimizer is deployed, +83.2 MB when
it is not — both independent of O, so this holds without building it.
docker-compose.yml ships ftw-optimizer as a default service, so the common
case is the 4.6 MB one. The Alpine size advantage largely disappears the moment
the optimizer is present, because trixie is then on the host regardless.

Two smaller corrections to section 2

• The updater shrinks on either route. It was 204 MB on docker:27-cli,
  which bundles far more than the CLI: 203 MB on trixie, 130 MB on alpine.
• docker and docker compose are static Go binaries, so copying them out
  of docker:27-cli is base-independent. Verified by running both from a
  debian and an alpine image; musl's loader reports Not a valid dynamic program for each, which is how it labels a static binary.

What each route gives up

• build: one alpine base for core and updater, with .local resolved in Go #746 — musl has no NSS plugin mechanism, so getent hosts zap.local and
  curl inside the container can never resolve .local, whatever is installed.
  FTW resolves correctly while the shell beside it cannot. No prebuilt glibc
  vendor binaries; debugging stays busybox.
• build: run one base across the whole stack (debian:trixie-slim) #731 + feat(net): resolve device .local names, asking avahi first #728 — one more base layer's worth of bytes, the largest CVE
  surface of the candidates, and it reverses ca37bef.

The two dimensions are independent, incidentally: #731's base could be paired
with #746's dependency-free resolver, giving up only the in-container getent
agreement.

3. Blockers that apply to any base change

1. scripts/test-container-boundaries.sh asserted ^FROM alpine: with no
   error message. It is a proxy for the real invariant — core must not drag in a
   Python interpreter — since Python originally reached core via the optimizer's
   base image. Worth stating directly instead.
2. wget is contractual, and absent from every candidate base. It is used by
   the HEALTHCHECK and by go/cmd/ftw-updater/main.go:1217, the readiness
   probe that decides whether an update commits. On failure the updater
   auto-rolls back.
3. gid 101 must be preserved. optimizer/ftw_optimizer/worker.py:101 chmods
   its socket 0o660 — group-only. Core reaches it purely because both
   containers share gid 101. Changing it degrades the planner to the Go DP
   fallback silently.

4. Why distroless/scratch is currently unreachable

ftw-updater can never update itself. componentSpec handles only core
and optimizer, and compose up -d is scoped to a single service. Old updaters
therefore persist in the field indefinitely — there is no release cadence that
drains them.

So a wget-less core would meet an old updater that docker execs wget, fail
its gate forever, and pin the host in an update → rollback loop. Both call sites
are stubbed in main_test.go, so CI would not catch it.

Making distroless reachable needs updater added as a third component first,
then a wait for convergence. Filing this as the standalone gap it is.

5. Latent issues found along the way

Independent of any base decision:

• time.Local silently degrades to UTC when zoneinfo is missing, mis-timing
  price and plan windows with no error. _ "time/tzdata" is not imported, and
  the tzdata tests t.Skipf rather than fail — so a base image without tzdata
  would keep CI green. Addressed in build: run one base across the whole stack (debian:trixie-slim) #731.
• Nothing watches the base images. No trivy/grype/scout/SBOM/provenance
  anywhere, and .github/dependabot.yml has no package-ecosystem: docker, so
  alpine:3.22, golang:1.26-alpine, docker:27-cli and
  python:3.12-slim-bookworm are unmonitored. The "Alpine has fewer CVEs"
  argument is currently unmeasured either way.
• Two comments describe a user that has never existed. scripts/install.sh
  and deploy/pi-gen/stage-ftw/01-ftw-setup/00-run.sh both attribute uid 100 to
  an alpine adduser -S. The Dockerfile creates no account on any base —
  USER 100:101 is bare numeric, which is why ENV HOME=/app/data is
  load-bearing. Verified: uid 100 and gid 101 have no passwd/group entry on
  debian:bookworm-slim either.
• No image-size assertion and no on-device digest pinning exist.
  selfupdate/registry.go checks tag existence only; rollback is local
  retagging. Beta and stable built from the same commit at different times can
  already contain different base rootfs.

6. Related

• feat(net): resolve device .local names, asking avahi first #728 — Go-side mDNS resolution. The actual fix for .local.
• build: run one base across the whole stack (debian:trixie-slim) #731 — moves core, updater and optimizer to debian:trixie-slim, one base across the stack.
• ci: notice when the container base falls behind Debian stable #733 — weekly check that files an issue when a newer Debian stable ships, so the pin moves deliberately.
• build: one alpine base for core and updater, with .local resolved in Go #746 — the alternative route: one alpine base for core + updater, .local resolved natively in Go. Merge either this or build: run one base across the whole stack (debian:trixie-slim) #731 + feat(net): resolve device .local names, asking avahi first #728, never both. Measurements in section 2b.