Skip to content

Mirror the images deployed by feature charts - #1319

Merged
ArnobKumarSaha merged 4 commits into
masterfrom
arnob-feat-imgs
Aug 16, 2026
Merged

Mirror the images deployed by feature charts#1319
ArnobKumarSaha merged 4 commits into
masterfrom
arnob-feat-imgs

Conversation

@ArnobKumarSaha

@ArnobKumarSaha ArnobKumarSaha commented Aug 16, 2026

Copy link
Copy Markdown
Member

Problem

catalog/ listed the feature charts but never the container images those charts deploy. Every feature-chart image that no other catalog publishes was missing from the mirror scripts:

reloader, prometheus-adapter, kyverno, longhorn, opencost, gatekeeper, sealed-secrets, external-secrets, topolvm, crossplane, the CSI drivers, the credential managers.

This surfaced from an air-gapped install: a live ACE cluster was running ghcr.io/stakater/reloader:v1.4.14 and registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.11.2, neither of which appears in any catalog.

These images can't be recovered from this repo. They exist only inside a Feature's spec.values, as maps rather than the plain image strings image-packer collects:

# opscenter-features/.../prometheus-adapter.yaml
image:
  repository: registry.k8s.io/prometheus-adapter/prometheus-adapter   # <- no tag

and the tag isn't in this tree at all — it comes from the feature chart's own appVersion. So no sample-values change here could fix it (unlike #1318, which fixed a genuinely different problem: subcharts that never rendered).

Fix

kmodules/image-packer#60 renders each feature chart at its pinned version, using the values its Feature carries, and writes catalog/feature-chart-images.yaml. This PR feeds that file to generate-scripts.

  • catalog/feature-chart-images.yaml79 images from 49 feature charts (new, generated)
  • hack/scripts/update-catalog.sh — the exclusion list below, plus one added --src
  • catalog/{copy,export,import}-images.sh, catalog/import-into-k3s.sh — regenerated

Which charts are excluded, and why

65 of the 114 feature charts belong to a catalog that already publishes their images. Rendering them here restated 398 of 545 entries and made this repo's mirror scripts carry a second copy of every database image — work the component installers already do.

owner charts published via
this repo 12 catalog/imagelist.yaml
component installers 53 each installer's own catalog/imagelist.yaml, collected per org by appscode-cloud/artifacts pkg/collect/orgs.go

The list was derived, not guessed: each component installer's charts/ directory intersected against catalog/feature-charts.yaml. image-packer reports any exclusion matching no chart, so it can't rot silently — the run reports 65 excluded, 0 unmatched.

stash, voyager and virtual-secrets charts are excluded because appscode-cloud/artifacts#4 adds those three as component installers. stash alone accounted for 53 image entries here.

Externally maintained charts are deliberately NOT excluded. cert-manager, flux2, keda, keda-add-ons-http, kube-prometheus-stack and snapshot-controller were rendered by artifacts from hand-curated hack/ci/*-ci-values.yaml. Those values had drifted from what ACE deploys — they built the flux2 kustomize and notification controllers that the ACE Feature disables — so rendering from the Feature here is the more accurate source, and artifacts retires its step.

Verification

  • The generated scripts change by addition only: +79 / -0 in each of the four, zero removed lines, no duplicate $CMD cp (364 total = 285 existing + 79 new).
  • catalog/feature-charts.yaml is byte-identical to master — the existing consumer is unaffected.
  • The new file is a superset of the six retired images/<chart>.yaml files, minus only the two flux2 controllers ACE disables.
  • Regenerated with generate-scripts followed by make add-license fmt.

Notes for reviewers

  • One image is deliberately dropped: us-docker.pkg.dev/secretmanager-csi/secrets-store-csi-driver-provider-gcp/plugin@sha256:a493a78… is pinned by digest with no tag, and generate-scripts derives both the tarball name and the destination reference from the tag. Dropped with a warning naming it rather than failing the run. It is the only digest-only reference in the entire catalog corpus.
  • gateway-api (v2025.3.14) and voyager-gateway (v2026.1.15) are excluded as voyagermesh-owned, but that repo is collected at the voyager anchor v2026.3.23. Same latent version skew the other components already have.
  • Ordering: needs an image-packer release carrying Collect images referenced by feature charts and container flags kmodules/image-packer#60--exclude-chart does not exist in the current binary. Then Collect stash, voyager and virtual-secrets; retire the externals step artifacts#4.

Unrelated bug this surfaced

charts/opscenter-features/templates/featuresets/opscenter-secret-management/reloader.yaml (and the identical block in charts/ace-installer/) sets the image at reloader.deployment.image.name, but appscode-charts/reloader:2.2.9 reads it from top-level image.repository — that path does not exist in the chart's values. The override is silently ignored, which is why clusters pull ghcr.io/stakater/reloader rather than ghcr.io/appscode/reloader. Not fixed here; the catalog now lists what is actually pulled.

catalog/ listed the feature charts themselves but never the container
images they deploy, so no third-party feature-chart image was ever
mirrored: opencost, kyverno, longhorn, topolvm, external-secrets,
gatekeeper, sealed-secrets, crossplane, nfsplugin, metrics-server,
reloader and prometheus-adapter were all absent.

Those images cannot be recovered from this tree. They appear only inside
a Feature's spec.values, as maps rather than the plain image strings
image-packer collects, and without a tag -- the tag comes from the
feature chart's own appVersion. kmodules/image-packer#60 renders each
feature chart at its pinned version to resolve them, and writes
catalog/feature-chart-images.yaml.

Feed that file to generate-scripts, adding 545 images to the mirror
scripts. The generated scripts change by addition only; no existing
mirror line is touched.

Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
The previous commit rendered all 114 feature charts, but 59 of them
belong to a catalog that already publishes their images: 12 to this
repo's own catalog/imagelist.yaml, 41 to one of the component installers
collected per org by appscode-cloud/artifacts (pkg/collect/orgs.go), and
6 to the externally maintained charts artifacts renders itself from
curated CI values (pkg/collect/externals.go).

Rendering them here restated 398 of 545 entries and made this repo's
mirror scripts carry a second copy of every database image, work the
component installers already do.

Exclude those 59 by name. The list is grouped by owning catalog, and
image-packer reports any entry that matches no chart, so it cannot rot
silently. 117 images remain -- reloader, prometheus-adapter, kyverno,
longhorn, opencost, gatekeeper, topolvm, stash, voyager and the rest
that no other catalog publishes.

Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
appscode-cloud/artifacts now collects stashed, voyagermesh and
virtual-secrets as component installers, so their charts join the
exclusion list -- stash alone was 53 of the entries here.

The six charts artifacts used to render from hand-curated hack/ci values
move the other way, out of the exclusion list and into this file. They
are feature charts like any other, and the values their Feature carries
are what ACE actually deploys: the curated values had drifted, building
the flux2 kustomize and notification controllers that the Feature
disables.

79 images, all of them belonging to no installer repo.

Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
@ArnobKumarSaha
ArnobKumarSaha merged commit ad4a7bb into master Aug 16, 2026
3 of 4 checks passed
@ArnobKumarSaha
ArnobKumarSaha deleted the arnob-feat-imgs branch August 16, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant