Collect images referenced by feature charts and container flags - #60
Merged
Conversation
list-feature-charts recorded only the chart OCI refs pinned by each Feature/FeatureSet, and nothing ever rendered those charts, so the container images they deploy reached no catalog. list --root-dir=charts cannot cover them either: in the installer tree those images appear only inside a Feature's spec.values as maps (image.repository, deployment.image.name), while collectImages records string values under an "image" key, and the tag is absent entirely -- it comes from the feature chart's own appVersion. Render each feature chart at its pinned version using the values its Feature carries, and write the images they reference to feature-chart-images.yaml. Charts that fail to render are reported by name rather than aborting the run, so one broken chart cannot silently empty the catalog. Against appscode-cloud/installer this collects 546 images from 114 charts, among them ghcr.io/stakater/reloader and registry.k8s.io/prometheus-adapter/prometheus-adapter, which no catalog listed before. feature-charts.yaml output is unchanged. Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
generate-scripts derives a tarball name and a destination reference from an image's tag, so it errors out on a reference carrying only a digest. Feature charts occasionally pin that way -- secrets-store-csi-driver- provider-gcp does -- and a single such ref would fail the whole update-catalog run. Drop those refs and name them in a warning, so the gap is visible rather than silent. Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
Most feature charts belong to an installer that already publishes its own catalog/imagelist.yaml, so rendering every chart makes feature-chart-images.yaml restate images another catalog owns -- for appscode-cloud/installer that was 398 of 545 entries, and it would mirror a second copy of every database image. Which charts those are is deployment policy, not something this tool can know, so take the names from the caller. An exclusion matching no chart is reported: a stale or misspelled entry would otherwise silently start collecting images again. feature-charts.yaml is unaffected -- the charts themselves still need mirroring. Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
An operator that launches other workloads takes their image as a flag -- --acme-http01-solver-image, --prometheus-config-reloader, --thanos-default-base-image -- and that reference appears nowhere else in the manifest, so it reached no catalog. appscode-cloud/artifacts carried a regex over rendered output to recover three such images; collecting them here retires that. name.ParseReference cannot be the test on its own: it defaults the registry to docker.io and the tag to latest, so --log-level=info parses as an image. Demand an explicit registry host and an explicit tag or digest, which an image passed this way always carries. Verified against appscode-cloud/installer: catalog/imagelist.yaml is byte-identical, so the scan adds no false positives, and the three images above now appear in feature-chart-images.yaml. Signed-off-by: Arnob Kumar Saha <arnob@appscode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two kinds of image reached no catalog.
Feature chart images.
list-feature-chartsrecords only the chart OCI refs pinned by eachFeature/FeatureSet. Nothing ever renders those charts.list --root-dir=chartscan't cover them either — in the installer tree the images appear only inside aFeature'sspec.values, as maps:while
collectImagesrecords only string values under a key namedimage. The tag isn't there at all — it comes from the feature chart's own appVersion.Flag-passed images. An operator that launches other workloads takes their image as a container flag, and that reference appears nowhere else in the manifest:
appscode-cloud/artifactscarried a regex over rendered output to recover the three above; collecting them here retires that.Fix
Render feature charts. Each at its pinned version, with the values its
Featurecarries — that's what makes the ACE-specific overrides resolve.pkg/lib/feature.go(new) —FeatureChart+FeatureChartImages(), 8 charts concurrentlypkg/lib/image.go— extractedwriteTempValues, added exportedCollectRenderedImagespkg/cmds/list_feature_charts.go—ListUIChartsreturns[]lib.FeatureChartcarryingspec.values, deduped on chart and valuesA chart that fails to render is reported by name rather than aborting, so one broken chart can't silently empty the catalog.
Collect flag-passed images.
name.ParseReferencecan't be the test on its own — it defaults the registry to docker.io and the tag to latest, so--log-level=infoparses as an image. Demand an explicit registry host and an explicit tag or digest, which an image passed this way always carries.--exclude-chart. Most feature charts belong to an installer that already publishes its owncatalog/imagelist.yaml, so rendering everything makes this file restate images another catalog owns — forappscode-cloud/installerthat was 398 of 545 entries. Which charts those are is deployment policy this tool can't know, so the names come from the caller (repeatable, comma-accepting). No ownership is hardcoded here. An exclusion matching no chart is reported, since a stale entry would otherwise silently start collecting images again.feature-charts.yamlis unaffected.Verification against appscode-cloud/installer
catalog/imagelist.yamlis byte-identical across all its charts with flag-scanning enabled — the scan adds no false positives.feature-charts.yamloutput byte-identical.images/<chart>.yamlfilesappscode-cloud/artifactsused to hand-curate, minus only the two flux2 controllers the ACEFeaturedisables.Notes for reviewers
--with-images(defaulttrue) turns off the network-heavy render.golang.org/x/sync/errgroup— vendored only as// indirect, so importing it directly would editgo.mod/vendor. Async.WaitGroup+ semaphore does the same job.helm template: 5 editor-featureset chart names exceed helm's 53-char release-name limit.generate-scriptsderives both the tarball name and destination reference from the tag, so one such ref would fail the whole run. Digest support there is worth a follow-up.collectImagesaboutimage.repository/image.namemaps — it yields untagged refs, so it wouldn't solve this, and would add false positives elsewhere.Downstream
appscode-cloud/installer#1319, then appscode-cloud/artifacts#4.
Unrelated bug this surfaced
The installer's reloader override is dead code: it writes
reloader.deployment.image.name, butappscode-charts/reloader:2.2.9reads the image from top-levelimage.repository— that path doesn't exist in the chart's values. So clusters pullghcr.io/stakater/reloader, notghcr.io/appscode/reloader. This PR emits what is actually pulled.