Skip to content

fix: Mitigate render-drift for archive-overlays#257

Draft
Tonisal-byte wants to merge 13 commits into
microsoft:mainfrom
Tonisal-byte:asalinas/render-autodetect-archive-overlays
Draft

fix: Mitigate render-drift for archive-overlays#257
Tonisal-byte wants to merge 13 commits into
microsoft:mainfrom
Tonisal-byte:asalinas/render-autodetect-archive-overlays

Conversation

@Tonisal-byte

Copy link
Copy Markdown
Contributor

We need a way to track the resulting hash from the sum of archive overlays applied to a source. This is a blocking issue for adding archive-overlays to upstream; otherwise the render checks fail from having a stale sources file pre-archive-overlay.

Example of new origin-stub:

[[components.apache-commons-compress.source-files]]
filename = "commons-compress-1.27.1-src.tar.gz"
hash = "c7a2cef26959e687ad19b96b5ba8393d7514095e13bf0f29bd41e6b3c3cb2260d8ff23283ff3d5fd137b2522b843e7f0f50ab46bcf0f66df5383674f35f223ab"
hash-type = "SHA512"
origin = { type = "overlay"}
replace-upstream = true
replace-reason = "Upstream source tarball contains test fixtures flagged as malware by the AZL RPM signing pipeline. These files are not needed at runtime and are removed to allow SRPM publication."

Antonio Salinas added 13 commits June 26, 2026 17:07
Add three new overlay types (tarball-file-remove, tarball-search-replace,
tarball-patch) that modify files inside source tarballs during source
preparation. Operations are performed in pure Go on the host.

Includes:
- internal/utils/tarball: reusable deterministic tar extract/repack library
- Overlay type registration, validation, and fingerprinting
- Source prep integration with sources file hash rehashing
- User documentation and TOML examples
Copilot AI review requested due to automatic review settings June 28, 2026 19:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces first-class support for archive-scoped overlays (overlays that mutate files inside a source tarball) and adds a mechanism to stabilize render --check-only by allowing configs to record the post-overlay archive hash via a new origin.type = "overlay" source-files entry.

Changes:

  • Add archive overlay processing (extract → apply overlays → deterministic repack) and refresh affected sources entries by rehashing repacked archives.
  • Add origin.type = "overlay" to source-files to record expected post-overlay hashes and avoid render drift when sources aren’t downloaded.
  • Harden archive extraction: sniff real compression from magic bytes, ignore PAX headers, and optionally fail on unsupported tar entry types to prevent silent data loss on repack.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
schemas/azldev.schema.json Extends schema enum to allow "overlay" as an origin type.
internal/utils/archive/archive.go Adds compression sniffing, extraction options, and convenience helpers for archive detection/extraction.
internal/utils/archive/archive_test.go Adds coverage for magic-byte sniffing, mislabeled compression, and unsupported tar entry handling.
internal/providers/sourceproviders/sourcemanager.go Skips downloading origin.type=overlay entries and guards against accidental download attempts.
internal/projectconfig/overlay.go Adds archive-scoped overlay path detection/validation and separates “archive” vs “loose file” overlay behavior.
internal/projectconfig/overlay_test.go Tests archive-scoped path validation and Modifies* classification.
internal/projectconfig/fingerprint_test.go Improves fingerprint tag validation by parsing name,options and constraining allowed options.
internal/projectconfig/configfile.go Validates overlay-origin source-files invariants (hash/hash-type required, replace-upstream enforced, uri forbidden).
internal/projectconfig/component.go Introduces OriginTypeOverlay and updates jsonschema tags accordingly.
internal/fingerprint/fingerprint_test.go Adds regression test ensuring archive scoping impacts component fingerprinting.
internal/app/azldev/core/sources/sourceprep.go Adds archive overlay application phase and rehashes sources entries for repacked archives; validates overlay hash expectations.
internal/app/azldev/core/sources/sourceprep_test.go Adds end-to-end tests for repack+rehash behavior and missing sources-entry erroring.
internal/app/azldev/core/sources/overlays.go Updates overlay application to use the new “loose files” predicate.
internal/app/azldev/core/sources/archiveoverlays.go Implements archive overlay grouping, extraction-root resolution, and atomic repack preserving real compression.
internal/app/azldev/core/sources/archiveoverlays_test.go Adds exported-path integration tests for archive overlay semantics (glob removal, search/replace, skip-sources behavior).
internal/app/azldev/core/sources/archiveoverlays_internal_test.go Adds internal unit/regression tests for grouping, extract-root resolution, strict extraction, and repack correctness.
internal/app/azldev/cmds/component/preparesources.go Refactors conditional preparer option wiring into a helper.
docs/user/reference/config/overlays.md Documents archive overlays (path scoping, extraction root, supported entry types, batching).
docs/user/reference/config/components.md Documents origin.type="overlay" and recommended workflow for recording post-overlay hashes.

Comment on lines +14 to +17
// Only regular files, directories, and symlinks are extracted. Other entry
// types (hardlinks, devices, FIFOs, etc.) are skipped by default, or cause a
// failure when [WithErrorOnUnsupportedEntry] is set — callers that repack the
// tree must set it so such entries aren't silently dropped.
Comment on lines +370 to +373
1. Add the archive overlay(s) in the component's `[[overlays]]` array.
2. Run `prep-sources` once — this repacks the archive and prints the computed hash in the error message.
3. Paste the computed `hash` and `hash-type` into the `source-files` entry above.
4. Run `prep-sources` again to confirm the hash matches, then commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants