diff --git a/docs/adr/54120-package-resources-with-scoped-ownership.md b/docs/adr/54120-package-resources-with-scoped-ownership.md new file mode 100644 index 00000000000..20533749350 --- /dev/null +++ b/docs/adr/54120-package-resources-with-scoped-ownership.md @@ -0,0 +1,48 @@ +# ADR-54120: Package Resources with Scoped Ownership + +**Date**: 2026-08-20 +**Status**: Draft +**Deciders**: pelikhan, copilot-swe-agent + +--- + +### Context + +Repository packages (`aw.yml` manifests) could install workflows, skills, and agents, but had no mechanism to bundle supplementary repository assets such as Issue Forms (`.github/ISSUE_TEMPLATE/*.yml`), `CODEOWNERS`, or policy files under `.github/aw/`. Consumers who needed these files had to copy them manually alongside `gh aw add`, breaking the self-contained package installation model. The gap was tracked in issue #52769. + +### Decision + +We will introduce a `resources:` field in the `aw.yml` package manifest. Each resource entry declares a package-relative `source` and a repository-root-relative `destination`. Destinations are restricted to an explicit allowlist (`.github/ISSUE_TEMPLATE/*.yml|*.yaml`, `.github/CODEOWNERS`, `.github/aw/**`). Resources are copied as inert content (no compilation, no secret injection). For every package installation, SHA-256-based ownership records are written under `.github/aw/packages/*.json`, and updates refuse to overwrite locally drifted files unless `--force` is passed. Stale resource files are removed on `gh aw update` when they are dropped from the manifest and unchanged since installation. + +### Alternatives Considered + +#### Alternative 1: Extend the existing `includes` / `files` field + +The `includes` field already supports explicit source-to-destination mappings for installable workflow files. Resources could be added there with a special flag or naming convention distinguishing inert-copy from compiled-workflow semantics. + +Rejected because mixing the two installation modes in one field creates ambiguity: `includes` entries go through compilation and `.md`-to-workflow translation steps that are inappropriate for raw YAML or JSON assets. Adding a discriminant flag would complicate the schema and parser without a natural extension point. + +#### Alternative 2: Document-only / manual copy instructions + +Packages could document supplementary files in their README and expect users to copy them manually. This preserves simplicity in the CLI. + +Rejected because it breaks the single-command (`gh aw add`) installation promise and requires package consumers to know which files to copy, defeating the purpose of a manifest-driven package system. + +### Consequences + +#### Positive +- Packages can ship a complete, self-contained repository setup โ€” workflows, skills, agents, issue templates, CODEOWNERS, and policy files โ€” in one `gh aw add` invocation. +- SHA-256 ownership records prevent silent overwrites of locally modified files, making update safety explicit and auditable. +- The allowlist of valid destinations prevents packages from writing to arbitrary repository paths, limiting the blast radius of a malicious or misconfigured package. + +#### Negative +- The destination allowlist (ISSUE_TEMPLATE, CODEOWNERS, `.github/aw/**`) must be maintained as product needs evolve; adding new allowed namespaces requires a code change and specification update. +- Packages that declare only `resources:` (no workflows/skills/agents) are now valid, which changes the emptiness check in `resolveRepositoryPackage` and may surface unexpected edge cases in tooling that assumes at least one workflow is present. + +#### Neutral +- A new `IsPackageResourceFile` discriminant is added to `WorkflowSpec` and `ResolvedWorkflow`; bootstrap profile helpers skip resource files when inferring Copilot Auth and GitHub App permission requirements, consistent with how skill and agent files are handled. +- The `gh aw remove` command gains cleanup logic that removes package-owned resource files when the last workflow from a package is removed and the files are still unchanged since installation. + +--- + +*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.* diff --git a/docs/src/content/docs/reference/repository-package-manifest.md b/docs/src/content/docs/reference/repository-package-manifest.md index b6818abf342..85bca0847b3 100644 --- a/docs/src/content/docs/reference/repository-package-manifest.md +++ b/docs/src/content/docs/reference/repository-package-manifest.md @@ -22,9 +22,16 @@ emoji: ๐Ÿค– description: Friendly repository automation for review and issue triage includes: - workflows/review.md # agentic workflow โ€” compiled on install + - source: templates/triage.yml # explicit source/destination mapping + destination: .github/workflows/triage.yml - skills/code-review # skill directory (must contain SKILL.md) - agents/reviewer.md # agent file - .github/workflows/ci.yml # raw Actions YAML โ€” copied verbatim +resources: + - source: templates/bug.yml + destination: .github/ISSUE_TEMPLATE/bug.yml + - source: policy/controls.json + destination: .github/aw/policy/controls.json ``` ## Quick reference @@ -36,7 +43,8 @@ includes: | `name` | string | Yes | Human-readable package name. Must be non-empty after trimming whitespace. | | `emoji` | string | No | Optional package emoji for display in package metadata. | | `description` | string | No | Optional package description. `gh aw add` warns when it exceeds 255 characters. | -| `includes` | array of strings | No | Package-root-relative paths. Type is inferred from folder naming: workflows (`workflows/`, `agentic-workflows/`, `.github/workflows/`), skills (`skills/`, `.github/skills/`), agents (`agents/`, `.github/agents/`). | +| `includes` | array of strings or mappings | No | Explicit install entries. Strings use path conventions; mappings declare package-relative `source` and repository-root-relative `destination` for workflow installs under `.github/workflows/`. | +| `resources` | array of mappings | No | Declarative repository assets copied as-is from package content to allowlisted destinations. | | `files` | array of strings | No | Deprecated alias. Use `includes` instead. | ## Documentation @@ -60,6 +68,21 @@ If `includes` is present, valid entries are used as the install bundle. Supporte - **Skills** โ€” directory paths under `skills/` or `.github/skills/` that contain `SKILL.md`. - **Agents** โ€” `.md` files under `agents/` or `.github/agents/`. +Mapping entries in `includes` can install workflow files from package-relative `source` paths into explicit repository-root `destination` paths under `.github/workflows/`. + +## Resources + +`resources` installs inert repository assets from package content: + +- `source` is always package-root-relative. +- `destination` is always repository-root-relative. +- Allowed destination namespaces: + - `.github/ISSUE_TEMPLATE/*.{yml,yaml}` (direct children only; nested subdirectories such as `.github/ISSUE_TEMPLATE/bug_report/bug.yml` fail manifest validation) + - `.github/CODEOWNERS` + - `.github/aw/**` + +`gh aw add` rejects duplicate (including case-insensitive duplicate) resource destinations, path traversal, and non-regular local resource sources. Installed resources are tracked with package-scoped ownership metadata in `.github/aw/packages/*.json`. + If `includes` is omitted or contains no valid workflow paths, `gh aw add` scans: - `workflows/` @@ -67,4 +90,6 @@ If `includes` is omitted or contains no valid workflow paths, `gh aw add` scans: For nested packages, those paths are resolved relative to the package root. +Packages can still be valid without workflows when they declare installable `resources`, skills, or agents. + The embedded JSON schema source of truth lives in `pkg/parser/schemas/aw_manifest_schema.json`. diff --git a/docs/src/content/docs/specs/repository-package-manifest-specification.md b/docs/src/content/docs/specs/repository-package-manifest-specification.md index 393c82b8d18..04d332ba733 100644 --- a/docs/src/content/docs/specs/repository-package-manifest-specification.md +++ b/docs/src/content/docs/specs/repository-package-manifest-specification.md @@ -7,7 +7,7 @@ sidebar: # aw.yml Repository Package Manifest Specification -**Version**: 0.2.0 +**Version**: 0.2.1 **Status**: Draft ## Abstract @@ -49,6 +49,7 @@ The manifest document MUST be a YAML mapping. Unknown top-level fields MUST be r | `license` | string | No | SPDX license identifier or license name for the package. | | `files` | array of strings | No | Deprecated. Explicit installable workflow file list. Use `includes` instead. | | `includes` | array of strings or mappings | No | Explicit installable package entries. String entries use path conventions; mapping entries declare an explicit source-to-destination install path. | +| `resources` | array of mappings | No | Declarative repository assets copied as-is to allowlisted destinations. | ### 4.2 `manifest-version` @@ -126,6 +127,28 @@ Mapping entries follow the same install semantics as string entries: `.md` sourc `gh aw add`, `gh aw add-wizard`, and `gh aw update` MUST use identical mapping semantics, and `gh aw update` MUST continue to track the manifest source of installed files. +### 4.10 `resources` + +If present, `resources` MUST be an array of mappings. Each mapping MUST contain: + +| Key | Type | Required | Meaning | +| --- | --- | --- | --- | +| `source` | string | Yes | Package-relative path of the asset to copy. | +| `destination` | string | Yes | Repository-root-relative destination path. | + +Resource `source` and `destination` values MUST NOT be absolute paths and MUST NOT escape their roots through path traversal. Local package sources MUST NOT be symbolic links, directories, or other non-regular file replacements. + +Resource destinations are restricted to non-hook repository asset namespaces: + +- `.github/ISSUE_TEMPLATE/*.yml` +- `.github/ISSUE_TEMPLATE/*.yaml` +- `.github/CODEOWNERS` +- `.github/aw/**` + +Implementations MUST reject duplicate or case-insensitive duplicate resource destinations before writing files. Resources are copied as inert content from the selected package ref; installers MUST NOT execute package-provided scripts or expose configured secrets to package content during installation. + +For each package installation, implementations MUST record package-scoped ownership metadata under `.github/aw/packages/`. The record MUST identify the package source, resolved immutable commit/ref, installed destination paths, source paths, and SHA-256 content digests. Implementations MUST refuse to overwrite existing resource files unless they are unchanged files owned by the same package, or unless the user explicitly passes `--force`. + ## 5. Installable file resolution Supported installable paths are: @@ -149,7 +172,7 @@ If `files` is omitted, or if no valid entries remain after filtering, the implem Auto-discovery considers only agentic workflow markdown (`.md`); raw `.yml` action workflows MUST be referenced explicitly in `files` to be installed. -If no installable workflow files are resolved, package validation MUST fail. +If no installable package assets are resolved (workflows, resources, skills, or agents), package validation MUST fail. ### 5.1 Install @@ -159,7 +182,8 @@ The install lifecycle (invoked by `gh aw add`) MUST proceed in the following ord 2. **Resolve** the installable file list per ยง5. 3. **Download** each resolved file from the remote package source. 4. **Compile** each agentic workflow markdown file into the target repository's workflow directory. Raw `.yml` files are copied verbatim without compilation. -5. **Write** all output files atomically before reporting success. +5. **Copy** declared `resources` as inert repository assets without executing them. +6. **Write** all output files and package ownership metadata atomically before reporting success. If any step fails, the implementation MUST abort and MUST NOT leave partial output files in the target directory. The implementation SHOULD emit an actionable error identifying the failing step. See ยง10 (Safeguards) for the normative rollback and permission-error requirements that apply to this lifecycle (R-PKG-003, R-PKG-004, R-PKG-006, R-PKG-007). @@ -169,7 +193,7 @@ The update lifecycle re-installs a package at a newer (or specified) version, ov **R-PKG-U001**: `gh aw add` with a version specifier (e.g., `owner/repo@v2.0.0`) MUST overwrite previously installed files from the same package with the new version's files, following the same install ordering defined in ยง5.1. -**R-PKG-U002**: Files that were present in the previous installation but are absent from the new version's resolved file list MUST be left in place. The implementation SHOULD emit a warning for each such orphaned file, identifying the file by path and noting that it was not present in the new version. +**R-PKG-U002**: Files that were present in the previous installation but are absent from the new version's resolved package-managed file list MUST be removed only when all of the following hold: (a) they are owned by the same package, (b) they are unchanged from the recorded digest, and (c) no replacement from the new version maps to the same path. When a new version entry maps to the same path, overwrite behavior is governed by R-PKG-U001. Implementations SHOULD warn when stale files are preserved because they were modified or ownership cannot be proven. **R-PKG-U003**: If overwriting a file fails (for example, due to a filesystem permission error or a locked file), the implementation MUST abort the update and MUST NOT leave the target directory in a mixed state combining old and new file versions. The implementation MUST emit an error identifying the file that could not be overwritten and the reason. @@ -213,7 +237,7 @@ Validation MUST fail for at least the following conditions: - current compiler version is lower than `min-version`; - unknown top-level fields, including `docs`; or - missing required `README.md`; or -- no installable workflow files resolved. +- no installable package assets (workflows, resources, skills, or agents) resolved. Implementations SHOULD emit warnings for at least the following conditions: diff --git a/package.md b/package.md index aaa9184e545..68def5fd32e 100644 --- a/package.md +++ b/package.md @@ -49,6 +49,9 @@ emoji: ๐Ÿค– files: - workflows/example.md - .github/workflows/repo-workflow.md +resources: + - source: templates/bug.yml + destination: .github/ISSUE_TEMPLATE/bug.yml ``` Requirements: @@ -58,6 +61,7 @@ Requirements: - `description`: concise and relevant to the actual workflows - `emoji`: optional package emoji (string) - `files`: complete list of installable agentic/shared workflows in this repository +- `resources`: optional package-root-relative assets copied to allowlisted destinations such as `.github/ISSUE_TEMPLATE/*.yml`, `.github/CODEOWNERS`, or `.github/aw/**` - File paths must be package-root-relative and point to existing markdown workflow files under `workflows/` or `.github/workflows/` Do not invent custom package metadata fields. diff --git a/pkg/cli/add_command.go b/pkg/cli/add_command.go index 2f05e205fb6..b2329a56fad 100644 --- a/pkg/cli/add_command.go +++ b/pkg/cli/add_command.go @@ -365,6 +365,15 @@ func addWorkflowsWithTracking(ctx context.Context, workflows []*ResolvedWorkflow } } + if err := writePackageOwnershipRecords(workflows, tracker, opts); err != nil { + if tracker != nil { + if rollbackErr := tracker.RollbackAllFiles(opts.Verbose); rollbackErr != nil { + return fmt.Errorf("failed to write package ownership records (rollback also failed): %w", errors.Join(err, rollbackErr)) + } + } + return err + } + if !opts.Quiet && len(workflows) > 1 { fmt.Fprintln(os.Stderr, console.FormatSuccessMessage(fmt.Sprintf("Successfully added all %d workflows", len(workflows)))) } @@ -406,6 +415,10 @@ func addWorkflowWithTracking(ctx context.Context, resolved *ResolvedWorkflow, tr if resolved.IsPackageAgentFile { return addAgentFileWithTracking(resolved, tracker, opts, gitRoot) } + // Package resources are copied as-is to their declared repository-relative destinations. + if resolved.IsPackageResourceFile { + return addResourceFileWithTracking(resolved, tracker, opts, gitRoot) + } sourceRepo := "" if sourceInfo != nil && !sourceInfo.IsLocal { sourceRepo = workflowSpec.RepoSlug @@ -469,6 +482,49 @@ func validateWorkflowDestination(githubWorkflowsDir, workflowName, sourceRepo st return false, fmt.Errorf("workflow '%s' already exists in .github/workflows/. Use a different name with -n flag, remove the existing workflow first, or use --force to overwrite", workflowName) } +func addResourceFileWithTracking(resolved *ResolvedWorkflow, tracker *FileTracker, opts AddOptions, gitRoot string) error { + destination := filepath.Clean(filepath.FromSlash(resolved.Spec.DestinationPath)) + if destination == "." || filepath.IsAbs(destination) || strings.HasPrefix(destination, ".."+string(os.PathSeparator)) { + return fmt.Errorf("resource destination %q is invalid", resolved.Spec.DestinationPath) + } + destFile := filepath.Join(gitRoot, destination) + rel, err := filepath.Rel(gitRoot, destFile) + if err != nil { + return fmt.Errorf("failed to validate resource destination %q: %w", resolved.Spec.DestinationPath, err) + } + if rel == ".." || strings.HasPrefix(rel, ".."+string(os.PathSeparator)) { + return fmt.Errorf("resource destination %q escapes repository root", resolved.Spec.DestinationPath) + } + + fileExists := fileutil.FileExists(destFile) + if fileExists && !opts.Force { + packageSource := packageSourceForSpec(resolved.Spec, resolved.SourceInfo) + if owned, drifted := packageOwnershipAllowsOverwrite(gitRoot, rel, packageSource); !owned || drifted { + if owned { + return fmt.Errorf("resource %q has local modifications; use --force to overwrite", resolved.Spec.DestinationPath) + } + return fmt.Errorf("resource %q already exists; use --force to overwrite", resolved.Spec.DestinationPath) + } + } + if err := os.MkdirAll(filepath.Dir(destFile), constants.DirPermPublic); err != nil { + return fmt.Errorf("failed to create resource directory %s: %w", filepath.Dir(destFile), err) + } + if tracker != nil { + if fileExists { + tracker.TrackModified(destFile) + } else { + tracker.TrackCreated(destFile) + } + } + if err := os.WriteFile(destFile, resolved.Content, constants.FilePermPublic); err != nil { + return fmt.Errorf("failed to write resource file %q: %w", destFile, err) + } + if !opts.Quiet { + fmt.Fprintln(os.Stderr, console.FormatSuccessMessage("Added resource: "+filepath.ToSlash(rel))) + } + return nil +} + func compileAddedWorkflow(ctx context.Context, destFile string, workflowSpec *WorkflowSpec, githubWorkflowsDir string, tracker *FileTracker, opts AddOptions) { // For remote workflows: now that the main workflow and all its imports are on disk, // parse the fully merged safe-outputs configuration to discover any dispatch or diff --git a/pkg/cli/add_command_test.go b/pkg/cli/add_command_test.go index ea90a8c037e..fd468337042 100644 --- a/pkg/cli/add_command_test.go +++ b/pkg/cli/add_command_test.go @@ -942,6 +942,108 @@ func TestAddWorkflowWithTracking_ActionWorkflow_Force(t *testing.T) { assert.Equal(t, newContent, written) } +func TestAddWorkflowsWithTracking_PackageResourceWritesOwnershipRecord(t *testing.T) { + tempDir := testutil.TempDir(t, "test-package-resource-*") + setupMinimalGitRepo(t, tempDir) + + resourceContent := []byte("name: Bug report\n") + workflows := []*ResolvedWorkflow{ + { + Spec: &WorkflowSpec{ + RepoSpec: RepoSpec{ + RepoSlug: "owner/repo", + Version: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + PackagePath: "packages/repo-assist", + }, + WorkflowPath: "packages/repo-assist/templates/bug.yml", + WorkflowName: "bug", + DestinationPath: ".github/ISSUE_TEMPLATE/bug.yml", + FromRepositoryManifest: true, + IsPackageResourceFile: true, + }, + Content: resourceContent, + SourceInfo: &FetchedWorkflow{ + Content: resourceContent, + CommitSHA: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + IsLocal: false, + SourcePath: "packages/repo-assist/templates/bug.yml", + }, + IsPackageResourceFile: true, + }, + } + + err := addWorkflowsWithTracking(context.Background(), workflows, NewFileTracker(), AddOptions{ + NoGitattributes: true, + DisableSecurityScanner: true, + Quiet: true, + }) + require.NoError(t, err) + + resourcePath := filepath.Join(tempDir, ".github", "ISSUE_TEMPLATE", "bug.yml") + written, err := os.ReadFile(resourcePath) + require.NoError(t, err) + assert.Equal(t, resourceContent, written) + + recordFiles, err := filepath.Glob(filepath.Join(tempDir, ".github", "aw", "packages", "*.json")) + require.NoError(t, err) + require.Len(t, recordFiles, 1) + record, err := os.ReadFile(recordFiles[0]) + require.NoError(t, err) + assert.Contains(t, string(record), `"source": "owner/repo/packages/repo-assist@aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"`) + assert.Contains(t, string(record), `"destination": ".github/ISSUE_TEMPLATE/bug.yml"`) + assert.Contains(t, string(record), `"sha256":`) +} + +func TestAddWorkflowsWithTracking_PackageResourceRejectsLocalDrift(t *testing.T) { + tempDir := testutil.TempDir(t, "test-package-resource-drift-*") + setupMinimalGitRepo(t, tempDir) + + spec := &WorkflowSpec{ + RepoSpec: RepoSpec{ + RepoSlug: "owner/repo", + Version: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + PackagePath: "packages/repo-assist", + }, + WorkflowPath: "packages/repo-assist/policy/controls.json", + WorkflowName: "controls", + DestinationPath: ".github/aw/policy/controls.json", + FromRepositoryManifest: true, + IsPackageResourceFile: true, + } + first := []*ResolvedWorkflow{{ + Spec: spec, + Content: []byte(`{"version":1}`), + SourceInfo: &FetchedWorkflow{Content: []byte(`{"version":1}`), CommitSHA: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}, + IsPackageResourceFile: true, + }} + err := addWorkflowsWithTracking(context.Background(), first, NewFileTracker(), AddOptions{ + NoGitattributes: true, + DisableSecurityScanner: true, + Quiet: true, + }) + require.NoError(t, err) + + resourcePath := filepath.Join(tempDir, ".github", "aw", "policy", "controls.json") + require.NoError(t, os.WriteFile(resourcePath, []byte(`{"local":true}`), 0644)) + + second := []*ResolvedWorkflow{{ + Spec: spec, + Content: []byte(`{"version":2}`), + SourceInfo: &FetchedWorkflow{Content: []byte(`{"version":2}`), CommitSHA: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}, + IsPackageResourceFile: true, + }} + err = addWorkflowsWithTracking(context.Background(), second, NewFileTracker(), AddOptions{ + NoGitattributes: true, + DisableSecurityScanner: true, + Quiet: true, + }) + require.Error(t, err) + require.ErrorContains(t, err, "local modifications") + written, readErr := os.ReadFile(resourcePath) + require.NoError(t, readErr) + assert.Equal(t, `{"local":true}`, string(written)) +} + func TestAddWorkflowsWithTracking_RollsBackWrittenFilesOnWriteFailure(t *testing.T) { tempDir := testutil.TempDir(t, "test-add-workflows-rollback-*") workflowsDir := setupMinimalGitRepo(t, tempDir) diff --git a/pkg/cli/add_package_manifest.go b/pkg/cli/add_package_manifest.go index 2a9195c6e9c..3f336aae5ac 100644 --- a/pkg/cli/add_package_manifest.go +++ b/pkg/cli/add_package_manifest.go @@ -56,6 +56,7 @@ type resolvedRepositoryPackage struct { License string DocsPath string InstallationSource []resolvedPackageInstallable + ResourceFiles []resolvedPackageResource Bootstrap *repositoryPackageBootstrap SkillFiles []resolvedPackageSkillFile AgentFiles []string @@ -74,6 +75,11 @@ type resolvedPackageInstallable struct { DestinationPath string } +type resolvedPackageResource struct { + SourcePath string + DestinationPath string +} + // packageInstallableSourcePaths returns the source paths of the given installables. func packageInstallableSourcePaths(installables []resolvedPackageInstallable) []string { paths := make([]string, 0, len(installables)) diff --git a/pkg/cli/add_package_manifest_parse.go b/pkg/cli/add_package_manifest_parse.go index e242d16fe0b..c343b0de783 100644 --- a/pkg/cli/add_package_manifest_parse.go +++ b/pkg/cli/add_package_manifest_parse.go @@ -23,6 +23,7 @@ type repositoryPackageManifest struct { License string Includes []repositoryPackageInclude Files []string + Resources []repositoryPackageResource Bootstrap *repositoryPackageBootstrap Skills []string // skill directory paths (e.g. "skills/my-skill") Agents []string // agent .md file paths (e.g. "agents/my-agent.md") @@ -142,6 +143,14 @@ func populateRepositoryPackageManifestMetadata(manifest *repositoryPackageManife } } + if resourcesValue, ok := root["resources"]; ok { + resources, err := extractManifestResources(resourcesValue, manifestPath) + if err != nil { + return nil, err + } + manifest.Resources = resources + } + if skillsValue, ok := root["skills"]; ok { skills, skillWarnings := extractManifestSkillDirs(skillsValue, manifestPath) manifest.Skills = skills diff --git a/pkg/cli/add_package_manifest_resolve.go b/pkg/cli/add_package_manifest_resolve.go index 85b785903ea..1f9faf0ba74 100644 --- a/pkg/cli/add_package_manifest_resolve.go +++ b/pkg/cli/add_package_manifest_resolve.go @@ -31,6 +31,7 @@ func resolveRepositoryPackage(ctx context.Context, repoSpec *RepoSpec, host stri if err != nil { return nil, err } + resourceFiles := normalizePackageResourcePaths(manifest.Resources, packagePath) docsPath, err := resolveRepositoryPackageDocsPath(ctx, owner, repo, packagePath, ref, host) if err != nil { @@ -52,11 +53,11 @@ func resolveRepositoryPackage(ctx context.Context, repoSpec *RepoSpec, host stri } warnings = append(warnings, extensionFiles.warnings...) - if len(installationSources) == 0 && len(extensionFiles.skillFiles) == 0 && len(extensionFiles.agentFiles) == 0 { - return nil, fmt.Errorf("repository %q does not contain any installable workflows, skills, or agents (either explicitly declared or auto-discovered). Add workflows under 'workflows/', skills under 'skills/', or agents under 'agents/', or declare them explicitly in aw.yml", repositoryPackageIdentifier(repoSpec.RepoSlug, packagePath)) + if len(installationSources) == 0 && len(resourceFiles) == 0 && len(extensionFiles.skillFiles) == 0 && len(extensionFiles.agentFiles) == 0 { + return nil, fmt.Errorf("repository %q does not contain any installable workflows, resources, skills, or agents (either explicitly declared or auto-discovered). Add workflows under 'workflows/', resources in aw.yml, skills under 'skills/', or agents under 'agents/', or declare them explicitly in aw.yml", repositoryPackageIdentifier(repoSpec.RepoSlug, packagePath)) } - return newResolvedRepositoryPackage(manifestPath, ref, docsPath, manifest, installationSources, extensionFiles, warnings), nil + return newResolvedRepositoryPackage(manifestPath, ref, docsPath, manifest, installationSources, resourceFiles, extensionFiles, warnings), nil } func splitRepositoryPackageSlug(repoSlug string) (string, string, error) { @@ -152,7 +153,7 @@ func resolveRepositoryPackageExtensionFiles(ctx context.Context, options reposit }, nil } -func newResolvedRepositoryPackage(manifestPath, ref, docsPath string, manifest *repositoryPackageManifest, installationSources []resolvedPackageInstallable, extensionFiles *repositoryPackageExtensionFiles, warnings []string) *resolvedRepositoryPackage { +func newResolvedRepositoryPackage(manifestPath, ref, docsPath string, manifest *repositoryPackageManifest, installationSources []resolvedPackageInstallable, resourceFiles []resolvedPackageResource, extensionFiles *repositoryPackageExtensionFiles, warnings []string) *resolvedRepositoryPackage { return &resolvedRepositoryPackage{ ManifestPath: manifestPath, ResolvedRef: ref, @@ -162,6 +163,7 @@ func newResolvedRepositoryPackage(manifestPath, ref, docsPath string, manifest * License: manifest.License, DocsPath: docsPath, InstallationSource: installationSources, + ResourceFiles: resourceFiles, Bootstrap: manifest.Bootstrap, SkillFiles: extensionFiles.skillFiles, AgentFiles: extensionFiles.agentFiles, diff --git a/pkg/cli/add_package_manifest_resources.go b/pkg/cli/add_package_manifest_resources.go new file mode 100644 index 00000000000..6a29be0e526 --- /dev/null +++ b/pkg/cli/add_package_manifest_resources.go @@ -0,0 +1,125 @@ +package cli + +import ( + "errors" + "fmt" + "path" + "path/filepath" + "strings" + + "github.com/github/gh-aw/pkg/constants" +) + +type repositoryPackageResource struct { + Source string + Destination string +} + +func extractManifestResources(value any, manifestPath string) ([]repositoryPackageResource, error) { + items, ok := value.([]any) + if !ok { + return nil, fmt.Errorf("invalid Agentic Workflow manifest %q: resources must be a list of source/destination mappings", manifestPath) + } + resources := make([]repositoryPackageResource, 0, len(items)) + seenDestinations := make(map[string]string, len(items)) + for _, item := range items { + mapping, ok := item.(map[string]any) + if !ok { + return nil, fmt.Errorf("invalid Agentic Workflow manifest %q: resources entries must be source/destination mappings", manifestPath) + } + resource, err := parseManifestResourceMapping(mapping, manifestPath) + if err != nil { + return nil, err + } + key := strings.ToLower(resource.Destination) + if previous, exists := seenDestinations[key]; exists { + return nil, fmt.Errorf("invalid Agentic Workflow manifest %q: resources entries %q and %q both install to %q. Each resource must have a unique destination", manifestPath, previous, resource.Source, resource.Destination) + } + seenDestinations[key] = resource.Source + resources = append(resources, resource) + } + return resources, nil +} + +func parseManifestResourceMapping(mapping map[string]any, manifestPath string) (repositoryPackageResource, error) { + source, _ := stringValue(mapping["source"]) + destination, _ := stringValue(mapping["destination"]) + source = strings.TrimSpace(source) + destination = strings.TrimSpace(destination) + if source == "" || destination == "" { + return repositoryPackageResource{}, fmt.Errorf("invalid Agentic Workflow manifest %q: resources entries require non-empty 'source' and 'destination'. Example:\nresources:\n - source: templates/bug.yml\n destination: .github/ISSUE_TEMPLATE/bug.yml", manifestPath) + } + + cleanedSource, err := cleanManifestRelativePath(source) + if err != nil { + return repositoryPackageResource{}, fmt.Errorf("invalid Agentic Workflow manifest %q: resources source %q is invalid: %w", manifestPath, source, err) + } + cleanedDestination, err := cleanManifestRelativePath(destination) + if err != nil { + return repositoryPackageResource{}, fmt.Errorf("invalid Agentic Workflow manifest %q: resources destination %q is invalid: %w", manifestPath, destination, err) + } + if err := validateManifestResourceDestination(cleanedDestination); err != nil { + return repositoryPackageResource{}, fmt.Errorf("invalid Agentic Workflow manifest %q: resources destination %q is invalid: %w", manifestPath, destination, err) + } + return repositoryPackageResource{Source: cleanedSource, Destination: cleanedDestination}, nil +} + +func validateManifestResourceDestination(destination string) error { + switch { + case strings.HasPrefix(destination, constants.GithubDir+"ISSUE_TEMPLATE/"): + remaining := strings.TrimPrefix(destination, constants.GithubDir+"ISSUE_TEMPLATE/") + if remaining == "" || strings.Contains(remaining, "/") { + return fmt.Errorf("issue template resources must be direct children of %sISSUE_TEMPLATE", constants.GithubDir) + } + lower := strings.ToLower(remaining) + if !strings.HasSuffix(lower, ".yml") && !strings.HasSuffix(lower, ".yaml") { + return errorsForResourceDestination() + } + return nil + case destination == constants.GithubDir+"CODEOWNERS": + return nil + case strings.HasPrefix(destination, constants.GithubDir+"aw/"): + remaining := strings.TrimPrefix(destination, constants.GithubDir+"aw/") + if remaining == "" || strings.HasPrefix(remaining, "../") { + return errorsForResourceDestination() + } + return nil + default: + return errorsForResourceDestination() + } +} + +func errorsForResourceDestination() error { + return errors.New("destinations must be .github/CODEOWNERS, .github/ISSUE_TEMPLATE/*.yml, .github/ISSUE_TEMPLATE/*.yaml, or under .github/aw/") +} + +func normalizePackageResourcePaths(resources []repositoryPackageResource, packagePath string) []resolvedPackageResource { + normalized := make([]resolvedPackageResource, 0, len(resources)) + for _, resource := range resources { + normalized = append(normalized, resolvedPackageResource{ + SourcePath: joinRepositoryPackagePath(packagePath, resource.Source), + DestinationPath: resource.Destination, + }) + } + return normalized +} + +func normalizeLocalPackageResourcePaths(resources []repositoryPackageResource, packageDir string) ([]resolvedPackageResource, error) { + normalized := make([]resolvedPackageResource, 0, len(resources)) + for _, resource := range resources { + absolutePath := filepath.Clean(filepath.Join(packageDir, filepath.FromSlash(resource.Source))) + if err := validateLocalPackageMappingSource(absolutePath, packageDir, resource.Source); err != nil { + return nil, err + } + normalized = append(normalized, resolvedPackageResource{ + SourcePath: absolutePath, + DestinationPath: resource.Destination, + }) + } + return normalized, nil +} + +func packageResourceName(resource resolvedPackageResource) string { + base := path.Base(filepath.ToSlash(resource.DestinationPath)) + return strings.TrimSuffix(base, path.Ext(base)) +} diff --git a/pkg/cli/add_package_manifest_test.go b/pkg/cli/add_package_manifest_test.go index 1b713174b87..07876f3ee20 100644 --- a/pkg/cli/add_package_manifest_test.go +++ b/pkg/cli/add_package_manifest_test.go @@ -92,6 +92,36 @@ files: getRepositoryPackageDefaultBranch = previousDefaultBranch getRepositoryPackageLatestRelease = previousLatestRelease }) + + t.Run("uses resources mappings", func(t *testing.T) { + downloadPackageFileFromGitHubForHost = func(_ context.Context, owner, repo, path, ref, host string) ([]byte, error) { + switch path { + case "packages/repo-assist/aw.yml": + return []byte(`name: Repo Assist +resources: + - source: templates/bug.yml + destination: .github/ISSUE_TEMPLATE/bug.yml + - source: policy/controls.json + destination: .github/aw/policy/controls.json +`), nil + case "packages/repo-assist/README.md": + return []byte("# Repo Assist\n"), nil + default: + return nil, createRepositoryPackageNotFoundError(path) + } + } + listPackageWorkflowFilesForHost = func(_ context.Context, owner, repo, ref, workflowPath, host string) ([]string, error) { + return nil, createRepositoryPackageNotFoundError(workflowPath) + } + + pkg, err := resolveRepositoryPackage(t.Context(), &RepoSpec{RepoSlug: "owner/repo", PackagePath: "packages/repo-assist"}, "") + require.NoError(t, err) + require.Len(t, pkg.ResourceFiles, 2) + assert.Equal(t, "packages/repo-assist/templates/bug.yml", pkg.ResourceFiles[0].SourcePath) + assert.Equal(t, ".github/ISSUE_TEMPLATE/bug.yml", pkg.ResourceFiles[0].DestinationPath) + assert.Equal(t, "packages/repo-assist/policy/controls.json", pkg.ResourceFiles[1].SourcePath) + assert.Equal(t, ".github/aw/policy/controls.json", pkg.ResourceFiles[1].DestinationPath) + }) getRepositoryPackageLatestRelease = func(_ context.Context, repoSlug, host string) (string, error) { assert.Equal(t, "owner/repo", repoSlug) assert.Equal(t, "github.com", host) diff --git a/pkg/cli/add_package_ownership.go b/pkg/cli/add_package_ownership.go new file mode 100644 index 00000000000..6f4e46ee1dd --- /dev/null +++ b/pkg/cli/add_package_ownership.go @@ -0,0 +1,522 @@ +package cli + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "os" + "path/filepath" + "slices" + "strings" + + "github.com/github/gh-aw/pkg/console" + "github.com/github/gh-aw/pkg/constants" + "github.com/github/gh-aw/pkg/fileutil" + "github.com/github/gh-aw/pkg/gitutil" + "github.com/github/gh-aw/pkg/workflow" +) + +const packageOwnershipSchemaVersion = 1 + +type packageOwnershipRecord struct { + SchemaVersion int `json:"schemaVersion"` + Package string `json:"package"` + Source string `json:"source"` + ResolvedCommit string `json:"resolvedCommit,omitempty"` + Installer string `json:"installer"` + Files []packageOwnershipFileEntry `json:"files"` +} + +func sha256Bytes(content []byte) string { + sum := sha256.Sum256(content) + return hex.EncodeToString(sum[:]) +} + +type packageOwnershipFileEntry struct { + Source string `json:"source"` + Destination string `json:"destination"` + SHA256 string `json:"sha256"` +} + +func writePackageOwnershipRecords(workflows []*ResolvedWorkflow, tracker *FileTracker, opts AddOptions) error { + groups := packageManagedWorkflowGroups(workflows) + if len(groups) == 0 { + return nil + } + gitRoot, err := gitutil.FindGitRoot() + if err != nil { + return fmt.Errorf("failed to find git root for package ownership records: %w", err) + } + for packageSource, group := range groups { + record, err := buildPackageOwnershipRecord(gitRoot, packageSource, group, opts) + if err != nil { + return err + } + recordPath := packageOwnershipRecordPath(gitRoot, packageSource) + if err := os.MkdirAll(filepath.Dir(recordPath), constants.DirPermPublic); err != nil { + return fmt.Errorf("failed to create package ownership directory: %w", err) + } + existed := fileutil.FileExists(recordPath) + if tracker != nil { + if existed { + tracker.TrackModified(recordPath) + } else { + tracker.TrackCreated(recordPath) + } + } + data, err := json.MarshalIndent(record, "", " ") + if err != nil { + return fmt.Errorf("failed to encode package ownership record: %w", err) + } + data = append(data, '\n') + if err := os.WriteFile(recordPath, data, constants.FilePermPublic); err != nil { + return fmt.Errorf("failed to write package ownership record %s: %w", recordPath, err) + } + } + return nil +} + +func packageManagedWorkflowGroups(workflows []*ResolvedWorkflow) map[string][]*ResolvedWorkflow { + groups := make(map[string][]*ResolvedWorkflow) + for _, resolved := range workflows { + if resolved == nil || resolved.Spec == nil || !resolved.Spec.FromRepositoryManifest { + continue + } + source := packageSourceForSpec(resolved.Spec, resolved.SourceInfo) + if source == "" { + continue + } + groups[source] = append(groups[source], resolved) + } + return groups +} + +func packageSourceForSpec(spec *WorkflowSpec, sourceInfo *FetchedWorkflow) string { + if spec.RepoSlug == "" { + return "" + } + ref := spec.Version + if sourceInfo != nil && sourceInfo.CommitSHA != "" { + ref = sourceInfo.CommitSHA + } + return manifestSourceWithRef(&RepoSpec{ + RepoSlug: spec.RepoSlug, + PackagePath: spec.PackagePath, + }, ref) +} + +func buildPackageOwnershipRecord(gitRoot, packageSource string, workflows []*ResolvedWorkflow, opts AddOptions) (*packageOwnershipRecord, error) { + record := &packageOwnershipRecord{ + SchemaVersion: packageOwnershipSchemaVersion, + Package: strings.Split(packageSource, "@")[0], + Source: packageSource, + ResolvedCommit: packageSourceRef(packageSource), + Installer: "gh-aw " + GetVersion(), + } + for _, resolved := range workflows { + destination, err := packageManagedDestination(resolved, opts) + if err != nil { + return nil, err + } + digest, err := fileSHA256(filepath.Join(gitRoot, filepath.FromSlash(destination))) + if err != nil { + return nil, err + } + record.Files = append(record.Files, packageOwnershipFileEntry{ + Source: filepath.ToSlash(resolved.Spec.WorkflowPath), + Destination: destination, + SHA256: digest, + }) + } + slices.SortFunc(record.Files, func(a, b packageOwnershipFileEntry) int { + return strings.Compare(a.Destination, b.Destination) + }) + return record, nil +} + +func packageManagedDestination(resolved *ResolvedWorkflow, opts AddOptions) (string, error) { + spec := resolved.Spec + switch { + case resolved.IsPackageResourceFile: + return filepath.ToSlash(filepath.Clean(filepath.FromSlash(spec.DestinationPath))), nil + case resolved.IsPackageSkillFile: + relPath, err := resolveSkillRelativePath(resolved) + if err != nil { + return "", err + } + return filepath.ToSlash(filepath.Join(workflow.GetEngineSkillDir(opts.EngineOverride), resolved.SkillName, relPath)), nil + case resolved.IsPackageAgentFile: + return filepath.ToSlash(filepath.Join(workflow.GetEngineSubAgentDir(opts.EngineOverride), filepath.Base(spec.WorkflowPath))), nil + case resolved.IsActionWorkflow: + return filepath.ToSlash(filepath.Join(packageOwnershipWorkflowDir(opts), spec.WorkflowName+".yml")), nil + default: + return filepath.ToSlash(filepath.Join(packageOwnershipWorkflowDir(opts), spec.WorkflowName+".md")), nil + } +} + +func packageOwnershipWorkflowDir(opts AddOptions) string { + if opts.WorkflowDir != "" { + return filepath.Clean(opts.WorkflowDir) + } + return constants.GetWorkflowDir() +} + +func packageSourceRef(source string) string { + if _, ref, ok := strings.Cut(source, "@"); ok { + return ref + } + return "" +} + +func packageOwnershipRecordPath(gitRoot, packageSource string) string { + return filepath.Join(gitRoot, ".github", "aw", "packages", stablePackageID(packageSource)+".json") +} + +func stablePackageID(packageSource string) string { + base := strings.Split(packageSource, "@")[0] + replacer := strings.NewReplacer("/", "-", "\\", "-", ":", "-", "@", "-") + slug := strings.Trim(replacer.Replace(strings.ToLower(base)), "-") + sum := sha256.Sum256([]byte(base)) + return fmt.Sprintf("%s-%s", slug, hex.EncodeToString(sum[:])[:12]) +} + +func fileSHA256(path string) (string, error) { + content, err := os.ReadFile(path) + if err != nil { + return "", fmt.Errorf("failed to read %s for digest: %w", path, err) + } + sum := sha256.Sum256(content) + return hex.EncodeToString(sum[:]), nil +} + +func packageOwnershipAllowsOverwrite(gitRoot, destination, packageSource string) (owned bool, drifted bool) { + records, err := readPackageOwnershipRecords(gitRoot) + if err != nil { + return false, false + } + normalized := filepath.ToSlash(filepath.Clean(destination)) + packageID := strings.Split(packageSource, "@")[0] + for _, record := range records { + if record.Package != packageID { + continue + } + for _, file := range record.Files { + if !strings.EqualFold(filepath.ToSlash(filepath.Clean(file.Destination)), normalized) { + continue + } + current, err := fileSHA256(filepath.Join(gitRoot, filepath.FromSlash(file.Destination))) + if err != nil { + return true, true + } + return true, current != file.SHA256 + } + } + return false, false +} + +func readPackageOwnershipRecords(gitRoot string) ([]packageOwnershipRecord, error) { + dir := filepath.Join(gitRoot, ".github", "aw", "packages") + entries, err := os.ReadDir(dir) + if err != nil { + if os.IsNotExist(err) { + return nil, nil + } + return nil, err + } + var records []packageOwnershipRecord + for _, entry := range entries { + if entry.IsDir() || !strings.HasSuffix(strings.ToLower(entry.Name()), ".json") { + continue + } + data, err := os.ReadFile(filepath.Join(dir, entry.Name())) + if err != nil { + continue + } + var record packageOwnershipRecord + if err := json.Unmarshal(data, &record); err != nil { + continue + } + records = append(records, record) + } + return records, nil +} + +func syncManifestManagedResources(ctx context.Context, repoSpec *RepoSpec, pkg *resolvedRepositoryPackage, ref string, opts UpdateWorkflowsOptions) error { + if pkg == nil || repoSpec == nil { + return nil + } + gitRoot, err := gitutil.FindGitRoot() + if err != nil { + return fmt.Errorf("failed to find git root for package resources: %w", err) + } + owner, repo, err := splitRepositoryPackageSlug(repoSpec.RepoSlug) + if err != nil { + return err + } + packageBase := repositoryPackageIdentifier(repoSpec.RepoSlug, repoSpec.PackagePath) + recordPath := packageOwnershipRecordPath(gitRoot, packageBase) + record := packageOwnershipRecord{ + SchemaVersion: packageOwnershipSchemaVersion, + Package: packageBase, + Source: manifestSourceWithRef(repoSpec, ref), + ResolvedCommit: ref, + Installer: "gh-aw " + GetVersion(), + } + if existing, err := readPackageOwnershipRecord(recordPath); err == nil && existing != nil { + record.Files = existing.Files + } + existingFiles := append([]packageOwnershipFileEntry(nil), record.Files...) + + desired := make(map[string]resolvedPackageResource, len(pkg.ResourceFiles)) + for _, resource := range pkg.ResourceFiles { + desired[filepath.ToSlash(filepath.Clean(resource.DestinationPath))] = resource + } + + type downloadedPackageResource struct { + resource resolvedPackageResource + destination string + destPath string + content []byte + sha256 string + } + var downloads []downloadedPackageResource + + for _, resource := range pkg.ResourceFiles { + destination := filepath.ToSlash(filepath.Clean(resource.DestinationPath)) + destPath := filepath.Join(gitRoot, filepath.FromSlash(destination)) + if fileutil.FileExists(destPath) && !opts.Force { + if owned, drifted := packageOwnershipAllowsOverwrite(gitRoot, destination, packageBase); !owned || drifted { + if owned { + return fmt.Errorf("resource %q has local modifications; use --force to overwrite", destination) + } + return fmt.Errorf("resource %q already exists; use --force to overwrite", destination) + } + } + content, err := downloadPackageFileFromGitHubForHost(ctx, owner, repo, resource.SourcePath, ref, "") + if err != nil { + return fmt.Errorf("failed to download package resource %s: %w", resource.SourcePath, err) + } + downloads = append(downloads, downloadedPackageResource{ + resource: resource, + destination: destination, + destPath: destPath, + content: content, + sha256: sha256Bytes(content), + }) + } + + type fileRollback struct { + path string + existed bool + content []byte + mode os.FileMode + } + var rollbacks []fileRollback + rollbackChanges := func() { + for _, rollback := range slices.Backward(rollbacks) { + if rollback.existed { + _ = os.MkdirAll(filepath.Dir(rollback.path), constants.DirPermPublic) + _ = os.WriteFile(rollback.path, rollback.content, rollback.mode) + continue + } + _ = os.Remove(rollback.path) + } + } + + for _, download := range downloads { + rollback := fileRollback{path: download.destPath} + if stat, err := os.Stat(download.destPath); err == nil { + rollback.existed = true + rollback.mode = stat.Mode().Perm() + content, readErr := os.ReadFile(download.destPath) + if readErr != nil { + rollbackChanges() + return fmt.Errorf("failed to read existing package resource %s: %w", download.destination, readErr) + } + rollback.content = content + } + if err := os.MkdirAll(filepath.Dir(download.destPath), constants.DirPermPublic); err != nil { + rollbackChanges() + return fmt.Errorf("failed to create package resource directory: %w", err) + } + if err := os.WriteFile(download.destPath, download.content, constants.FilePermPublic); err != nil { + rollbackChanges() + return fmt.Errorf("failed to write package resource %s: %w", download.destination, err) + } + rollbacks = append(rollbacks, rollback) + } + + var retained []packageOwnershipFileEntry + var staleRemoved []string + for _, entry := range existingFiles { + destination := filepath.ToSlash(filepath.Clean(entry.Destination)) + if !isPackageResourceDestination(destination) { + retained = append(retained, entry) + continue + } + if _, stillDesired := desired[destination]; stillDesired { + continue + } + + path := filepath.Join(gitRoot, filepath.FromSlash(destination)) + current, digestErr := fileSHA256(path) + if digestErr != nil || current != entry.SHA256 { + retained = append(retained, entry) + continue + } + + rollback := fileRollback{path: path} + if stat, err := os.Stat(path); err == nil { + rollback.existed = true + rollback.mode = stat.Mode().Perm() + content, readErr := os.ReadFile(path) + if readErr != nil { + rollbackChanges() + return fmt.Errorf("failed to read stale package resource %s before removal: %w", destination, readErr) + } + rollback.content = content + } + rollbacks = append(rollbacks, rollback) + if err := os.Remove(path); err != nil && !os.IsNotExist(err) { + rollbackChanges() + return fmt.Errorf("failed to remove stale package resource %s: %w", destination, err) + } + staleRemoved = append(staleRemoved, destination) + } + + record.Files = retained + for _, download := range downloads { + record.Files = upsertPackageOwnershipFile(record.Files, packageOwnershipFileEntry{ + Source: download.resource.SourcePath, + Destination: download.destination, + SHA256: download.sha256, + }) + } + for _, destination := range staleRemoved { + fmt.Fprintln(os.Stderr, console.FormatInfoMessage("Removed stale package resource: "+destination)) + } + if len(record.Files) == 0 { + return nil + } + slices.SortFunc(record.Files, func(a, b packageOwnershipFileEntry) int { + return strings.Compare(a.Destination, b.Destination) + }) + if err := os.MkdirAll(filepath.Dir(recordPath), constants.DirPermPublic); err != nil { + return fmt.Errorf("failed to create package ownership directory: %w", err) + } + data, err := json.MarshalIndent(record, "", " ") + if err != nil { + return fmt.Errorf("failed to encode package ownership record: %w", err) + } + data = append(data, '\n') + if err := os.WriteFile(recordPath, data, constants.FilePermPublic); err != nil { + return fmt.Errorf("failed to write package ownership record: %w", err) + } + return nil +} + +func readPackageOwnershipRecord(path string) (*packageOwnershipRecord, error) { + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var record packageOwnershipRecord + if err := json.Unmarshal(data, &record); err != nil { + return nil, err + } + return &record, nil +} + +func upsertPackageOwnershipFile(entries []packageOwnershipFileEntry, next packageOwnershipFileEntry) []packageOwnershipFileEntry { + for i := range entries { + if strings.EqualFold(filepath.ToSlash(filepath.Clean(entries[i].Destination)), filepath.ToSlash(filepath.Clean(next.Destination))) { + entries[i] = next + return entries + } + } + return append(entries, next) +} + +func isPackageResourceDestination(destination string) bool { + return strings.EqualFold(destination, constants.GithubDir+"CODEOWNERS") || + strings.HasPrefix(destination, constants.GithubDir+"ISSUE_TEMPLATE/") || + strings.HasPrefix(destination, constants.GithubDir+"aw/") +} + +func removePackageOwnedFilesIfUnused(packageBase string) error { + gitRoot, err := gitutil.FindGitRoot() + if err != nil { + return err + } + if packageBase == "" || packageHasRemainingWorkflows(gitRoot, packageBase) { + return nil + } + recordPath := packageOwnershipRecordPath(gitRoot, packageBase) + record, err := readPackageOwnershipRecord(recordPath) + if err != nil { + if os.IsNotExist(err) { + return nil + } + return err + } + var kept []packageOwnershipFileEntry + for _, entry := range record.Files { + destination := filepath.ToSlash(filepath.Clean(entry.Destination)) + path := filepath.Join(gitRoot, filepath.FromSlash(destination)) + if _, statErr := os.Stat(path); os.IsNotExist(statErr) && isMarkdownOwnedWorkflowDestination(destination) { + continue + } + current, digestErr := fileSHA256(path) + if digestErr == nil && current == entry.SHA256 { + if err := os.Remove(path); err != nil && !os.IsNotExist(err) { + kept = append(kept, entry) + continue + } + fmt.Fprintln(os.Stderr, console.FormatSuccessMessage("Removed package-owned file: "+destination)) + continue + } + kept = append(kept, entry) + } + if len(kept) == 0 { + if err := os.Remove(recordPath); err != nil && !os.IsNotExist(err) { + return err + } + return nil + } + record.Files = kept + data, err := json.MarshalIndent(record, "", " ") + if err != nil { + return err + } + data = append(data, '\n') + return os.WriteFile(recordPath, data, constants.FilePermPublic) +} + +func isMarkdownOwnedWorkflowDestination(destination string) bool { + if !strings.HasPrefix(destination, constants.WorkflowsDirSlash) { + return false + } + lower := strings.ToLower(destination) + return strings.HasSuffix(lower, ".md") || strings.HasSuffix(lower, ".lock.yml") +} + +func packageHasRemainingWorkflows(gitRoot, packageBase string) bool { + pattern := filepath.Join(gitRoot, constants.GetWorkflowDir(), "*.md") + files, err := filepath.Glob(pattern) + if err != nil { + return false + } + for _, file := range files { + source := readFullSourceFromFile(file) + repoSpec, ok, err := parseManifestSourceSpec(source) + if err != nil || !ok || repoSpec == nil { + continue + } + if repositoryPackageIdentifier(repoSpec.RepoSlug, repoSpec.PackagePath) == packageBase { + return true + } + } + return false +} diff --git a/pkg/cli/add_workflow_resolution.go b/pkg/cli/add_workflow_resolution.go index 125cda606b0..273b13bf6aa 100644 --- a/pkg/cli/add_workflow_resolution.go +++ b/pkg/cli/add_workflow_resolution.go @@ -50,6 +50,9 @@ type ResolvedWorkflow struct { // IsPackageAgentFile is true when the file is an agent .md from an aw.yml package // manifest. The file is installed as-is to the agentic engine agents folder. IsPackageAgentFile bool + // IsPackageResourceFile is true when the file is a declarative repository resource + // from an aw.yml package manifest. The file is installed as-is to DestinationPath. + IsPackageResourceFile bool // SkillName is the skill directory name for package skill files (e.g. "my-skill"). // Only meaningful when IsPackageSkillFile is true. SkillName string @@ -346,6 +349,17 @@ func resolvePackageOrActionWorkflow(spec, resolvedSpec *WorkflowSpec, fetched *F }, true } + if spec.IsPackageResourceFile { + resolutionLog.Printf("Resolved package resource file: spec=%s, destination=%s, content_size=%d bytes", + spec.String(), spec.DestinationPath, len(fetched.Content)) + return &ResolvedWorkflow{ + Spec: resolvedSpec, + Content: fetched.Content, + SourceInfo: fetched, + IsPackageResourceFile: true, + }, true + } + if isActionWorkflowPath(resolvedSpec.WorkflowPath) { resolutionLog.Printf("Resolved action workflow: spec=%s, content_size=%d bytes", spec.String(), len(fetched.Content)) @@ -482,6 +496,10 @@ func resolveLocalRepositoryPackage(source string) (*resolvedRepositoryPackage, e if err := validateUniqueManifestInstallDestinations(installationSources, manifestPath); err != nil { return nil, err } + resourceFiles, err := normalizeLocalPackageResourcePaths(manifest.Resources, packageDir) + if err != nil { + return nil, err + } skillFiles, skillWarnings, err := resolveLocalPackageSkillFiles(packageDir, append(append([]string{}, manifest.Skills...), includeSkillDirs...)) if err != nil { @@ -495,8 +513,8 @@ func resolveLocalRepositoryPackage(source string) (*resolvedRepositoryPackage, e } warnings = append(warnings, agentWarnings...) - if len(installationSources) == 0 && len(skillFiles) == 0 && len(agentFiles) == 0 { - return nil, fmt.Errorf("repository package at %q does not contain any installable workflows, skills, or agents (either explicitly declared or auto-discovered)", packageDir) + if len(installationSources) == 0 && len(resourceFiles) == 0 && len(skillFiles) == 0 && len(agentFiles) == 0 { + return nil, fmt.Errorf("repository package at %q does not contain any installable workflows, resources, skills, or agents (either explicitly declared or auto-discovered)", packageDir) } return &resolvedRepositoryPackage{ @@ -507,6 +525,7 @@ func resolveLocalRepositoryPackage(source string) (*resolvedRepositoryPackage, e License: manifest.License, DocsPath: filepath.Join(packageDir, "README.md"), InstallationSource: installationSources, + ResourceFiles: resourceFiles, Bootstrap: manifest.Bootstrap, SkillFiles: skillFiles, AgentFiles: agentFiles, @@ -612,6 +631,15 @@ func appendLocalRepositoryPackageWorkflowSpecs(parsedSpecs []*WorkflowSpec, pkg FromRepositoryManifest: true, }) } + for _, resource := range pkg.ResourceFiles { + parsedSpecs = append(parsedSpecs, &WorkflowSpec{ + WorkflowPath: resource.SourcePath, + WorkflowName: packageResourceName(resource), + DestinationPath: resource.DestinationPath, + FromRepositoryManifest: true, + IsPackageResourceFile: true, + }) + } for _, skillFile := range pkg.SkillFiles { base := filepath.Base(skillFile.SourcePath) workflowName := skillFile.SkillName + "/" + strings.TrimSuffix(base, filepath.Ext(base)) @@ -778,6 +806,21 @@ func appendRepositoryPackageWorkflowSpecs(parsedSpecs []*WorkflowSpec, repoSpec FromRepositoryManifest: true, }) } + for _, resource := range pkg.ResourceFiles { + parsedSpecs = append(parsedSpecs, &WorkflowSpec{ + RepoSpec: RepoSpec{ + RepoSlug: repoSpec.RepoSlug, + Version: effectiveVersion, + PackagePath: repoSpec.PackagePath, + }, + WorkflowPath: resource.SourcePath, + WorkflowName: packageResourceName(resource), + DestinationPath: resource.DestinationPath, + Host: host, + FromRepositoryManifest: true, + IsPackageResourceFile: true, + }) + } // Append skill file specs. Each spec carries IsPackageSkillFile=true and the SkillName // so that the installation step can route the file to the correct skill directory. diff --git a/pkg/cli/bootstrap_profile_helpers.go b/pkg/cli/bootstrap_profile_helpers.go index 7bb6e0d8a4e..5f8085afcc7 100644 --- a/pkg/cli/bootstrap_profile_helpers.go +++ b/pkg/cli/bootstrap_profile_helpers.go @@ -177,7 +177,7 @@ func profileSourcesUseActionsTokenCopilotAuth(ctx context.Context, sources []str } hasCopilot := false for _, candidate := range resolved.Workflows { - if candidate == nil || candidate.IsActionWorkflow || candidate.IsPackageSkillFile || candidate.IsPackageAgentFile { + if candidate == nil || candidate.IsActionWorkflow || candidate.IsPackageSkillFile || candidate.IsPackageAgentFile || candidate.IsPackageResourceFile { continue } engine := strings.TrimSpace(candidate.Engine) diff --git a/pkg/cli/bootstrap_profile_inference.go b/pkg/cli/bootstrap_profile_inference.go index ad20dc99c88..f001744951d 100644 --- a/pkg/cli/bootstrap_profile_inference.go +++ b/pkg/cli/bootstrap_profile_inference.go @@ -35,7 +35,7 @@ func inferBootstrapGitHubAppRequirements(ctx context.Context, sources []string) permissions := map[string]string{} eventSet := map[string]struct{}{} for _, candidate := range resolved.Workflows { - if candidate == nil || candidate.IsActionWorkflow || candidate.IsPackageSkillFile || candidate.IsPackageAgentFile { + if candidate == nil || candidate.IsActionWorkflow || candidate.IsPackageSkillFile || candidate.IsPackageAgentFile || candidate.IsPackageResourceFile { continue } frontmatter, err := parser.ExtractFrontmatterFromContent(string(candidate.Content)) diff --git a/pkg/cli/remove_command.go b/pkg/cli/remove_command.go index 8c0263eff27..f99ce87208a 100644 --- a/pkg/cli/remove_command.go +++ b/pkg/cli/remove_command.go @@ -137,7 +137,13 @@ func RemoveWorkflows(pattern string, keepOrphans bool, workflowDir string) error // Remove the files var removedFiles []string + removedPackageSources := make(map[string]struct{}) for _, file := range filesToRemove { + if source := readFullSourceFromFile(file); source != "" { + if repoSpec, ok, err := parseManifestSourceSpec(source); err == nil && ok && repoSpec != nil { + removedPackageSources[repositoryPackageIdentifier(repoSpec.RepoSlug, repoSpec.PackagePath)] = struct{}{} + } + } if err := os.Remove(file); err != nil { fmt.Fprintln(os.Stderr, console.FormatWarningMessage(fmt.Sprintf("Failed to remove %s: %v", file, err))) } else { @@ -162,6 +168,11 @@ func RemoveWorkflows(pattern string, keepOrphans bool, workflowDir string) error fmt.Fprintln(os.Stderr, console.FormatWarningMessage(fmt.Sprintf("Failed to clean up orphaned includes: %v", err))) } } + for packageSource := range removedPackageSources { + if err := removePackageOwnedFilesIfUnused(packageSource); err != nil { + fmt.Fprintln(os.Stderr, console.FormatWarningMessage(fmt.Sprintf("Failed to remove package-owned files for %s: %v", packageSource, err))) + } + } // Stage changes to git if in a git repository if len(removedFiles) > 0 && isGitRepo() { diff --git a/pkg/cli/spec.go b/pkg/cli/spec.go index 49775ef7045..92543c1170b 100644 --- a/pkg/cli/spec.go +++ b/pkg/cli/spec.go @@ -50,6 +50,9 @@ type WorkflowSpec struct { // IsPackageAgentFile is true when this spec refers to an agent .md file from an aw.yml // package manifest. The file is installed as-is to the agentic engine agents folder. IsPackageAgentFile bool + // IsPackageResourceFile is true when this spec refers to a declarative repository + // resource from an aw.yml package manifest. The file is installed as-is to DestinationPath. + IsPackageResourceFile bool // DestinationPath is the repository-root-relative install path resolved from a package // manifest entry (e.g. ".github/workflows/reviewer.md"). WorkflowName is derived from // it, so installation writes the file under the declared destination name. diff --git a/pkg/cli/update_manifest.go b/pkg/cli/update_manifest.go index d7ed47c9b58..4db9bd18816 100644 --- a/pkg/cli/update_manifest.go +++ b/pkg/cli/update_manifest.go @@ -77,6 +77,7 @@ func updateManifestWorkflowGroup(ctx context.Context, source string, grouped []* updateManifestLog.Printf("updateManifestWorkflowGroup: source=%s, workflows=%d, force=%v, no_merge=%v", source, len(grouped), opts.Force, opts.NoMerge) var successes []string var failures []updateFailure + var groupedSuccesses []string if len(grouped) == 0 { return successes, failures @@ -157,7 +158,7 @@ func updateManifestWorkflowGroup(ctx context.Context, source string, grouped []* failures = append(failures, updateFailure{Name: wf.Name, Error: err.Error()}) continue } - successes = append(successes, wf.Name) + groupedSuccesses = append(groupedSuccesses, wf.Name) continue } @@ -178,7 +179,7 @@ func updateManifestWorkflowGroup(ctx context.Context, source string, grouped []* failures = append(failures, updateFailure{Name: wf.Name, Error: err.Error()}) continue } - successes = append(successes, wf.Name) + groupedSuccesses = append(groupedSuccesses, wf.Name) } targetDir := filepath.Dir(grouped[0].Path) @@ -190,13 +191,20 @@ func updateManifestWorkflowGroup(ctx context.Context, source string, grouped []* failures = append(failures, updateFailure{Name: name, Error: err.Error()}) continue } - successes = append(successes, name) + groupedSuccesses = append(groupedSuccesses, name) } + if err := syncManifestManagedResources(ctx, repoSpec, latestPkg, latestRef, opts); err != nil { + for _, name := range groupedSuccesses { + failures = append(failures, updateFailure{Name: name, Error: err.Error()}) + } + return successes, failures + } assetEngine := resolveManifestAssetEngine(grouped, opts) if err := reconcileManifestManagedAssets(ctx, repoSpec.RepoSlug, currentPkg, latestPkg, assetEngine); err != nil { failures = append(failures, updateFailure{Name: source, Error: err.Error()}) } + successes = append(successes, groupedSuccesses...) return successes, failures } diff --git a/pkg/parser/schemas/aw_manifest_schema.json b/pkg/parser/schemas/aw_manifest_schema.json index e133211d35a..50b0fae50b1 100644 --- a/pkg/parser/schemas/aw_manifest_schema.json +++ b/pkg/parser/schemas/aw_manifest_schema.json @@ -81,6 +81,25 @@ "type": "string" } }, + "resources": { + "type": "array", + "description": "Declarative repository assets installed as-is from immutable package content. Each entry maps a package-relative source to an allowlisted repository destination.", + "items": { + "type": "object", + "required": ["source", "destination"], + "additionalProperties": false, + "properties": { + "source": { + "type": "string", + "minLength": 1 + }, + "destination": { + "type": "string", + "minLength": 1 + } + } + } + }, "branding": { "type": "object", "description": "Follows GitHub Actions marketplace branding. See https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions#branding",