diff --git a/docs/setup/agentic-overrides.md b/docs/setup/agentic-overrides.md index c99fc57aa..ccbb51c89 100644 --- a/docs/setup/agentic-overrides.md +++ b/docs/setup/agentic-overrides.md @@ -6,7 +6,8 @@ **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* - [Agentic overrides — modifying framework workflows in an adopter](#agentic-overrides--modifying-framework-workflows-in-an-adopter) - - [Where override files live](#where-override-files-live) + - [Override surfaces — two directories, one lookup chain](#override-surfaces--two-directories-one-lookup-chain) + - [Adoption and `.gitignore`](#adoption-and-gitignore) - [What an override file may contain](#what-an-override-file-may-contain) - [Skip a step](#skip-a-step) - [Replace a step](#replace-a-step) @@ -40,19 +41,52 @@ run-time and applies before executing default behaviour. This document is the **contract** between adopter authors of override files and framework authors of skills that read them. -## Where override files live +## Override surfaces — two directories, one lookup chain + +Framework skills consult **two** override directories in precedence +order, first hit wins: + +1. **`.apache-magpie-local/.md`** — personal, gitignored. + Per-developer preferences (local clone paths, a release manager + enabling an extra MCP, wording adjustments) that should not be + committed to the shared repo. Never committed; never pushed. + Works on a repo that has not yet adopted Magpie — the user adds + one `.gitignore` line by hand so the directory stays untracked, + then drops their overrides there. The same additive-only + guardrail applies: it cannot weaken the safety, confidentiality, + or privacy baseline. + +2. **`.apache-magpie-overrides/.md`** — committed, + project-wide. Shared modifications every contributor on the + project sees (custom steps, project-specific defaults, integrations + with project tooling). ```text -/.apache-magpie-overrides/ -├── README.md (the dir's own readme, -│ scaffolded by -│ /magpie-setup adopt) -├── .md (e.g. pr-management-triage.md) -└── .md +/ +├── .apache-magpie-local/ (gitignored, per-person) +│ └── .md (e.g. pr-management-triage.md) +├── .apache-magpie-overrides/ (committed, project-wide) +│ ├── README.md (scaffolded by /magpie-setup adopt) +│ ├── .md +│ └── .md ``` -The directory is **committed** in the adopter repo (the whole -point is for overrides to ship with the project's repo). +When both directories contain a file for the same skill, the +personal-local file wins — its instructions are applied first, +then (by default) the committed file's instructions are also applied +unless the personal file explicitly says to skip it. Neither file +is required to exist; a skill that finds neither proceeds with +framework defaults. + +### Adoption and `.gitignore` + +`/magpie-setup adopt` adds `/.apache-magpie-local/` to the adopter +repo's `.gitignore` automatically. On a repo that has not adopted +Magpie, add the line manually: + +```text +/.apache-magpie-local/ +``` ## What an override file may contain @@ -128,17 +162,24 @@ maintainer (or a future agent on a later run): Every framework skill that supports overrides starts each invocation with this opening protocol: -1. Read `/.apache-magpie-overrides/.md` - if it exists. Surface the file's title and the list of - override headlines (`### Override N — ...`) to the user - before doing anything else. -2. Apply the overrides: each `### Override N — ...` section - modifies the skill's default behaviour for this run. The - agent interprets the instructions in the override section - and adjusts the rest of the skill's flow accordingly. -3. After the skill finishes, recap which overrides were - applied (and any the agent decided not to apply with the - reasoning), so the user has an audit trail. +1. Read `/.apache-magpie-local/.md` + (personal, gitignored) if it exists. +2. Read `/.apache-magpie-overrides/.md` + (committed, project-wide) if it exists. +3. Surface the titles and override headlines + (`### Override N — ...`) from both files to the user + before doing anything else. Indicate which file each + came from so the user can tell personal from shared + overrides at a glance. +4. Apply both sets of overrides: personal-local first, then + committed. Each `### Override N — ...` section modifies + the skill's default behaviour for this run. The agent + interprets the instructions and adjusts the rest of the + skill's flow accordingly. +5. After the skill finishes, recap which overrides were + applied (source file + override headline), and any the + agent decided not to apply with the reasoning, so the user + has an audit trail. A skill that does **not** yet support overrides documents that explicitly in its `SKILL.md`. The @@ -155,7 +196,12 @@ A framework agent NEVER: `/.apache-magpie/`. The snapshot is a build artefact — every modification gets blown away on the next `/magpie-setup upgrade`. Local mods go into - `.apache-magpie-overrides/`. + `.apache-magpie-local/` (personal) or + `.apache-magpie-overrides/` (shared). +- Commits or pushes `.apache-magpie-local/` content. The + personal override directory is gitignored by design — it + carries per-person paths, credentials, and capability + enablements the contributor has not chosen to share. - Proposes overrides be merged in by editing the framework source in the snapshot. Framework changes go via PR to `apache/magpie`. @@ -165,6 +211,9 @@ A framework agent NEVER: human decide (the override expresses adopter intent — re-anchoring it correctly is human judgement, not pattern-matching). +- Weakens the safety, confidentiality, or privacy baseline + from either override surface. An override that attempts + to do so is ignored and the conflict is surfaced. ## Reconciliation on framework upgrade @@ -246,6 +295,13 @@ this by: ## Cross-references -- [`setup` skill](../../skills/setup/SKILL.md) — the entry point that manages the snapshot + scaffolds overrides. -- [`overrides.md` sub-action](../../skills/setup/overrides.md) — interactive override creation. +- [`setup` skill](../../skills/setup/SKILL.md) — the entry point + that manages the snapshot, scaffolds overrides, and adds the + `.gitignore` entries for both override directories. +- [`overrides.md` sub-action](../../skills/setup/overrides.md) — + interactive override creation (lets the user choose between the + personal-local and committed surfaces). - [Top-level README](../../README.md) — adoption flow. +- [`setup-status` skill](../../skills/setup-status/SKILL.md) — + the adoption dashboard, which reports whether both override + directories are present. diff --git a/skills/setup-status/SKILL.md b/skills/setup-status/SKILL.md index 0397cb596..d7d710804 100644 --- a/skills/setup-status/SKILL.md +++ b/skills/setup-status/SKILL.md @@ -59,12 +59,15 @@ it rather than duplicating its checks. ## Adopter overrides Before running the default behaviour documented below, this skill -consults -[`.apache-magpie-overrides/setup-status.md`](../../docs/setup/agentic-overrides.md) -in the adopter repo if it exists, and applies any agent-readable -overrides it finds. See +consults (in order, first hit wins): + +1. `.apache-magpie-local/setup-status.md` — personal, gitignored. +2. `.apache-magpie-overrides/setup-status.md` — committed, + project-wide. + +Both files are applied if present. See [`docs/setup/agentic-overrides.md`](../../docs/setup/agentic-overrides.md) -for the contract. +for the full lookup contract. **Hard rule**: agents NEVER modify the snapshot under `/.apache-magpie/`. Local modifications go in the diff --git a/skills/setup-status/collect.md b/skills/setup-status/collect.md index 90b132292..711f58c05 100644 --- a/skills/setup-status/collect.md +++ b/skills/setup-status/collect.md @@ -35,7 +35,8 @@ python3 /skills/setup-status/scripts/collect_status.py --format json | `agent_targets` | One record per registry target (see below). | | `active_target_ids` | The subset of registry ids whose directory is present on disk. | | `families` | The installed-skill roster grouped by family (see below). | -| `overrides` | `{present, has_readme}` for `.apache-magpie-overrides/`. | +| `overrides` | `{present, has_readme, skill_count}` for `.apache-magpie-overrides/` (committed, shared). | +| `local_overrides` | `{present, has_readme, skill_count}` for `.apache-magpie-local/` (gitignored, personal). Always reported; `present: false` when the directory does not exist. | | `post_checkout_hook` | `{present, executable, has_verify_recipe}`. | | `gitignore` | Coverage flags (see below). | @@ -106,7 +107,9 @@ for the *intended* set, and use the on-disk read for the ## `gitignore` Top-level flags (`snapshot_ignored`, `local_lock_ignored`, -`settings_local_ignored`) plus a per-target map. Each target +`local_overrides_ignored`, `settings_local_ignored`) plus a +per-target map. `local_overrides_ignored` is `true` when +`/.apache-magpie-local/` appears in `.gitignore`. Each target carries `glob_ignored` + `setup_unignored` (the **normal-adopter** pattern: ignore the symlinks, keep the bootstrap tracked) and `all_unignored` (the **self-adoption** pattern: every `magpie-*` diff --git a/skills/setup-status/render.md b/skills/setup-status/render.md index aaab263d8..d79b412ea 100644 --- a/skills/setup-status/render.md +++ b/skills/setup-status/render.md @@ -69,7 +69,8 @@ security ✅ 12 · pr-management ✅ 8 · issue ✅ 8 · release-management ✅ ### Drift & integrity - **drift:** n/a (method:local …) · **snapshot:** in-repo source (local) -- **overrides:** — · **hook:** — +- **shared overrides** (`.apache-magpie-overrides/`): — · **personal overrides** (`.apache-magpie-local/`): — +- **hook:** — - → deep check (integrity, permissions, worktrees): `/magpie-setup verify` ``` @@ -107,6 +108,8 @@ this before assigning health: | `gitignore.targets[].all_unignored` | ✅ expected — symlinks are committed | not the pattern used; ignore | | `gitignore.targets[].glob_ignored` + `setup_unignored` | not used | ✅ expected — symlinks gitignored, bootstrap tracked | | `drift.checked == false` | ✅ nothing to drift against | depends on `reason` (see [`collect.md`](collect.md#drift)) | +| `local_overrides.present == false` | ✅ optional personal surface — not required | ✅ same — `.apache-magpie-local/` is always optional | +| `gitignore.local_overrides_ignored == false` | advisory: add `/.apache-magpie-local/` to `.gitignore` | same advisory | Never report a self-adopted framework checkout as unhealthy merely for lacking a snapshot, a local lock, or ignored symlinks — those diff --git a/skills/setup-status/scripts/collect_status.py b/skills/setup-status/scripts/collect_status.py index ec716743f..4e752e9f2 100644 --- a/skills/setup-status/scripts/collect_status.py +++ b/skills/setup-status/scripts/collect_status.py @@ -19,10 +19,11 @@ Read-only. Enumerates the on-disk adoption artefacts — the two lock files, the framework-skill symlinks across every -agent target, the snapshot, the overrides directory, the -post-checkout hook, and the .gitignore coverage — and emits a -single JSON document the ``setup-status`` skill renders into a -dashboard. +agent target, the snapshot, the overrides directories (both the +committed .apache-magpie-overrides/ and the personal +.apache-magpie-local/), the post-checkout hook, and the +.gitignore coverage — and emits a single JSON document the +``setup-status`` skill renders into a dashboard. The script never fetches over the network and never writes: the upstream-tip drift check and any remediation belong to @@ -302,6 +303,7 @@ def gitignore_coverage(root: Path, targets: list[dict]) -> dict: "present": gi.is_file(), "snapshot_ignored": "/.apache-magpie/" in lines, "local_lock_ignored": "/.apache-magpie.local.lock" in lines, + "local_overrides_ignored": "/.apache-magpie-local/" in lines, "settings_local_ignored": "/.claude/settings.local.json" in lines, "targets": {}, } @@ -323,6 +325,22 @@ def gitignore_coverage(root: Path, targets: list[dict]) -> dict: return cov +def override_dir_status(root: Path, dirname: str) -> dict: + """Describe one override directory (committed or personal-local).""" + d = root / dirname + if not d.is_dir(): + return {"present": False, "has_readme": False, "skill_count": 0} + skill_files = [ + p for p in d.iterdir() + if p.is_file() and p.suffix == ".md" and p.name != "README.md" + ] + return { + "present": True, + "has_readme": (d / "README.md").is_file(), + "skill_count": len(skill_files), + } + + def hook_status(root: Path) -> dict: hook = root / ".git" / "hooks" / "post-checkout" if not hook.is_file(): @@ -435,9 +453,20 @@ def render_markdown(d: dict) -> str: out.append("### Drift & integrity") out.append("") out.append(f"- **drift:** {drift_line} · **snapshot:** {snap}") + ov = d["overrides"] + local_ov = d["local_overrides"] + ov_text = f"present ({ov['skill_count']} skill(s))" if ov["present"] else "—" + local_ov_text = ( + f"present ({local_ov['skill_count']} skill(s))" + if local_ov["present"] + else "—" + ) out.append( - f"- **overrides:** {'present' if d['overrides']['present'] else '—'} · " - f"**hook:** {'installed' if d['post_checkout_hook']['present'] else '—'}" + f"- **shared overrides** (`.apache-magpie-overrides/`): {ov_text} · " + f"**personal overrides** (`.apache-magpie-local/`): {local_ov_text}" + ) + out.append( + f"- **hook:** {'installed' if d['post_checkout_hook']['present'] else '—'}" ) out.append("- → deep check (integrity, permissions, worktrees): `/magpie-setup verify`") return "\n".join(out) + "\n" @@ -485,10 +514,8 @@ def main(argv: list[str] | None = None) -> int: "agent_targets": targets, "active_target_ids": [t["id"] for t in targets if t["present"]], "families": families_installed(canonical["entries"]), - "overrides": { - "present": (root / ".apache-magpie-overrides").is_dir(), - "has_readme": (root / ".apache-magpie-overrides" / "README.md").is_file(), - }, + "overrides": override_dir_status(root, ".apache-magpie-overrides"), + "local_overrides": override_dir_status(root, ".apache-magpie-local"), "post_checkout_hook": hook_status(root), "gitignore": gitignore_coverage(root, targets), } diff --git a/skills/setup/SKILL.md b/skills/setup/SKILL.md index d05a21452..619b865ac 100644 --- a/skills/setup/SKILL.md +++ b/skills/setup/SKILL.md @@ -246,6 +246,8 @@ Gitignored in the adopter repo: - `` (the entire framework snapshot — gigabytes potentially). - `` (per-machine state). +- `.apache-magpie-local/` (personal, per-developer override + directory — see Golden rule 7). - The `magpie-*` symlinks `setup adopt` creates in every active target dir — the canonical ones in `.agents/skills/` (they target the gitignored snapshot) and the relays in @@ -298,13 +300,21 @@ silently mis-applies. **Golden rule 7 — agentic overrides are read at run-time.** Every framework skill that supports overrides starts its run -by checking `.apache-magpie-overrides/.md` for -adopter-specific instructions and applying them before -executing the default behaviour. The override file is plain -markdown the agent interprets — no templating engine, no -patch tool. See +by consulting **two** directories in precedence order (first +hit wins): + +1. `.apache-magpie-local/.md` — personal, + gitignored. Per-developer overrides that are never + committed. +2. `.apache-magpie-overrides/.md` — committed, + project-wide. Overrides shared with every contributor. + +Both files are plain markdown the agent interprets — no +templating engine, no patch tool. The additive-only guardrail +applies to both: neither may weaken the framework's safety, +confidentiality, or privacy baseline. See [`docs/setup/agentic-overrides.md`](../../docs/setup/agentic-overrides.md) -for the contract. +for the full contract including the lookup protocol. **Golden rule 8 — family membership is declared in frontmatter; two families are *always* installed, the rest diff --git a/skills/setup/adopt.md b/skills/setup/adopt.md index 72843b782..70f29ae3b 100644 --- a/skills/setup/adopt.md +++ b/skills/setup/adopt.md @@ -556,6 +556,7 @@ idempotent — re-add them if they're missing. ```text /.apache-magpie/ /.apache-magpie.local.lock +/.apache-magpie-local/ /.apache-magpie-sources/ /.apache-magpie.sources.local.lock /.claude/settings.local.json @@ -757,11 +758,27 @@ in the framework for the full contract. **Hard rule**: never modify the snapshot under `/.apache-magpie/`. Local mods go here. Framework changes go via PR to `apache/magpie`. + +**Personal (non-shared) overrides** belong in +`/.apache-magpie-local/` (gitignored). Use that +directory for per-developer paths, local tooling, or role- +specific capability enablements you do not want committed +to this repo. ``` This directory is **committed** (overrides ship with the adopter repo). +Tell the user about the personal override surface: + +> *"`.apache-magpie-local/` is gitignored (already in +> `.gitignore`). Create it at any time for per-person +> overrides that should not be committed — capability +> enablements, local clone paths, wording you want only +> for yourself. The framework reads it before the +> committed `.apache-magpie-overrides/` on every skill +> invocation."* + ## Step 9b — Scaffold `user.md` (FRESH only) Create the operator's per-user configuration file. The security diff --git a/skills/setup/overrides.md b/skills/setup/overrides.md index 33800a843..391426094 100644 --- a/skills/setup/overrides.md +++ b/skills/setup/overrides.md @@ -24,21 +24,57 @@ in the framework. This file is the operational helper. - `` — required. The skill name to scaffold / open the override for (e.g. `pr-management-triage`). +- `--local` — place the override in `.apache-magpie-local/` + (personal, gitignored) instead of `.apache-magpie-overrides/` + (committed, project-wide). Default: prompt. ## Step 0 — Pre-flight -1. The repo must be adopted (see [`verify.md`](verify.md) - check 1 + check 5). If not, redirect to `/magpie-setup - adopt`. -2. The named `` must exist in the snapshot at - `/.apache-magpie/skills//`. - If not, name the typo and list the available framework - skills. +1. The named `` must exist in the snapshot at + `/.apache-magpie/skills//` (or + in the in-repo `skills/` for a self-adopted framework + checkout). If not, name the typo and list the available + framework skills. +2. For the committed surface (`.apache-magpie-overrides/`): the + repo must be adopted (see [`verify.md`](verify.md) check 1 + + check 5). If not, redirect to `/magpie-setup adopt` **or** + suggest using `--local` to create a personal override + without adopting. + +## Step 0b — Choose the override surface + +If `--local` was passed, route to the personal surface: +`` = `/.apache-magpie-local/.md`. + +If `--local` was not passed and the repo is adopted, offer a +choice: + +> *"Where should this override live?* +> +> - **Personal** (`.apache-magpie-local/`, gitignored) — +> only you see it; works even on repos you have not adopted +> Magpie into. Use for per-person paths, local tooling, +> role-specific capabilities. +> - **Shared** (`.apache-magpie-overrides/`, committed) — +> all contributors see it on their next clone or pull. +> Use for project-wide process changes.*" + +If the repo is not adopted and `--local` was not passed: +offer only the personal surface and note that the committed +surface requires adoption first. + +Default to **personal** when the user has not expressed a +preference. ## Step 1 — Resolve the override path -`` = -`/.apache-magpie-overrides/.md`. +Per the surface chosen in Step 0b: +- **Personal:** `` = `/.apache-magpie-local/.md`. +- **Shared:** `` = `/.apache-magpie-overrides/.md`. + +Also check the *other* surface: if a file already exists +there for the same skill, surface its headlines so the user +can decide whether to consolidate. If `` already exists, this is an *open* operation: surface the file's current content, ask the user @@ -70,12 +106,17 @@ Use the canonical scaffold below. ## Override file scaffold +Use the same scaffold for both surfaces. The `Surface:` comment +distinguishes personal from shared overrides. + ```markdown + +The collector emitted this dashboard: + +```markdown +## apache-magpie adoption — myproject + +**mode:** git-branch · **pinned:** main · **verdict:** ✅ healthy + +### Agent targets + +| Target | Dir | Kind | Skills | Status | +|---|---|---|---|---| +| universal | `.agents/skills` | canonical-snapshot | 12 | ✅ wired | +| claude-code | `.claude/skills` | relay | 12 | ✅ wired | + +**serves** (which agents read each target dir): + +- `universal` — Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, Cline, Zed, Warp, Amp, … +- `claude-code` — Claude Code + +### Skill families + +| Family | Type | Installed | +|---|---|---| +| security | opt-in | ✅ 12 | +| pr-management | opt-in | — none | + +### Drift & integrity + +- **drift:** ✅ in sync · **snapshot:** present +- **shared overrides** (`.apache-magpie-overrides/`): present (1 skill(s)) · **personal overrides** (`.apache-magpie-local/`): — +- **hook:** installed +- → deep check (integrity, permissions, worktrees): `/magpie-setup verify` +``` diff --git a/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-2-local-present/expected.json b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-2-local-present/expected.json new file mode 100644 index 000000000..c87c08c81 --- /dev/null +++ b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-2-local-present/expected.json @@ -0,0 +1 @@ +{"shared_overrides_present": false, "local_overrides_present": true, "local_overrides_skill_count": 3, "presentation_mode": "verbatim"} diff --git a/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-2-local-present/report.md b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-2-local-present/report.md new file mode 100644 index 000000000..ecf40d778 --- /dev/null +++ b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-2-local-present/report.md @@ -0,0 +1,33 @@ + + +The collector emitted this dashboard: + +```markdown +## apache-magpie adoption — myproject + +**mode:** git-branch · **pinned:** main · **verdict:** ✅ healthy + +### Agent targets + +| Target | Dir | Kind | Skills | Status | +|---|---|---|---|---| +| universal | `.agents/skills` | canonical-snapshot | 12 | ✅ wired | + +**serves** (which agents read each target dir): + +- `universal` — Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, Cline, Zed, Warp, Amp, … + +### Skill families + +| Family | Type | Installed | +|---|---|---| +| security | opt-in | ✅ 12 | + +### Drift & integrity + +- **drift:** ✅ in sync · **snapshot:** present +- **shared overrides** (`.apache-magpie-overrides/`): — · **personal overrides** (`.apache-magpie-local/`): present (3 skill(s)) +- **hook:** installed +- → deep check (integrity, permissions, worktrees): `/magpie-setup verify` +``` diff --git a/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-3-both-surfaces/expected.json b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-3-both-surfaces/expected.json new file mode 100644 index 000000000..54e3ac72c --- /dev/null +++ b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-3-both-surfaces/expected.json @@ -0,0 +1 @@ +{"shared_overrides_present": true, "local_overrides_present": true, "local_overrides_skill_count": 1, "presentation_mode": "verbatim"} diff --git a/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-3-both-surfaces/report.md b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-3-both-surfaces/report.md new file mode 100644 index 000000000..582d25563 --- /dev/null +++ b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-3-both-surfaces/report.md @@ -0,0 +1,33 @@ + + +The collector emitted this dashboard: + +```markdown +## apache-magpie adoption — myproject + +**mode:** git-branch · **pinned:** main · **verdict:** ✅ healthy + +### Agent targets + +| Target | Dir | Kind | Skills | Status | +|---|---|---|---|---| +| universal | `.agents/skills` | canonical-snapshot | 12 | ✅ wired | + +**serves** (which agents read each target dir): + +- `universal` — Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, Cline, Zed, Warp, Amp, … + +### Skill families + +| Family | Type | Installed | +|---|---|---| +| security | opt-in | ✅ 12 | + +### Drift & integrity + +- **drift:** ✅ in sync · **snapshot:** present +- **shared overrides** (`.apache-magpie-overrides/`): present (2 skill(s)) · **personal overrides** (`.apache-magpie-local/`): present (1 skill(s)) +- **hook:** installed +- → deep check (integrity, permissions, worktrees): `/magpie-setup verify` +``` diff --git a/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-4-gitignore-missing/expected.json b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-4-gitignore-missing/expected.json new file mode 100644 index 000000000..e5f99150d --- /dev/null +++ b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-4-gitignore-missing/expected.json @@ -0,0 +1 @@ +{"shared_overrides_present": true, "local_overrides_present": true, "local_overrides_skill_count": 2, "presentation_mode": "verbatim"} diff --git a/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-4-gitignore-missing/report.md b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-4-gitignore-missing/report.md new file mode 100644 index 000000000..e82b21d3a --- /dev/null +++ b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/case-4-gitignore-missing/report.md @@ -0,0 +1,36 @@ + + +The collector emitted this dashboard: + +```markdown +## apache-magpie adoption — myproject + +**mode:** git-branch · **pinned:** main · **verdict:** ✅ healthy + +### Agent targets + +| Target | Dir | Kind | Skills | Status | +|---|---|---|---|---| +| universal | `.agents/skills` | canonical-snapshot | 12 | ✅ wired | + +**serves** (which agents read each target dir): + +- `universal` — Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, Cline, Zed, Warp, Amp, … + +### Skill families + +| Family | Type | Installed | +|---|---|---| +| security | opt-in | ✅ 12 | + +### Drift & integrity + +- **drift:** ✅ in sync · **snapshot:** present +- **shared overrides** (`.apache-magpie-overrides/`): present (1 skill(s)) · **personal overrides** (`.apache-magpie-local/`): present (2 skill(s)) +- **hook:** installed +- → deep check (integrity, permissions, worktrees): `/magpie-setup verify` +``` + +The raw JSON also showed: `gitignore.local_overrides_ignored: false` +The `.apache-magpie-local/` directory is present on disk but `/.apache-magpie-local/` is not in `.gitignore`. diff --git a/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/output-spec.md b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/output-spec.md new file mode 100644 index 000000000..13f1033cb --- /dev/null +++ b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/output-spec.md @@ -0,0 +1,22 @@ + + +## Output format + +Return ONLY valid JSON with this structure: + +```json +{ + "shared_overrides_present": true | false, + "local_overrides_present": true | false, + "local_overrides_skill_count": , + "presentation_mode": "verbatim" | "paraphrase" +} +``` + +`shared_overrides_present` is `true` when `.apache-magpie-overrides/` is reported as present. +`local_overrides_present` is `true` when `.apache-magpie-local/` is reported as present. +`local_overrides_skill_count` is the number of skill override files in `.apache-magpie-local/` (0 when absent). +`presentation_mode` is `"verbatim"` when the script output is presented as-is; `"paraphrase"` when the agent would reformat or summarise it. +The correct answer for `presentation_mode` is always `"verbatim"` — the script owns the rendering. +Do not include any text outside the JSON object. diff --git a/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/step-config.json b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/step-config.json new file mode 100644 index 000000000..e06cc5cfb --- /dev/null +++ b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/step-config.json @@ -0,0 +1,4 @@ +{ + "skill_md": "skills/setup-status/SKILL.md", + "step_heading": "## Step 1 — Render the dashboard" +} diff --git a/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/user-prompt-template.md b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/user-prompt-template.md new file mode 100644 index 000000000..f977445f4 --- /dev/null +++ b/tools/skill-evals/evals/setup-status/step-4-local-overrides/fixtures/user-prompt-template.md @@ -0,0 +1,8 @@ + + +## Collector script output + +{report} + +You are at Step 1 of setup-status (render the dashboard). Based on the collector output, identify which override surfaces are present. Return JSON only. diff --git a/tools/skill-evals/evals/setup/README.md b/tools/skill-evals/evals/setup/README.md index de98a4b6f..6c76cf8ff 100644 --- a/tools/skill-evals/evals/setup/README.md +++ b/tools/skill-evals/evals/setup/README.md @@ -5,11 +5,12 @@ Behavioral evals for the `setup` skill. -## Suites (5 cases total) +## Suites (9 cases total) | Suite | Step | Cases | What it covers | |---|---|---|---| | step-verify-drift | verify.md § Check 3 (drift) | 5 | clean, method/URL mismatch, ref mismatch, svn-zip SHA-512 mismatch, local lock missing | +| step-overrides-surface | overrides.md § Step 0b | 4 | adopted no flag (offer choice), --local flag (personal), not adopted (personal only), both surfaces exist | ## Run @@ -31,3 +32,7 @@ uv run --project tools/skill-evals skill-eval \ - `step-verify-drift` cases are fully auto-comparable: all three output fields (`status`, `severity`, `remediation`) are enumerated strings. +- `step-overrides-surface` tests the new `--local` flag and personal- + vs-shared surface selection introduced by the `magpie-local-convention` + work item. The default surface when the repo is adopted and no flag is + passed is `"offer-choice"`; `override_path` reports the personal default. diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-1-adopted-no-flag/expected.json b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-1-adopted-no-flag/expected.json new file mode 100644 index 000000000..42df96293 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-1-adopted-no-flag/expected.json @@ -0,0 +1 @@ +{"surface": "offer-choice", "override_path": ".apache-magpie-local/pr-management-triage.md", "requires_adoption": false} diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-1-adopted-no-flag/report.md b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-1-adopted-no-flag/report.md new file mode 100644 index 000000000..a76632ba4 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-1-adopted-no-flag/report.md @@ -0,0 +1,7 @@ + + +Invocation: `/magpie-setup override pr-management-triage` +Flags: none +Repo is adopted (`.apache-magpie.lock` exists). +Neither `.apache-magpie-local/pr-management-triage.md` nor `.apache-magpie-overrides/pr-management-triage.md` exists yet. diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-2-local-flag/expected.json b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-2-local-flag/expected.json new file mode 100644 index 000000000..91a421723 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-2-local-flag/expected.json @@ -0,0 +1 @@ +{"surface": "personal", "override_path": ".apache-magpie-local/pr-management-triage.md", "requires_adoption": false} diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-2-local-flag/report.md b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-2-local-flag/report.md new file mode 100644 index 000000000..f45d565fe --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-2-local-flag/report.md @@ -0,0 +1,7 @@ + + +Invocation: `/magpie-setup override pr-management-triage --local` +Flags: --local +Repo is adopted (`.apache-magpie.lock` exists). +`.apache-magpie-local/pr-management-triage.md` does not yet exist. diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-3-not-adopted/expected.json b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-3-not-adopted/expected.json new file mode 100644 index 000000000..91a421723 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-3-not-adopted/expected.json @@ -0,0 +1 @@ +{"surface": "personal", "override_path": ".apache-magpie-local/pr-management-triage.md", "requires_adoption": false} diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-3-not-adopted/report.md b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-3-not-adopted/report.md new file mode 100644 index 000000000..84b936651 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-3-not-adopted/report.md @@ -0,0 +1,7 @@ + + +Invocation: `/magpie-setup override pr-management-triage` +Flags: none +Repo is NOT adopted (`.apache-magpie.lock` does not exist). +`.apache-magpie-local/pr-management-triage.md` does not yet exist. diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-4-both-exist/expected.json b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-4-both-exist/expected.json new file mode 100644 index 000000000..42df96293 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-4-both-exist/expected.json @@ -0,0 +1 @@ +{"surface": "offer-choice", "override_path": ".apache-magpie-local/pr-management-triage.md", "requires_adoption": false} diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-4-both-exist/report.md b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-4-both-exist/report.md new file mode 100644 index 000000000..d77102f56 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/case-4-both-exist/report.md @@ -0,0 +1,8 @@ + + +Invocation: `/magpie-setup override pr-management-triage` +Flags: none +Repo is adopted (`.apache-magpie.lock` exists). +`.apache-magpie-local/pr-management-triage.md` already exists with 2 override sections. +`.apache-magpie-overrides/pr-management-triage.md` already exists with 1 override section. diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/output-spec.md b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/output-spec.md new file mode 100644 index 000000000..7ae72816c --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/output-spec.md @@ -0,0 +1,19 @@ + + +## Output format + +Return ONLY valid JSON with this structure: + +```json +{ + "surface": "personal" | "shared" | "offer-choice", + "override_path": "", + "requires_adoption": true | false +} +``` + +`surface` is `"personal"` when `--local` is passed or the repo is not adopted (only personal available); `"shared"` when the user explicitly requests the committed surface; `"offer-choice"` when the repo is adopted and no `--local` flag was passed (the user must choose). +`override_path` is the relative path to the override file that would be created/opened (relative to the repo root). Use `.apache-magpie-local/.md` for personal, `.apache-magpie-overrides/.md` for shared. +`requires_adoption` is `true` if creating this override requires the repo to be adopted first. +Do not include any text outside the JSON object. diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/step-config.json b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/step-config.json new file mode 100644 index 000000000..9a9a8c2f4 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/step-config.json @@ -0,0 +1,4 @@ +{ + "skill_md": "skills/setup/overrides.md", + "step_heading": "## Step 0b — Choose the override surface" +} diff --git a/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/user-prompt-template.md b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/user-prompt-template.md new file mode 100644 index 000000000..93d5a2f54 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-overrides-surface/fixtures/user-prompt-template.md @@ -0,0 +1,8 @@ + + +## Invocation context + +{report} + +You are at Step 0b of `/magpie-setup override`. Choose the correct override surface. Return JSON only.