From 92e24b810047898fe5f2a4da8c2cf32fcb691402 Mon Sep 17 00:00:00 2001 From: Nate W Date: Mon, 10 Aug 2026 18:12:00 -0700 Subject: [PATCH 01/47] Add AI-assisted TechDocs Assessment spec Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ai-assisted-techdocs-assessment-spec.md | 333 ++++++++++++++++++ 1 file changed, 333 insertions(+) create mode 100644 docs/analysis/ai-assisted-techdocs-assessment-spec.md diff --git a/docs/analysis/ai-assisted-techdocs-assessment-spec.md b/docs/analysis/ai-assisted-techdocs-assessment-spec.md new file mode 100644 index 0000000..c74b3c7 --- /dev/null +++ b/docs/analysis/ai-assisted-techdocs-assessment-spec.md @@ -0,0 +1,333 @@ +--- +title: AI-assisted TechDocs Assessment +--- + +# AI-assisted TechDocs Assessment + +**What this is.** A working specification for an AI-assisted, human-supervised +CNCF TechDocs assessment workflow. It is requirements-first: it defines the +outcomes the system must produce and the boundaries it must never cross, then +lets tooling follow. + +--- + +## 1. Background and problem + +A CNCF TechDocs assessment evaluates a project's documentation against the +TechDocs [criteria] and produces a prioritized improvement plan. It is valuable +and in demand, but each assessment takes 6–8 weeks of a skilled technical +writer's time, and that expertise is scarce. Throughput is the constraint: more +projects want assessments than the expert pool can serve. + +Two things make now the moment to change that: + +- The method is sound; the throughput isn't. The existing methodology + ([criteria], the [howto], and the analysis, implementation, and issues + deliverables) is well-established. The problem to solve is speed and scale, not + what a good assessment is. +- Maintainers now have agentic AI. GitHub has granted CNCF maintainers + [Copilot Enterprise][copilot-blog]. That lets us put an AI agent on the + mechanical and first-draft work, supervised by a human expert, to compress the + cycle. We treat the roughly 2-week figure as a pilot hypothesis, not a fixed + target (G-1). + +Thesis: keep the methodology and the human judgment; use an AI agent to +compress the labor. AI drafts, humans decide, all in the open. + +## 2. Goals and non-goals + +### Goals + +- G-1: Compress the cycle. Reduce the 6–8 week assessment toward roughly 2 + weeks, a pilot hypothesis to validate rather than a commitment (section 10). +- G-2: Hold quality. Output meets or exceeds the current human baseline + (Flatcar, Knative, Helm), verified by the method in section 10, not asserted. + AI drafts fail by being fluent, well-formatted, and generically wrong, a mode + that survives a readability pass, so catching it is an explicit requirement. +- G-3: Human in the loop throughout. The agent proposes; a technical writer and + project stakeholders dispose. Every phase is reviewed before it advances. +- G-4: Work in the open. The whole process happens on cncf/techdocs via issues + and PRs, and the deliverables are public. Assessments can be candid about a + project's documentation gaps; publishing in the open is deliberate and shapes + how sign-off works (HC-2). +- G-5: Repeatable and project-agnostic. The same workflow serves any CNCF + project with minimal per-project setup. +- G-6: Lower the request barrier. A structured intake lets a project (or the + TechDocs team) request an assessment with all needed context up front. +- G-7: Produce the three standard deliverables. Assessment, implementation plan, + and an issue backlog, matching the existing methodology. + +### Non-goals (phase one) + +- NG-1: No external GitHub writes. The system does not create or modify any + file, PR, issue, comment, or setting outside cncf/techdocs. (Hard constraint; + see HC-1 in section 3.) +- NG-2: No auto-filing of issues. The backlog is delivered as files in + cncf/techdocs. Filing them as real issues in project repos is future work (a + separate, opt-in script), explicitly out of scope now. +- NG-3: Not a replacement for human judgment. No unattended, end-to-end + autonomous assessments; no bypassing writer review or stakeholder sign-off. +- NG-4: Not a change to the criteria or methodology. We automate the existing + method. Evolving the criteria (for example AI-readiness) is tracked separately + (see PR #357). The one exception P-2 allows is editorial change to keep the + methodology docs machine-consumable (formatting, structure); changes to the + criteria themselves are out of scope. +- NG-5: Not a public self-serve tool. Scope is CNCF projects assessed through + the defined process, not arbitrary external users. + +## 3. Principles and hard constraints + +### Principles + +- P-1: AI proposes, humans dispose. Every state change that matters is gated by + a human. +- P-2: Methodology as the single source of truth. The methodology docs at + [docs/analysis/][analysis-dir] (`criteria.md`, `howto.md`, and `templates/`) + are canonical. The agent reads and applies them at run time; it does not + reinvent them or copy them into a separate prompt corpus. A single source of + truth means updating the methodology automatically updates the system's + behavior, and no parallel copy can drift. This addresses the + criteria-duplication concern raised in PR #357 review. + - Reproducibility pin. For a given assessment run, the agent binds to a + specific ref (commit or tag) of the methodology docs so the run is + reproducible (P-4). "Auto-updates" means each new assessment picks up the + latest canonical docs, not that an in-flight assessment shifts under review. + - Consumability is a contract. Because the docs are the interface, an + operational layer may add how-to-run guidance (output format, file naming, + orchestration) but must reference the canonical content, never restate it. + Any methodology edits for machine-consumability are editorial (formatting, + structure), never changes to the criteria themselves (NG-4). +- P-3: Requirements before tools. Define the required outcomes and hard + boundaries first; choose tools (agent surface, credentials, scripts, prompts) + afterward to satisfy them, so tools can change without changing the spec. When + a boundary needs a tool to enforce it, require the property (an enforced write + boundary), not a named product. +- P-4: Reproducible and auditable. Every claim in a deliverable traces to a + cited, committed source. Prompts, configuration, and any data-generating steps + are versioned and re-runnable. + +### Hard constraints (the "never" list) + +- HC-1: No writes outside cncf/techdocs. Agents may read and do external + research (web, public repos), but must never write to any GitHub resource + outside cncf/techdocs: no files, PRs, issues, comments, reviews, labels, + releases, or workflow runs. This is enforced architecturally, not by + instruction: the agent runs with a credential scoped to cncf/techdocs only, so + a write elsewhere is impossible, not merely disallowed. Full design in + section 8. +- HC-2: Phase gating. Each phase (assessment, implementation, backlog) is + reviewed by a technical writer, then goes to stakeholders before the next + begins. Sign-off separates two things. A factual-accuracy check: stakeholders + confirm they have read the deliverable and had the chance to correct factual + errors, which is required to advance. Agreement with the conclusions: not + required, and a project's disagreement is recorded in the deliverable rather + than allowed to block or soften it. Merging phase N makes phase N+1 eligible; a + technical writer still starts it (section 5). A reviewer/stakeholder dispute is + arbitrated by the platform owner (section 4). +- HC-3: Deliverables are files in cncf/techdocs. Including the issue backlog + (one file per proposed issue). Nothing is filed to an external repo. +- HC-4: No unattended autonomy. Any change to repository state lands through a + human-reviewed PR. The agent never self-merges. +- HC-5: Provenance required. Quantitative claims (counts, link status, + inventories) must be generated by a committed, re-runnable step and cite their + source; no unverifiable figures in a deliverable. +- HC-6: Disclose AI involvement. The assessed project is told, up front in the + intake and in a header on every deliverable, that the analysis was AI-drafted + and human-reviewed. No project should discover this after the fact. +- HC-7: Findings are verified, not assumed. Qualitative findings are checked + against the actual project sources, not accepted because they read plausibly. + The verification method is in section 10. + +--- + +## 4. Roles + +Roles are defined by function. Each is filled by an actor (a person, group, or +tool), and one actor may fill more than one role. + +- Requester. A stakeholder who initiates the assessment by filing the request + (section 7); for example a maintainer, tech lead, or trusted community member. + Filing does not start an assessment. +- Drafter. Produces the first-pass draft of each phase's deliverable as a draft + PR, then refines it in conversation with the reviewer. +- Reviewer. Accepts a request to begin work, then reviews and refines each draft + in conversation, verifies findings against source (HC-7), and marks it ready. + Owns the draft's quality, but does not give its final sign-off; that is + independent (see Approver). +- Stakeholders. The project party with a stake in its documentation and + direction: maintainers, tech leads, trusted community members, and others. + They set priorities, provide project-level context, and confirm factual + accuracy at each phase gate. They do not hold a veto over the conclusions; + disagreement is recorded, not used to block or soften a deliverable (HC-2). +- SMEs (subject-matter experts). Supply and verify technical ground truth for the + areas under assessment: answering detailed questions and checking findings and + recommendations for accuracy. Often maintainers or experienced contributors, + but the role is knowledge, not authority; an SME need not have sign-off. +- Writers. The audience for the issue backlog: community members who take up + the resulting issues and do the documentation work. +- Approver. Gives each phase its independent quality sign-off and merges the PR. + Any qualified technical writer who did not draft that phase, so no one signs + off on their own work and the role spreads across the team instead of + bottlenecking. Confirms the verification in section 10 was done. +- Platform owner. Owns the machine itself: the prompts, the operational layer, + and the scoped credential (HC-1). Sets policy, handles aborts (section 5), and + arbitrates a reviewer/stakeholder dispute (HC-2). A central role, kept out of + the per-assessment critical path. + +## 5. Lifecycle + +Each phase runs the same six steps: + +1. Request. Phase A is triggered by the intake issue (section 7); phases B and C + become eligible once the previous phase is merged. +2. Accept. A technical writer triages and explicitly accepts the request, for + every phase, not just A; eligibility alone does not start work (P-1). +3. Draft. The agent produces the deliverable as a draft PR. +4. Review. The reviewer refines the draft in conversation with the agent, + verifies findings against source (HC-7), and marks it ready. +5. Stakeholder review. Stakeholders confirm factual accuracy (required to + advance) and record any disagreement with the conclusions (not required; + HC-2). +6. Merge. The approver signs off and merges the PR into cncf/techdocs, which + makes the next phase eligible. + +Failure path: if a draft is bad enough that fixing it would cost more than +starting over, the reviewer may discard it and restart or write by hand rather +than sink time into editing. Repeated failure is escalated to the platform +owner, who may abort the run. An assessment must never end up slower than doing +it by hand. + +## 6. Phases and deliverables + +- Phase A: Assessment (`analysis.md`). Detailed first. +- Phase B: Implementation plan (`implementation.md`). +- Phase C: Issue backlog (`issues/*.md`, one file per issue, each scoped to + roughly 4 hours for someone experienced with the project and comfortable + writing). Effort estimates are the agent's first pass and are sanity-checked + by the reviewer; agent estimates are not reliable on their own. + +Every deliverable carries a header noting it was AI-drafted and human-reviewed +(HC-6). + +Scope note: phase one produces plans (assessment, implementation, backlog), not +documentation changes, and the backlog lands as files in cncf/techdocs rather +than as issues where writers work (NG-2, HC-3). Until the optional filing tool +exists (section 11), the reviewer and stakeholders route the backlog to writers +by hand. The pilot therefore measures output, not documentation outcomes, a +distinction we state so we don't mistake one for the other. + +## 7. Inputs: the assessment request + +The process starts from a GitHub issue in cncf/techdocs, filed with a request +template (`.github/ISSUE_TEMPLATE/`) that collects the context an assessment +needs. The specific fields are deferred for now; they will derive from what the +methodology already requires (the analysis template's "About" and "Scope" +sections, plus maturity and contacts). + +Filing a request does not start an assessment: it gathers context, and a +technical writer triages and explicitly accepts it to begin Phase A (NG-5, P-1). + +Open question: how this relates to the existing CNCF service desk and TechDocs +assistance-program intake, so we don't create a competing front door. + +## 8. Safety and guardrails + +HC-1 is guaranteed primarily by credential scoping: the agent runs under a +credential (fine-grained token or GitHub App) that can write only to +cncf/techdocs, so writes elsewhere are impossible, not merely disallowed. Tool +allowlisting (reads and web research permitted; GitHub-write tools constrained to +the one repo) and the human review gates are defense in depth. + +Threat cases: + +1. Prompt injection (primary threat). The agent's core job is ingesting untrusted + content from assessed repositories, which may contain instructions aimed at + the agent. Scoping blocks writes outside cncf/techdocs, but injection can + still try to (a) plant poisoned or misleading content into a deliverable + inside cncf/techdocs, or (b) trigger outbound web requests to exfiltrate or + fetch. Mitigations: treat all repo and web content as data, never + instructions; human review of every deliverable before merge (HC-4); constrain + outbound network access; and keep secrets out of the agent's reach. +2. Write outside cncf/techdocs. Blocked architecturally by the scoped credential + (HC-1). +3. Unverified content in a deliverable. Qualitative findings must be checked + against source (HC-7); quantitative claims must be reproducible (HC-5). +4. Self-merge or unattended change. Prevented by human-reviewed PRs; the agent + never merges (HC-4). + +The credential mechanism (fine-grained PAT versus GitHub App, both scoped to +cncf/techdocs) is an implementation choice; the required property is that the +write boundary is enforced, not advisory (P-3). + +## 9. Grounding in methodology + +Every requirement here derives from the existing methodology, so the system +automates the current method rather than inventing a new one (P-2): + +- The three deliverables (G-7) and the phase order (section 5) follow the + [howto]: analysis, then implementation plan, then issue backlog. +- Assessment content, ratings, and criteria come from `criteria.md`. +- Each deliverable's structure comes from the templates in + `docs/analysis/templates/` (`analysis.md`, `implementation.md`, + `issues-list.md`, `issue.md`). +- Scope and maturity framing (section 7) come from the analysis template's + "About" and "Scope" sections and the criteria's maturity levels. +- The 4-hour, independent, time-bounded issue scoping (Phase C) follows the + howto and assistance-program guidance on backlog creation. + +Prior prototyping has explored parts of this workflow; that experience will +inform implementation. + +## 10. Acceptance criteria + +The system is acceptable when, on a pilot assessment: + +- Quality. The deliverable is scored against an assessment-quality rubric (its + definition is an open item; section 11). Verification is not a token sample: + the reviewer checks every rating-bearing finding, or at minimum a set number + per criterion, biased toward the highest-risk claims, and records in the + deliverable which findings were verified (HC-7). Final sign-off is given by the + approver (section 4), a writer who did not draft that phase, to avoid grading + one's own work. The bar is parity with the human baselines (Flatcar, Knative, + Helm). +- Cycle time. Measured against the time decomposition from G-1 (writer working + time versus waiting on people). The roughly 2-week target is evaluated as a + hypothesis; missing it prompts a look at which gates or waits dominate, not a + quiet redefinition. +- Safety. Zero writes outside cncf/techdocs, audited from the scoped credential's + activity, and no unmitigated prompt-injection incident. +- Completeness and reproducibility. All three deliverables produced; every + quantitative claim reproducible from a committed step (HC-5); AI involvement + disclosed (HC-6). + +Pilot caveat: one assessment is n=1 for a quality claim. Choose the pilot +deliberately, neither a flattering easy project nor an impossibly hard one, state +the choice, and broaden before drawing general conclusions. + +## 11. Open questions and future work + +- Filing issues into project repos. A separate, opt-in tool to create the backlog + issues in a project's own repository (NG-2). Out of scope for phase one, and it + must preserve HC-1. +- Intake relationship. How the request template fits with the existing CNCF + service desk and assistance-program intake (section 7), without a competing + front door. +- AI-readiness criteria. Whether and how the criteria evolve for AI consumption + (for example llms.txt), tracked in PR #357. +- Pilot measurement. How we capture the G-1 time decomposition and cycle-time + data to test the 2-week hypothesis (section 10). +- Assessment-quality rubric. Define the meta-rubric that scores an assessment's + quality, distinct from `criteria.md` (which scores a project's docs), and + validate it by scoring the Flatcar, Knative, and Helm baselines to set a + reference band (sections 2, 10). +- Small-team staffing. Sustaining the approver separation (a phase's approver + must be a writer who did not draft it; sections 4, 10) when the same few + writers wear multiple hats. +- Scaling. Running assessments for several projects concurrently. + +--- + +[criteria]: https://github.com/cncf/techdocs/blob/main/docs/analysis/criteria.md +[howto]: https://github.com/cncf/techdocs/blob/main/docs/analysis/howto.md +[analysis-dir]: https://github.com/cncf/techdocs/tree/main/docs/analysis +[copilot-blog]: https://contribute.cncf.io/blog/2025/12/16/github-copilot-enterprise-for-maintainers/ From e2313564bb6252c7445c145288b4507a48d49995 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 08:38:08 -0700 Subject: [PATCH 02/47] Address review: approver independence; propose per-file issue backlog Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ai-assisted-techdocs-assessment-spec.md | 50 ++++++++++++------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/docs/analysis/ai-assisted-techdocs-assessment-spec.md b/docs/analysis/ai-assisted-techdocs-assessment-spec.md index c74b3c7..cb32c1b 100644 --- a/docs/analysis/ai-assisted-techdocs-assessment-spec.md +++ b/docs/analysis/ai-assisted-techdocs-assessment-spec.md @@ -64,14 +64,18 @@ compress the labor. AI drafts, humans decide, all in the open. see HC-1 in section 3.) - NG-2: No auto-filing of issues. The backlog is delivered as files in cncf/techdocs. Filing them as real issues in project repos is future work (a - separate, opt-in script), explicitly out of scope now. + separate, opt-in script; see section 11), explicitly out of scope now. The + per-file backlog format proposed in section 6 is chosen partly to make that + future script straightforward. - NG-3: Not a replacement for human judgment. No unattended, end-to-end autonomous assessments; no bypassing writer review or stakeholder sign-off. -- NG-4: Not a change to the criteria or methodology. We automate the existing - method. Evolving the criteria (for example AI-readiness) is tracked separately - (see PR #357). The one exception P-2 allows is editorial change to keep the - methodology docs machine-consumable (formatting, structure); changes to the - criteria themselves are out of scope. +- NG-4: Not a change to the criteria or evaluation method. We automate the + existing assessment method; the criteria and how a project is evaluated are + unchanged. Evolving the criteria (for example AI-readiness) is tracked + separately (see PR #357). This spec does propose one deliberate, limited + structural change to the methodology: the issue-backlog file layout (section + 6), framed as a proposal for discussion, not a silent change. Editorial changes + to keep the methodology docs machine-consumable (P-2) are also allowed. - NG-5: Not a public self-serve tool. Scope is CNCF projects assessed through the defined process, not arbitrary external users. @@ -166,9 +170,10 @@ tool), and one actor may fill more than one role. - Writers. The audience for the issue backlog: community members who take up the resulting issues and do the documentation work. - Approver. Gives each phase its independent quality sign-off and merges the PR. - Any qualified technical writer who did not draft that phase, so no one signs - off on their own work and the role spreads across the team instead of - bottlenecking. Confirms the verification in section 10 was done. + A qualified technical writer who was neither the drafter nor a reviewer of that + phase, so no one signs off on work they produced or refined, and the role + spreads across the team instead of bottlenecking. Confirms the verification in + section 10 was done. - Platform owner. Owns the machine itself: the prompts, the operational layer, and the scoped credential (HC-1). Sets policy, handles aborts (section 5), and arbitrates a reviewer/stakeholder dispute (HC-2). A central role, kept out of @@ -201,10 +206,21 @@ it by hand. - Phase A: Assessment (`analysis.md`). Detailed first. - Phase B: Implementation plan (`implementation.md`). -- Phase C: Issue backlog (`issues/*.md`, one file per issue, each scoped to +- Phase C: Issue backlog. Each proposed issue is a separate file scoped to roughly 4 hours for someone experienced with the project and comfortable - writing). Effort estimates are the agent's first pass and are sanity-checked - by the reviewer; agent estimates are not reliable on their own. + writing, plus an umbrella/index file that lists them. Effort estimates are the + agent's first pass and are sanity-checked by the reviewer; agent estimates are + not reliable on their own. + + Proposed methodology change. The current method delivers the backlog as one + `_PROJECT_-issues.md` file (`howto.md`, `templates/issues-list.md`, + `templates/issue.md`). This spec proposes splitting the individual issues into + one file each while keeping a single umbrella/index document. Rationale: a + directory of one-issue-per-file is the natural input for the future filing + script (NG-2, section 11), which can batch-create issues with `gh` instead of + someone copy-pasting from a large combined file; it also keeps any single file + readable. This is raised as a proposal for discussion; the methodology docs are + not changed by this PR. Every deliverable carries a header noting it was AI-drafted and human-reviewed (HC-6). @@ -287,9 +303,9 @@ The system is acceptable when, on a pilot assessment: the reviewer checks every rating-bearing finding, or at minimum a set number per criterion, biased toward the highest-risk claims, and records in the deliverable which findings were verified (HC-7). Final sign-off is given by the - approver (section 4), a writer who did not draft that phase, to avoid grading - one's own work. The bar is parity with the human baselines (Flatcar, Knative, - Helm). + approver (section 4), who was neither the drafter nor a reviewer of that phase, + to avoid signing off on one's own work. The bar is parity with the human + baselines (Flatcar, Knative, Helm). - Cycle time. Measured against the time decomposition from G-1 (writer working time versus waiting on people). The roughly 2-week target is evaluated as a hypothesis; missing it prompts a look at which gates or waits dominate, not a @@ -321,8 +337,8 @@ the choice, and broaden before drawing general conclusions. validate it by scoring the Flatcar, Knative, and Helm baselines to set a reference band (sections 2, 10). - Small-team staffing. Sustaining the approver separation (a phase's approver - must be a writer who did not draft it; sections 4, 10) when the same few - writers wear multiple hats. + must be neither the drafter nor a reviewer of that phase; sections 4, 10) when + the same few writers wear multiple hats. - Scaling. Running assessments for several projects concurrently. --- From 84d0c8f74eb99a2a26861a8914ffa6b59ae9b0eb Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 09:12:42 -0700 Subject: [PATCH 03/47] Address review: approver independence, per-file backlog proposal; fix CI Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .cspell.yml | 7 ++ .../ai-assisted-techdocs-assessment-spec.md | 95 ++++++++++--------- 2 files changed, 56 insertions(+), 46 deletions(-) diff --git a/.cspell.yml b/.cspell.yml index 5a4db57..b50e2b1 100644 --- a/.cspell.yml +++ b/.cspell.yml @@ -20,19 +20,26 @@ ignorePaths: # ignoreRegExpList: # - CodeBlock words: + - agentic - backstore + - bottlenecking - Chalin - CLOTributor - CNCF + - cncf + - consumability - Docsy - dwelsch + - exfiltrate - fluxcd - github + - howto - keda - knative - kedacore - krator - krook + - llms - mkdocs - nate - nvmrc diff --git a/docs/analysis/ai-assisted-techdocs-assessment-spec.md b/docs/analysis/ai-assisted-techdocs-assessment-spec.md index cb32c1b..28f60aa 100644 --- a/docs/analysis/ai-assisted-techdocs-assessment-spec.md +++ b/docs/analysis/ai-assisted-techdocs-assessment-spec.md @@ -23,16 +23,16 @@ Two things make now the moment to change that: - The method is sound; the throughput isn't. The existing methodology ([criteria], the [howto], and the analysis, implementation, and issues - deliverables) is well-established. The problem to solve is speed and scale, not - what a good assessment is. -- Maintainers now have agentic AI. GitHub has granted CNCF maintainers - [Copilot Enterprise][copilot-blog]. That lets us put an AI agent on the - mechanical and first-draft work, supervised by a human expert, to compress the - cycle. We treat the roughly 2-week figure as a pilot hypothesis, not a fixed - target (G-1). + deliverables) is well-established. The problem to solve is speed and scale, + not what a good assessment is. +- Maintainers now have agentic AI. GitHub has granted CNCF maintainers [Copilot + Enterprise][copilot-blog]. That lets us put an AI agent on the mechanical and + first-draft work, supervised by a human expert, to compress the cycle. We + treat the roughly 2-week figure as a pilot hypothesis, not a fixed target + (G-1). -Thesis: keep the methodology and the human judgment; use an AI agent to -compress the labor. AI drafts, humans decide, all in the open. +Thesis: keep the methodology and the human judgment; use an AI agent to compress +the labor. AI drafts, humans decide, all in the open. ## 2. Goals and non-goals @@ -74,8 +74,9 @@ compress the labor. AI drafts, humans decide, all in the open. unchanged. Evolving the criteria (for example AI-readiness) is tracked separately (see PR #357). This spec does propose one deliberate, limited structural change to the methodology: the issue-backlog file layout (section - 6), framed as a proposal for discussion, not a silent change. Editorial changes - to keep the methodology docs machine-consumable (P-2) are also allowed. + 6), framed as a proposal for discussion, not a silent change. Editorial + changes to keep the methodology docs machine-consumable (P-2) are also + allowed. - NG-5: Not a public self-serve tool. Scope is CNCF projects assessed through the defined process, not arbitrary external users. @@ -125,9 +126,9 @@ compress the labor. AI drafts, humans decide, all in the open. confirm they have read the deliverable and had the chance to correct factual errors, which is required to advance. Agreement with the conclusions: not required, and a project's disagreement is recorded in the deliverable rather - than allowed to block or soften it. Merging phase N makes phase N+1 eligible; a - technical writer still starts it (section 5). A reviewer/stakeholder dispute is - arbitrated by the platform owner (section 4). + than allowed to block or soften it. Merging phase N makes phase N+1 eligible; + a technical writer still starts it (section 5). A reviewer/stakeholder dispute + is arbitrated by the platform owner (section 4). - HC-3: Deliverables are files in cncf/techdocs. Including the issue backlog (one file per proposed issue). Nothing is filed to an external repo. - HC-4: No unattended autonomy. Any change to repository state lands through a @@ -163,15 +164,16 @@ tool), and one actor may fill more than one role. They set priorities, provide project-level context, and confirm factual accuracy at each phase gate. They do not hold a veto over the conclusions; disagreement is recorded, not used to block or soften a deliverable (HC-2). -- SMEs (subject-matter experts). Supply and verify technical ground truth for the - areas under assessment: answering detailed questions and checking findings and - recommendations for accuracy. Often maintainers or experienced contributors, - but the role is knowledge, not authority; an SME need not have sign-off. -- Writers. The audience for the issue backlog: community members who take up - the resulting issues and do the documentation work. +- SMEs (subject-matter experts). Supply and verify technical ground truth for + the areas under assessment: answering detailed questions and checking findings + and recommendations for accuracy. Often maintainers or experienced + contributors, but the role is knowledge, not authority; an SME need not have + sign-off. +- Writers. The audience for the issue backlog: community members who take up the + resulting issues and do the documentation work. - Approver. Gives each phase its independent quality sign-off and merges the PR. - A qualified technical writer who was neither the drafter nor a reviewer of that - phase, so no one signs off on work they produced or refined, and the role + A qualified technical writer who was neither the drafter nor a reviewer of + that phase, so no one signs off on work they produced or refined, and the role spreads across the team instead of bottlenecking. Confirms the verification in section 10 was done. - Platform owner. Owns the machine itself: the prompts, the operational layer, @@ -219,8 +221,8 @@ it by hand. directory of one-issue-per-file is the natural input for the future filing script (NG-2, section 11), which can batch-create issues with `gh` instead of someone copy-pasting from a large combined file; it also keeps any single file - readable. This is raised as a proposal for discussion; the methodology docs are - not changed by this PR. + readable. This is raised as a proposal for discussion; the methodology docs + are not changed by this PR. Every deliverable carries a header noting it was AI-drafted and human-reviewed (HC-6). @@ -251,19 +253,19 @@ assistance-program intake, so we don't create a competing front door. HC-1 is guaranteed primarily by credential scoping: the agent runs under a credential (fine-grained token or GitHub App) that can write only to cncf/techdocs, so writes elsewhere are impossible, not merely disallowed. Tool -allowlisting (reads and web research permitted; GitHub-write tools constrained to -the one repo) and the human review gates are defense in depth. +allowlisting (reads and web research permitted; GitHub-write tools constrained +to the one repo) and the human review gates are defense in depth. Threat cases: -1. Prompt injection (primary threat). The agent's core job is ingesting untrusted - content from assessed repositories, which may contain instructions aimed at - the agent. Scoping blocks writes outside cncf/techdocs, but injection can - still try to (a) plant poisoned or misleading content into a deliverable - inside cncf/techdocs, or (b) trigger outbound web requests to exfiltrate or - fetch. Mitigations: treat all repo and web content as data, never - instructions; human review of every deliverable before merge (HC-4); constrain - outbound network access; and keep secrets out of the agent's reach. +1. Prompt injection (primary threat). The agent's core job is ingesting + untrusted content from assessed repositories, which may contain instructions + aimed at the agent. Scoping blocks writes outside cncf/techdocs, but + injection can still try to (a) plant poisoned or misleading content into a + deliverable inside cncf/techdocs, or (b) trigger outbound web requests to + exfiltrate or fetch. Mitigations: treat all repo and web content as data, + never instructions; human review of every deliverable before merge (HC-4); + constrain outbound network access; and keep secrets out of the agent's reach. 2. Write outside cncf/techdocs. Blocked architecturally by the scoped credential (HC-1). 3. Unverified content in a deliverable. Qualitative findings must be checked @@ -302,29 +304,29 @@ The system is acceptable when, on a pilot assessment: definition is an open item; section 11). Verification is not a token sample: the reviewer checks every rating-bearing finding, or at minimum a set number per criterion, biased toward the highest-risk claims, and records in the - deliverable which findings were verified (HC-7). Final sign-off is given by the - approver (section 4), who was neither the drafter nor a reviewer of that phase, - to avoid signing off on one's own work. The bar is parity with the human - baselines (Flatcar, Knative, Helm). + deliverable which findings were verified (HC-7). Final sign-off is given by + the approver (section 4), who was neither the drafter nor a reviewer of that + phase, to avoid signing off on one's own work. The bar is parity with the + human baselines (Flatcar, Knative, Helm). - Cycle time. Measured against the time decomposition from G-1 (writer working time versus waiting on people). The roughly 2-week target is evaluated as a hypothesis; missing it prompts a look at which gates or waits dominate, not a quiet redefinition. -- Safety. Zero writes outside cncf/techdocs, audited from the scoped credential's - activity, and no unmitigated prompt-injection incident. +- Safety. Zero writes outside cncf/techdocs, audited from the scoped + credential's activity, and no unmitigated prompt-injection incident. - Completeness and reproducibility. All three deliverables produced; every quantitative claim reproducible from a committed step (HC-5); AI involvement disclosed (HC-6). Pilot caveat: one assessment is n=1 for a quality claim. Choose the pilot -deliberately, neither a flattering easy project nor an impossibly hard one, state -the choice, and broaden before drawing general conclusions. +deliberately, neither a flattering easy project nor an impossibly hard one, +state the choice, and broaden before drawing general conclusions. ## 11. Open questions and future work -- Filing issues into project repos. A separate, opt-in tool to create the backlog - issues in a project's own repository (NG-2). Out of scope for phase one, and it - must preserve HC-1. +- Filing issues into project repos. A separate, opt-in tool to create the + backlog issues in a project's own repository (NG-2). Out of scope for phase + one, and it must preserve HC-1. - Intake relationship. How the request template fits with the existing CNCF service desk and assistance-program intake (section 7), without a competing front door. @@ -346,4 +348,5 @@ the choice, and broaden before drawing general conclusions. [criteria]: https://github.com/cncf/techdocs/blob/main/docs/analysis/criteria.md [howto]: https://github.com/cncf/techdocs/blob/main/docs/analysis/howto.md [analysis-dir]: https://github.com/cncf/techdocs/tree/main/docs/analysis -[copilot-blog]: https://contribute.cncf.io/blog/2025/12/16/github-copilot-enterprise-for-maintainers/ +[copilot-blog]: + https://contribute.cncf.io/blog/2025/12/16/github-copilot-enterprise-for-maintainers/ From a9356008b7c49f07e01ed9eb6448954de9d45eb0 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 10:13:00 -0700 Subject: [PATCH 04/47] Restructure spec into Part I (requirements) and Part II (operational) Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../ai-assisted-techdocs-assessment-spec.md | 37 +++++++++++++++---- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/docs/analysis/ai-assisted-techdocs-assessment-spec.md b/docs/analysis/ai-assisted-techdocs-assessment-spec.md index 28f60aa..0ad1125 100644 --- a/docs/analysis/ai-assisted-techdocs-assessment-spec.md +++ b/docs/analysis/ai-assisted-techdocs-assessment-spec.md @@ -5,12 +5,15 @@ title: AI-assisted TechDocs Assessment # AI-assisted TechDocs Assessment **What this is.** A working specification for an AI-assisted, human-supervised -CNCF TechDocs assessment workflow. It is requirements-first: it defines the -outcomes the system must produce and the boundaries it must never cross, then -lets tooling follow. +CNCF TechDocs assessment workflow, in two parts. Part I is requirements: the +outcomes the system must produce and the boundaries it must never cross. Part II +is the operational specification: how those requirements bind to concrete +tooling, and the plan for building the system. --- +## Part I: Requirements + ## 1. Background and problem A CNCF TechDocs assessment evaluates a project's documentation against the @@ -64,7 +67,7 @@ the labor. AI drafts, humans decide, all in the open. see HC-1 in section 3.) - NG-2: No auto-filing of issues. The backlog is delivered as files in cncf/techdocs. Filing them as real issues in project repos is future work (a - separate, opt-in script; see section 11), explicitly out of scope now. The + separate, opt-in script; see section 17), explicitly out of scope now. The per-file backlog format proposed in section 6 is chosen partly to make that future script straightforward. - NG-3: Not a replacement for human judgment. No unattended, end-to-end @@ -219,7 +222,7 @@ it by hand. `templates/issue.md`). This spec proposes splitting the individual issues into one file each while keeping a single umbrella/index document. Rationale: a directory of one-issue-per-file is the natural input for the future filing - script (NG-2, section 11), which can batch-create issues with `gh` instead of + script (NG-2, section 17), which can batch-create issues with `gh` instead of someone copy-pasting from a large combined file; it also keeps any single file readable. This is raised as a proposal for discussion; the methodology docs are not changed by this PR. @@ -230,7 +233,7 @@ Every deliverable carries a header noting it was AI-drafted and human-reviewed Scope note: phase one produces plans (assessment, implementation, backlog), not documentation changes, and the backlog lands as files in cncf/techdocs rather than as issues where writers work (NG-2, HC-3). Until the optional filing tool -exists (section 11), the reviewer and stakeholders route the backlog to writers +exists (section 17), the reviewer and stakeholders route the backlog to writers by hand. The pilot therefore measures output, not documentation outcomes, a distinction we state so we don't mistake one for the other. @@ -301,7 +304,7 @@ inform implementation. The system is acceptable when, on a pilot assessment: - Quality. The deliverable is scored against an assessment-quality rubric (its - definition is an open item; section 11). Verification is not a token sample: + definition is an open item; section 17). Verification is not a token sample: the reviewer checks every rating-bearing finding, or at minimum a set number per criterion, biased toward the highest-risk claims, and records in the deliverable which findings were verified (HC-7). Final sign-off is given by @@ -322,7 +325,25 @@ Pilot caveat: one assessment is n=1 for a quality claim. Choose the pilot deliberately, neither a flattering easy project nor an impossibly hard one, state the choice, and broaden before drawing general conclusions. -## 11. Open questions and future work +--- + +## Part II: Operational specification + +Part I defines what must be true and what must never happen. Part II binds those +requirements to concrete tooling: named products, file paths, configuration, and +the plan for building the system. The layers are deliberately separate (P-3). +Requirements govern: if a binding here conflicts with Part I, Part I wins, and +the tooling can be swapped without reopening the requirements. + +The binding targets the GitHub Copilot cloud agent (documented until recently as +"Copilot coding agent"). Its behavior as stated here is taken from GitHub's +public documentation as of August 2026; platform behavior we could not verify +from public documentation is treated as a build-time check (section 17), not +assumed. Part II covers, in order: the platform binding, lifecycle bindings, +components and repository layout, agent definitions, the provenance block, and +the build plan. + +## 17. Open questions and future work - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase From 314d0033c0c4f7c5349f4ea4ab23d8a192c8fb70 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 10:13:00 -0700 Subject: [PATCH 05/47] Move spec to docs/ top level, keeping the methodology corpus clean Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/{analysis => }/ai-assisted-techdocs-assessment-spec.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/{analysis => }/ai-assisted-techdocs-assessment-spec.md (100%) diff --git a/docs/analysis/ai-assisted-techdocs-assessment-spec.md b/docs/ai-assisted-techdocs-assessment-spec.md similarity index 100% rename from docs/analysis/ai-assisted-techdocs-assessment-spec.md rename to docs/ai-assisted-techdocs-assessment-spec.md From aac77f19dcd673b605ba52eb608d61d5c5e69a11 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 10:22:54 -0700 Subject: [PATCH 06/47] Add section 11: platform binding to Copilot cloud agent Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/ai-assisted-techdocs-assessment-spec.md | 92 ++++++++++++++++---- 1 file changed, 77 insertions(+), 15 deletions(-) diff --git a/docs/ai-assisted-techdocs-assessment-spec.md b/docs/ai-assisted-techdocs-assessment-spec.md index 0ad1125..b98a4cf 100644 --- a/docs/ai-assisted-techdocs-assessment-spec.md +++ b/docs/ai-assisted-techdocs-assessment-spec.md @@ -120,8 +120,8 @@ the labor. AI drafts, humans decide, all in the open. research (web, public repos), but must never write to any GitHub resource outside cncf/techdocs: no files, PRs, issues, comments, reviews, labels, releases, or workflow runs. This is enforced architecturally, not by - instruction: the agent runs with a credential scoped to cncf/techdocs only, so - a write elsewhere is impossible, not merely disallowed. Full design in + instruction: the agent runs inside a write boundary scoped to cncf/techdocs, + so a write elsewhere is impossible, not merely disallowed. Full design in section 8. - HC-2: Phase gating. Each phase (assessment, implementation, backlog) is reviewed by a technical writer, then goes to stakeholders before the next @@ -180,9 +180,9 @@ tool), and one actor may fill more than one role. spreads across the team instead of bottlenecking. Confirms the verification in section 10 was done. - Platform owner. Owns the machine itself: the prompts, the operational layer, - and the scoped credential (HC-1). Sets policy, handles aborts (section 5), and - arbitrates a reviewer/stakeholder dispute (HC-2). A central role, kept out of - the per-assessment critical path. + and the configuration that enforces the write boundary (HC-1). Sets policy, + handles aborts (section 5), and arbitrates a reviewer/stakeholder dispute + (HC-2). A central role, kept out of the per-assessment critical path. ## 5. Lifecycle @@ -253,11 +253,11 @@ assistance-program intake, so we don't create a competing front door. ## 8. Safety and guardrails -HC-1 is guaranteed primarily by credential scoping: the agent runs under a -credential (fine-grained token or GitHub App) that can write only to -cncf/techdocs, so writes elsewhere are impossible, not merely disallowed. Tool -allowlisting (reads and web research permitted; GitHub-write tools constrained -to the one repo) and the human review gates are defense in depth. +HC-1 is guaranteed primarily by an enforced write boundary: the agent can write +only to cncf/techdocs, so writes elsewhere are impossible, not merely +disallowed. Tool allowlisting (reads and web research permitted; GitHub-write +tools constrained to the one repo) and the human review gates are defense in +depth. Threat cases: @@ -276,9 +276,9 @@ Threat cases: 4. Self-merge or unattended change. Prevented by human-reviewed PRs; the agent never merges (HC-4). -The credential mechanism (fine-grained PAT versus GitHub App, both scoped to -cncf/techdocs) is an implementation choice; the required property is that the -write boundary is enforced, not advisory (P-3). +The enforcement mechanism is an implementation choice; the required property is +that the write boundary is enforced, not advisory (P-3). The chosen binding and +its documented guarantees are specified in Part II (section 11). ## 9. Grounding in methodology @@ -315,8 +315,8 @@ The system is acceptable when, on a pilot assessment: time versus waiting on people). The roughly 2-week target is evaluated as a hypothesis; missing it prompts a look at which gates or waits dominate, not a quiet redefinition. -- Safety. Zero writes outside cncf/techdocs, audited from the scoped - credential's activity, and no unmitigated prompt-injection incident. +- Safety. Zero writes outside cncf/techdocs, audited from the platform's + activity records, and no unmitigated prompt-injection incident. - Completeness and reproducibility. All three deliverables produced; every quantitative claim reproducible from a committed step (HC-5); AI involvement disclosed (HC-6). @@ -343,6 +343,58 @@ assumed. Part II covers, in order: the platform binding, lifecycle bindings, components and repository layout, agent definitions, the provenance block, and the build plan. +## 11. Platform binding + +The system runs on the [GitHub Copilot cloud agent][cloud-agent-about] operating +inside cncf/techdocs. This binding is chosen because the platform's documented +containment model satisfies HC-1 as shipped, rather than requiring credential +infrastructure we build and maintain ourselves. + +- The write boundary (HC-1). The cloud agent can only make changes in the + repository where its task was started, on a single branch, through a single + pull request per task, using a token the platform issues for that run. The + built-in GitHub MCP server connects with read-only access to the current + repository by default. Together these provide the enforced write boundary Part + I requires; we mint and manage no credential of our own. The platform does not + publicly enumerate the agent token's exact permission scopes, so confirming + them empirically is a build-time check (section 17). +- Outbound network. The agent's internet access is governed by a [default-deny + firewall][cloud-agent-firewall] with an allowlist. The default allowlist + covers package registries and GitHub content domains, not the general web, so + reading an assessed project's live documentation site requires an explicit + allowlist entry. We treat this as a feature: the intake (section 7) collects + the project's documentation domains, and the platform owner adds them at + acceptance, making each assessment's external reads an explicit, auditable, + reversible contract. Requests the firewall blocks are disclosed automatically + in the pull request, which feeds the safety audit in section 10. +- MCP policy. The firewall does not apply to MCP servers, and MCP tools are the + one documented mechanism that can widen the agent's write reach, so [MCP + configuration][cloud-agent-mcp] is the control surface that matters most. + Policy: the GitHub MCP server stays at its read-only default; any additional + MCP server must be read-only with its tools explicitly allowlisted; no + write-capable MCP tool is permitted. MCP configuration lives in repository + settings rather than in a versioned file, so the platform owner records the + current configuration in the operational docs whenever it changes. +- Execution environment. The agent runs in an ephemeral [GitHub Actions-based + environment][cloud-agent-env] with a hard session cap (currently 59 minutes). + That cap shapes the design: deterministic data collection (section 13) runs as + committed scripts rather than inside drafting sessions, so the agent spends + its session on judgment, not inventory. The environment is prepared by + `.github/workflows/copilot-setup-steps.yml`. +- Delegation is permission-gated. Only users with [write access to the + repository][cloud-agent-access] can delegate work to the agent, which lets the + lifecycle enforce P-1 with platform permissions instead of process discipline + (section 12). Scheduled or event-triggered agent runs are not available on + public repositories, which costs us nothing: every run is human-started by + design (P-1). +- Preconditions. For CNCF's GitHub organization, the cloud agent and MCP + policies are disabled by default and must be enabled by organization + administrators, with repository access granted for cncf/techdocs. Usage draws + on the AI credits pooled across CNCF's Copilot Enterprise seats plus GitHub + Actions minutes, and spending beyond the included credits is enabled by + default at the organization level, so administrators should review the cap. + These preconditions open the build plan (section 16). + ## 17. Open questions and future work - Filing issues into project repos. A separate, opt-in tool to create the @@ -371,3 +423,13 @@ the build plan. [analysis-dir]: https://github.com/cncf/techdocs/tree/main/docs/analysis [copilot-blog]: https://contribute.cncf.io/blog/2025/12/16/github-copilot-enterprise-for-maintainers/ +[cloud-agent-about]: + https://docs.github.com/en/copilot/concepts/agents/cloud-agent/about-cloud-agent +[cloud-agent-firewall]: + https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-the-firewall +[cloud-agent-mcp]: + https://docs.github.com/en/copilot/concepts/agents/cloud-agent/mcp-and-cloud-agent +[cloud-agent-env]: + https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-cloud-agent/customize-the-agent-environment +[cloud-agent-access]: + https://docs.github.com/en/copilot/concepts/agents/cloud-agent/access-management From e3714aa57ab691b841f2732b23d589e83af3b20a Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 10:30:42 -0700 Subject: [PATCH 07/47] Add section 12: lifecycle bindings to GitHub primitives Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/ai-assisted-techdocs-assessment-spec.md | 46 +++++++++++++++++++- 1 file changed, 44 insertions(+), 2 deletions(-) diff --git a/docs/ai-assisted-techdocs-assessment-spec.md b/docs/ai-assisted-techdocs-assessment-spec.md index b98a4cf..3061010 100644 --- a/docs/ai-assisted-techdocs-assessment-spec.md +++ b/docs/ai-assisted-techdocs-assessment-spec.md @@ -146,8 +146,6 @@ the labor. AI drafts, humans decide, all in the open. against the actual project sources, not accepted because they read plausibly. The verification method is in section 10. ---- - ## 4. Roles Roles are defined by function. Each is filled by an actor (a person, group, or @@ -395,6 +393,50 @@ infrastructure we build and maintain ourselves. default at the organization level, so administrators should review the cap. These preconditions open the build plan (section 16). +## 12. Lifecycle bindings + +Each lifecycle step (section 5) binds to a native GitHub primitive, so the +workflow needs no orchestration layer of its own. An assessment's state is +always readable from its issues and pull requests. + +- Request (step 1): an issue. Phase A's request is the intake issue (section 7); + phase B's and C's tracking issues are opened automatically when the previous + phase merges (see Merge below). An issue may sit unassigned indefinitely; + eligibility is not work. +- Accept (step 2): assignment. A writer accepts by assigning the issue to + Copilot, adding themselves as an assignee to record who accepted; they act as + the phase's reviewer. Because only users with write access can delegate to the + agent (section 11), P-1 is enforced by repository permissions, not convention. +- Draft (step 3): a draft pull request. The agent works on its own branch in + cncf/techdocs and opens a draft PR linked to the tracking issue, carrying the + provenance block (section 15). +- Review (step 4): PR review comments. The reviewer refines the draft in + conversation, handing revisions back by mentioning `@copilot` in review + comments, and performs the verification required by section 10. Whether + mention-triggered revisions require write access is a build-time check + (section 17). +- Stakeholder review (step 5): mention, not access. The reviewer marks the PR + ready for review and mentions the stakeholders named in the intake. + cncf/techdocs is public, so stakeholders can review and comment without any + special access; formal review requests are limited to collaborators, which is + why the binding is a mention plus a factual-accuracy confirmation recorded on + the PR, with any disagreement recorded in the deliverable itself (HC-2). +- Merge (step 6): merge plus one workflow. The approver (section 4) merges. A + merge-triggered GitHub Actions workflow (standard `GITHUB_TOKEN` with + `issues: write`) then opens the next phase's tracking issue, linking the + intake and the merged deliverable, and assigns no one: eligible, not started + (HC-2, P-1). Approver independence cannot be natively enforced by GitHub; it + is verifiable from the public PR record, and an advisory CI check that flags a + violation is a build-plan candidate (section 16). +- Failure and abort leave a trail. Discarding a draft (section 5) is closing its + PR with the reason recorded in a comment; the tracking issue stays open for a + restart or a hand-written phase. A platform-owner abort closes the tracking + issue, with the rationale recorded there. +- The timeline is the measurement. Gate transitions (opened, accepted, draft PR, + ready, approved, merged) are timestamped in issue and PR history, so the + pilot's cycle-time decomposition (section 10) is harvested from the GitHub + timeline rather than a separate log. + ## 17. Open questions and future work - Filing issues into project repos. A separate, opt-in tool to create the From 7edd5387c480c08bfeb527c80f39cefe26e2c486 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 10:43:30 -0700 Subject: [PATCH 08/47] Add section 13: components and repository layout Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/ai-assisted-techdocs-assessment-spec.md | 40 ++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/docs/ai-assisted-techdocs-assessment-spec.md b/docs/ai-assisted-techdocs-assessment-spec.md index 3061010..4345213 100644 --- a/docs/ai-assisted-techdocs-assessment-spec.md +++ b/docs/ai-assisted-techdocs-assessment-spec.md @@ -437,6 +437,46 @@ always readable from its issues and pull requests. pilot's cycle-time decomposition (section 10) is harvested from the GitHub timeline rather than a separate log. +## 13. Components and repository layout + +Everything the system is made of lives in cncf/techdocs, versioned and +reviewable like any other change; none of it exists yet. The operational layer +points at the methodology, it never restates it (P-2). + +- Intake template: `.github/ISSUE_TEMPLATE/assessment-request.yml`. The + structured request form (section 7), including the up-front AI disclosure + (HC-6). +- Repository instructions: `.github/copilot-instructions.md`. Ground rules for + any agent work in this repository. +- Agent profiles: `.github/agents/*.agent.md`. One per phase: the drafter's + task, inputs, and constraints (section 14). +- Environment setup: `.github/workflows/copilot-setup-steps.yml`. Prepares the + agent's ephemeral environment (section 11). +- Phase-advance workflow: `.github/workflows/assessment-phase.yml`. On merge, + opens the next phase's tracking issue (section 12). +- Provenance check: a workflow in `.github/workflows/` backed by + `scripts/assessment/`. CI that fails a deliverable PR whose provenance block + is missing or malformed (section 15). +- Data collection: `scripts/assessment/`. The deterministic inventory scripts + behind HC-5 (section 15). +- Labels: `.github/settings.yml`. An `assessment` label plus per-phase labels, + managed declaratively alongside the repository's existing label set. +- Deliverables: `analyses///`. The existing convention: + `analysis.md`, `implementation.md`, and the backlog. +- Backlog files: `analyses///issues/`. One file per proposed + issue plus an index, if the section 6 proposal is accepted. + +Notes: + +- The methodology corpus at [docs/analysis/][analysis-dir] is deliberately + absent from this list: the system reads it, but owns no file in it (P-2, + NG-4). This spec likewise lives outside the corpus. +- The backlog file naming across existing analyses already varies (`issues.md`, + `issues-list.md`); whatever the section 6 decision, the layout above pins one + convention going forward. +- MCP configuration is the one component not in the list because the platform + keeps it in repository settings, not a file (section 11). + ## 17. Open questions and future work - Filing issues into project repos. A separate, opt-in tool to create the From ac0ac1a7a75765e3ef6a36113a540db8c76f9f5e Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 10:54:27 -0700 Subject: [PATCH 09/47] Relabel platform owner to administrator/platform owner; drop arbitration Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/ai-assisted-techdocs-assessment-spec.md | 33 ++++++++++---------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/docs/ai-assisted-techdocs-assessment-spec.md b/docs/ai-assisted-techdocs-assessment-spec.md index 4345213..6ce7e73 100644 --- a/docs/ai-assisted-techdocs-assessment-spec.md +++ b/docs/ai-assisted-techdocs-assessment-spec.md @@ -130,8 +130,7 @@ the labor. AI drafts, humans decide, all in the open. errors, which is required to advance. Agreement with the conclusions: not required, and a project's disagreement is recorded in the deliverable rather than allowed to block or soften it. Merging phase N makes phase N+1 eligible; - a technical writer still starts it (section 5). A reviewer/stakeholder dispute - is arbitrated by the platform owner (section 4). + a technical writer still starts it (section 5). - HC-3: Deliverables are files in cncf/techdocs. Including the issue backlog (one file per proposed issue). Nothing is filed to an external repo. - HC-4: No unattended autonomy. Any change to repository state lands through a @@ -177,10 +176,10 @@ tool), and one actor may fill more than one role. that phase, so no one signs off on work they produced or refined, and the role spreads across the team instead of bottlenecking. Confirms the verification in section 10 was done. -- Platform owner. Owns the machine itself: the prompts, the operational layer, - and the configuration that enforces the write boundary (HC-1). Sets policy, - handles aborts (section 5), and arbitrates a reviewer/stakeholder dispute - (HC-2). A central role, kept out of the per-assessment critical path. +- Administrator/platform owner. Owns the machine itself: the prompts, the + operational layer, and the configuration that enforces the write boundary + (HC-1). Sets policy and handles aborts (section 5). A central role, kept out + of the per-assessment critical path. ## 5. Lifecycle @@ -201,9 +200,9 @@ Each phase runs the same six steps: Failure path: if a draft is bad enough that fixing it would cost more than starting over, the reviewer may discard it and restart or write by hand rather -than sink time into editing. Repeated failure is escalated to the platform -owner, who may abort the run. An assessment must never end up slower than doing -it by hand. +than sink time into editing. Repeated failure is escalated to the +administrator/platform owner, who may abort the run. An assessment must never +end up slower than doing it by hand. ## 6. Phases and deliverables @@ -361,18 +360,18 @@ infrastructure we build and maintain ourselves. covers package registries and GitHub content domains, not the general web, so reading an assessed project's live documentation site requires an explicit allowlist entry. We treat this as a feature: the intake (section 7) collects - the project's documentation domains, and the platform owner adds them at - acceptance, making each assessment's external reads an explicit, auditable, - reversible contract. Requests the firewall blocks are disclosed automatically - in the pull request, which feeds the safety audit in section 10. + the project's documentation domains, and the administrator/platform owner adds + them at acceptance, making each assessment's external reads an explicit, + auditable, reversible contract. Requests the firewall blocks are disclosed + automatically in the pull request, which feeds the safety audit in section 10. - MCP policy. The firewall does not apply to MCP servers, and MCP tools are the one documented mechanism that can widen the agent's write reach, so [MCP configuration][cloud-agent-mcp] is the control surface that matters most. Policy: the GitHub MCP server stays at its read-only default; any additional MCP server must be read-only with its tools explicitly allowlisted; no write-capable MCP tool is permitted. MCP configuration lives in repository - settings rather than in a versioned file, so the platform owner records the - current configuration in the operational docs whenever it changes. + settings rather than in a versioned file, so the administrator/platform owner + records the current configuration in the operational docs whenever it changes. - Execution environment. The agent runs in an ephemeral [GitHub Actions-based environment][cloud-agent-env] with a hard session cap (currently 59 minutes). That cap shapes the design: deterministic data collection (section 13) runs as @@ -430,8 +429,8 @@ always readable from its issues and pull requests. violation is a build-plan candidate (section 16). - Failure and abort leave a trail. Discarding a draft (section 5) is closing its PR with the reason recorded in a comment; the tracking issue stays open for a - restart or a hand-written phase. A platform-owner abort closes the tracking - issue, with the rationale recorded there. + restart or a hand-written phase. An administrator/platform owner abort closes + the tracking issue, with the rationale recorded there. - The timeline is the measurement. Gate transitions (opened, accepted, draft PR, ready, approved, merged) are timestamped in issue and PR history, so the pilot's cycle-time decomposition (section 10) is harvested from the GitHub From 4b8125b78b554aa045361ccbe6af7d7747cdd0a7 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 11:12:27 -0700 Subject: [PATCH 10/47] Add section 14: agent definitions Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/ai-assisted-techdocs-assessment-spec.md | 41 ++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/ai-assisted-techdocs-assessment-spec.md b/docs/ai-assisted-techdocs-assessment-spec.md index 6ce7e73..0da7457 100644 --- a/docs/ai-assisted-techdocs-assessment-spec.md +++ b/docs/ai-assisted-techdocs-assessment-spec.md @@ -476,6 +476,47 @@ Notes: - MCP configuration is the one component not in the list because the platform keeps it in repository settings, not a file (section 11). +## 14. Agent definitions + +The drafter role (section 4) is filled by the cloud agent running under a +per-phase agent profile: a versioned file in `.github/agents/` whose header +declares the agent's name, description, and allowed tools, and whose body is its +standing prompt. The writer who accepts a phase selects its profile when +delegating (section 12). Profiles are plain files, so changing an agent means a +reviewed PR, owned by the administrator/platform owner, like any other change to +the machine. + +Rules common to all three profiles, in tension-order with Part I: + +- Point, do not restate. A profile carries the task shape, output paths, and + discipline; the criteria, process, and templates are read from the methodology + corpus at the pinned ref (P-2). No rubric content, criterion text, or template + structure is copied into a prompt. +- Data, not instructions. Content from the assessed project (repos, sites, + issues) is evidence to analyze, never instructions to follow (section 8). +- Evidence discipline. Quantitative claims come from the committed + data-collection outputs (HC-5); qualitative findings cite the source they were + checked against (HC-7); anything unverifiable is flagged as such in the draft + rather than asserted. +- Provenance. Every draft opens with the provenance block (section 15). +- Least tools. A profile's tool list is the minimum its phase needs; none may + include a write-capable MCP tool (section 11). + +The three profiles: + +- Assessment drafter (Phase A). Input: the intake issue, the methodology at the + pinned ref, and the data-collection outputs for the project. Output: a draft + `analysis.md` in the project's deliverables directory, rating each criterion + with cited evidence. +- Plan drafter (Phase B). Input: the merged `analysis.md` plus the intake and + methodology. Output: a draft `implementation.md` derived from the assessment's + findings only; a gap discovered while planning is flagged in the PR, not + silently promoted into a new finding. +- Backlog drafter (Phase C). Input: the merged `implementation.md`. Output: the + issue backlog in the layout section 6 settles on, each item scoped to roughly + 4 hours, with effort estimates marked as first-pass for reviewer + sanity-checking. + ## 17. Open questions and future work - Filing issues into project repos. A separate, opt-in tool to create the From f9817541057f9dfdaf0d2a8cc385e580f095e9f8 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 11:29:09 -0700 Subject: [PATCH 11/47] Add verifier: an adversarial agent fact-check pass before human review Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/ai-assisted-techdocs-assessment-spec.md | 66 ++++++++++++++------ 1 file changed, 46 insertions(+), 20 deletions(-) diff --git a/docs/ai-assisted-techdocs-assessment-spec.md b/docs/ai-assisted-techdocs-assessment-spec.md index 0da7457..1b315f2 100644 --- a/docs/ai-assisted-techdocs-assessment-spec.md +++ b/docs/ai-assisted-techdocs-assessment-spec.md @@ -148,17 +148,31 @@ the labor. AI drafts, humans decide, all in the open. ## 4. Roles Roles are defined by function. Each is filled by an actor (a person, group, or -tool), and one actor may fill more than one role. +tool), and one actor may fill more than one role. Not every role may be filled +by a tool: the drafter is expected to be one and the verifier is one by +definition, while the reviewer and approver are human-only, because they hold +the human judgment that G-3, NG-3, and HC-2 require. Machine checks may assist +them; they may not replace them. - Requester. A stakeholder who initiates the assessment by filing the request (section 7); for example a maintainer, tech lead, or trusted community member. Filing does not start an assessment. - Drafter. Produces the first-pass draft of each phase's deliverable as a draft PR, then refines it in conversation with the reviewer. +- Verifier. An agent that runs an adversarial fact-check pass over each draft + before human review: resolving citations, checking claims against the sources + they cite, and flagging anything unsupported. It is prompted to find + unsupported claims, not to confirm the draft, so it does not inherit the + drafter's blind spots. Its report feeds the reviewer's verification (HC-7); it + never substitutes for it. The layering exists because a fabrication that + reaches a project's maintainers wastes their time and costs the system its + credibility. - Reviewer. Accepts a request to begin work, then reviews and refines each draft in conversation, verifies findings against source (HC-7), and marks it ready. Owns the draft's quality, but does not give its final sign-off; that is - independent (see Approver). + independent (see Approver). A human role: it starts from the verifier's report + and the machine checks (provenance lint, template conformance), but the + verification and the readiness call are the reviewer's own. - Stakeholders. The project party with a stake in its documentation and direction: maintainers, tech leads, trusted community members, and others. They set priorities, provide project-level context, and confirm factual @@ -189,9 +203,10 @@ Each phase runs the same six steps: become eligible once the previous phase is merged. 2. Accept. A technical writer triages and explicitly accepts the request, for every phase, not just A; eligibility alone does not start work (P-1). -3. Draft. The agent produces the deliverable as a draft PR. -4. Review. The reviewer refines the draft in conversation with the agent, - verifies findings against source (HC-7), and marks it ready. +3. Draft. The drafter produces the deliverable as a draft PR. +4. Review. The verifier's fact-check pass runs first. The reviewer then refines + the draft in conversation with the drafter, verifies findings against source + (HC-7), and marks it ready. 5. Stakeholder review. Stakeholders confirm factual accuracy (required to advance) and record any disagreement with the conclusions (not required; HC-2). @@ -409,11 +424,13 @@ always readable from its issues and pull requests. - Draft (step 3): a draft pull request. The agent works on its own branch in cncf/techdocs and opens a draft PR linked to the tracking issue, carrying the provenance block (section 15). -- Review (step 4): PR review comments. The reviewer refines the draft in - conversation, handing revisions back by mentioning `@copilot` in review - comments, and performs the verification required by section 10. Whether - mention-triggered revisions require write access is a build-time check - (section 17). +- Review (step 4): PR review comments. The reviewer triggers the verifier's pass + on the draft PR; its report lands as a PR comment, part of the same public + record. The reviewer then refines the draft in conversation, handing revisions + back by mentioning `@copilot` in review comments, and performs the + verification required by section 10. Whether mention-triggered revisions + require write access, and how a verifier run is invoked against an existing + PR, are build-time checks (section 17). - Stakeholder review (step 5): mention, not access. The reviewer marks the PR ready for review and mentions the stakeholders named in the intake. cncf/techdocs is public, so stakeholders can review and comment without any @@ -447,8 +464,8 @@ points at the methodology, it never restates it (P-2). (HC-6). - Repository instructions: `.github/copilot-instructions.md`. Ground rules for any agent work in this repository. -- Agent profiles: `.github/agents/*.agent.md`. One per phase: the drafter's - task, inputs, and constraints (section 14). +- Agent profiles: `.github/agents/*.agent.md`. One per phase for drafting, plus + one for verification: each agent's task, inputs, and constraints (section 14). - Environment setup: `.github/workflows/copilot-setup-steps.yml`. Prepares the agent's ephemeral environment (section 11). - Phase-advance workflow: `.github/workflows/assessment-phase.yml`. On merge, @@ -478,15 +495,15 @@ Notes: ## 14. Agent definitions -The drafter role (section 4) is filled by the cloud agent running under a -per-phase agent profile: a versioned file in `.github/agents/` whose header +The drafter and verifier roles (section 4) are filled by the cloud agent running +under an agent profile: a versioned file in `.github/agents/` whose header declares the agent's name, description, and allowed tools, and whose body is its -standing prompt. The writer who accepts a phase selects its profile when -delegating (section 12). Profiles are plain files, so changing an agent means a -reviewed PR, owned by the administrator/platform owner, like any other change to -the machine. +standing prompt. The writer who accepts a phase selects its drafting profile +when delegating (section 12). Profiles are plain files, so changing an agent +means a reviewed PR, owned by the administrator/platform owner, like any other +change to the machine. -Rules common to all three profiles, in tension-order with Part I: +Rules common to all profiles, in tension-order with Part I: - Point, do not restate. A profile carries the task shape, output paths, and discipline; the criteria, process, and templates are read from the methodology @@ -502,7 +519,7 @@ Rules common to all three profiles, in tension-order with Part I: - Least tools. A profile's tool list is the minimum its phase needs; none may include a write-capable MCP tool (section 11). -The three profiles: +The three drafting profiles: - Assessment drafter (Phase A). Input: the intake issue, the methodology at the pinned ref, and the data-collection outputs for the project. Output: a draft @@ -517,6 +534,15 @@ The three profiles: 4 hours, with effort estimates marked as first-pass for reviewer sanity-checking. +The verification profile: + +- Verifier (every phase). Input: the draft PR's deliverable and the sources it + cites. Output: a report on the PR listing each checked claim as supported, + unsupported, or unverifiable, with the failing ones quoted. Its prompt is + adversarial (find unsupported claims), not confirmatory (check the draft is + fine), so it does not share the drafter's failure mode (G-2). Its tool list is + read-only; it edits nothing. + ## 17. Open questions and future work - Filing issues into project repos. A separate, opt-in tool to create the From f05c64051c5ec718cac1db6f9f7d78b2be020675 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 11:45:38 -0700 Subject: [PATCH 12/47] Add section 15: the provenance block Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/ai-assisted-techdocs-assessment-spec.md | 51 ++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/docs/ai-assisted-techdocs-assessment-spec.md b/docs/ai-assisted-techdocs-assessment-spec.md index 1b315f2..eeb1276 100644 --- a/docs/ai-assisted-techdocs-assessment-spec.md +++ b/docs/ai-assisted-techdocs-assessment-spec.md @@ -543,6 +543,55 @@ The verification profile: fine), so it does not share the drafter's failure mode (G-2). Its tool list is read-only; it edits nothing. +## 15. The provenance block + +The provenance block is where Part I's disclosure and evidence duties converge +(HC-5, HC-6, HC-7, P-2): a fixed-format section at the top of every +deliverable's body, immediately after the title. It is body content rather than +front matter because of how deliverables are published: cncf/techdocs is the +source of truth, and its content is synced at build time into the +[contribute.cncf.io site][contribute-site-sync], whose rebuild a merge to main +triggers. Body content survives that sync verbatim and renders wherever the +deliverable is read (the repository, the site, a fork); front matter is consumed +as metadata and does not reach the page. HC-6 requires that no project discover +AI involvement after the fact, so the disclosure rides in the content itself. As +plain markdown (a heading and labeled bullets) it needs nothing from any +renderer, and the fixed labels make it machine-checkable. + +Its fields, each a labeled bullet: + +- Disclosure. A fixed sentence: this deliverable was drafted by an AI agent and + reviewed, verified, and approved by humans (HC-6). +- Drafter. The agent profile that produced the draft, by name and ref (section + 14). +- Verifier. The verifier profile, by name and ref, and a link to its report on + the PR (section 14). +- Methodology. The ref of the methodology corpus the draft was produced against + (P-2), so the deliverable pins the criteria and templates it was measured by. +- Data. The data-collection commands that were run and the committed paths of + their outputs, so every quantitative claim traces to a reproducible step + (HC-5). +- Verification record. Which rating-bearing findings the reviewer verified + against source, satisfying the section 10 floor (HC-7). + +The fields have owners. The drafter emits the block with everything it can know: +disclosure, its own profile, the methodology ref, and the data paths. The +verifier field and the verification record belong to the review step: the report +link is added once the verifier's pass runs, and the record starts as a marked +placeholder that only the reviewer fills; a drafter asserting its own work +verified would be a fabrication about fabrication-checking. + +The provenance check (section 13) enforces the block in CI: a deliverable PR +fails if the block is missing, a field is absent or malformed, the methodology +ref does not resolve, or a listed data path is not in the tree. While the PR is +a draft, the verification record may be a placeholder; once it is marked ready +for review, the check runs strict and an unfilled record fails. Judgment stays +human: CI proves the block is present and well-formed, the approver confirms the +verification behind it was real (section 10). + +The block is defined by this spec and layered above the template body, so the +methodology corpus and its templates are not modified (P-2, NG-4). + ## 17. Open questions and future work - Filing issues into project repos. A separate, opt-in tool to create the @@ -569,6 +618,8 @@ The verification profile: [criteria]: https://github.com/cncf/techdocs/blob/main/docs/analysis/criteria.md [howto]: https://github.com/cncf/techdocs/blob/main/docs/analysis/howto.md [analysis-dir]: https://github.com/cncf/techdocs/tree/main/docs/analysis +[contribute-site-sync]: + https://github.com/cncf/techdocs/blob/main/.github/workflows/trigger-contribute-site-netlify.yml [copilot-blog]: https://contribute.cncf.io/blog/2025/12/16/github-copilot-enterprise-for-maintainers/ [cloud-agent-about]: From 0abc4656b906d08c4f7e08784d19272668b1df20 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 11:51:30 -0700 Subject: [PATCH 13/47] Move spec to specs/, out of the website-synced tree The contribute.cncf.io site syncs docs/ and analyses/ at build time. The website is a curated surface for contributors and projects; a build specification is repo-audience material, so it moves out of the synced tree. Publication transparency is unchanged: the spec remains public in the repository. Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- {docs => specs}/ai-assisted-techdocs-assessment-spec.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {docs => specs}/ai-assisted-techdocs-assessment-spec.md (100%) diff --git a/docs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md similarity index 100% rename from docs/ai-assisted-techdocs-assessment-spec.md rename to specs/ai-assisted-techdocs-assessment-spec.md From 96f2fec97fdf81f4337ecbdae654239e45a3e69a Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 11:56:14 -0700 Subject: [PATCH 14/47] Note the website intake page as future work Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index eeb1276..bf941fd 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -599,7 +599,9 @@ methodology corpus and its templates are not modified (P-2, NG-4). one, and it must preserve HC-1. - Intake relationship. How the request template fits with the existing CNCF service desk and assistance-program intake (section 7), without a competing - front door. + front door. Related: the contribute.cncf.io site needs a page for projects on + requesting an assessment and what to expect, including the AI disclosure, + updated as this process changes. - AI-readiness criteria. Whether and how the criteria evolve for AI consumption (for example llms.txt), tracked in PR #357. - Pilot measurement. How we capture the G-1 time decomposition and cycle-time From 86680b0878ffce85238a9e0ebc2b2ef705bf7f1d Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 12:02:25 -0700 Subject: [PATCH 15/47] Add section 16: the build plan Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index bf941fd..a9156fd 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -592,6 +592,61 @@ verification behind it was real (section 10). The block is defined by this spec and layered above the template body, so the methodology corpus and its templates are not modified (P-2, NG-4). +## 16. Build plan + +The system is built the way it runs. Each component in section 13 starts as a +tracking issue in cncf/techdocs, is drafted by the cloud agent where a file is +the deliverable, and lands through a human-reviewed PR: the same +issue-to-draft-to-review loop the assessments will use (section 5). Building the +system with its own loop is the point: by the time the first project is +assessed, every role has rehearsed and the platform's behavior has been observed +rather than assumed. The earliest steps run before the instructions and +environment components exist; the loop holds anyway, just with less scaffolding. + +Preflight, before any build step: the administrator/platform owner works through +the preconditions in section 11: cloud agent and MCP policies enabled for +cncf/techdocs, MCP left at its read-only default, writer access confirmed, and +the credit cap reviewed. These are organization and repository settings, not +files, so this step is recorded in its tracking issue rather than a PR. + +The build steps, in dependency order, each sized to one issue: + +1. Labels (`.github/settings.yml`). Deliberately trivial first delegation: its + real product is observed platform behavior: the agent's branch naming, what + access review-comment revisions require, whether Actions runs on agent PRs + wait for approval, and the token's effective scopes. Done when the labels + exist and the observations are recorded against the section 17 checks. +2. Repository instructions (`.github/copilot-instructions.md`). Ground rules for + any agent work in this repository, carrying the section 8 discipline. Done + when merged after review against sections 8 and 14. +3. Intake template (`.github/ISSUE_TEMPLATE/assessment-request.yml`). Done when + a dry-run request files cleanly, collecting the section 7 fields, the + project's documentation domains for the firewall allowlist (section 11), and + the up-front AI disclosure (HC-6). +4. Environment setup (`.github/workflows/copilot-setup-steps.yml`). Done when an + agent session's logs show the prepared environment. +5. Data-collection scripts (`scripts/assessment/`). Done when running the + committed command against a sample project twice yields the same committed + outputs (HC-5). +6. Provenance check (workflow plus `scripts/assessment/`). Done when a fixture + PR with a malformed or unfilled block fails, a well-formed one passes, and + the draft-versus-ready leniency behaves as section 15 specifies. +7. Agent profiles (`.github/agents/`, one PR per profile). The three drafters, + then the verifier (section 14). Done when each profile, run against a + fixture, writes only its declared outputs with the provenance block present, + and the verifier's report correctly flags a planted unsupported claim. +8. Phase-advance workflow (`.github/workflows/assessment-phase.yml`). Done when + a merged fixture PR opens the next phase's tracking issue, linked and + unassigned (section 12). +9. Approver-independence check (advisory; a build-plan candidate from section + 12). Comments when a phase's approver also drafted or reviewed it; never + blocks. Done when it flags a staged violation, or explicitly deferred. + +The build is complete when steps 1 through 8 are merged, the section 17 +build-time checks have recorded answers, and one end-to-end walkthrough of the +section 5 lifecycle on a fixture project has run clean. Then the pilot is +chosen, deliberately, per the caveat in section 10. + ## 17. Open questions and future work - Filing issues into project repos. A separate, opt-in tool to create the From 04a0eaf14a961cc9ae36547d9e1d88d366a116c9 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 12:06:51 -0700 Subject: [PATCH 16/47] Refresh open questions: collect build-time checks, drop answered item Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index a9156fd..72cd0cb 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -649,6 +649,12 @@ chosen, deliberately, per the caveat in section 10. ## 17. Open questions and future work +- Build-time checks. Platform behavior we could not verify from public + documentation, to be answered by observation during the build (section 16) and + recorded: the agent token's effective permission scopes (section 11); the + agent's branch naming; whether review-comment revisions require write access + (section 12); whether Actions runs on agent PRs wait for approval; and how a + verifier run is invoked against an existing PR (section 12). - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase one, and it must preserve HC-1. @@ -659,8 +665,6 @@ chosen, deliberately, per the caveat in section 10. updated as this process changes. - AI-readiness criteria. Whether and how the criteria evolve for AI consumption (for example llms.txt), tracked in PR #357. -- Pilot measurement. How we capture the G-1 time decomposition and cycle-time - data to test the 2-week hypothesis (section 10). - Assessment-quality rubric. Define the meta-rubric that scores an assessment's quality, distinct from `criteria.md` (which scores a project's docs), and validate it by scoring the Flatcar, Knative, and Helm baselines to set a From 95c928c35cf8705d95e23891e32728f971992a0b Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 14:18:43 -0700 Subject: [PATCH 17/47] Preserve the methodology's phase B skip path and issue time bound Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 63 ++++++++++++------- 1 file changed, 39 insertions(+), 24 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 72cd0cb..fff8448 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -57,8 +57,10 @@ the labor. AI drafts, humans decide, all in the open. project with minimal per-project setup. - G-6: Lower the request barrier. A structured intake lets a project (or the TechDocs team) request an assessment with all needed context up front. -- G-7: Produce the three standard deliverables. Assessment, implementation plan, - and an issue backlog, matching the existing methodology. +- G-7: Produce the standard deliverables. Assessment, implementation plan, and + an issue backlog, matching the existing methodology, including its skip path: + small projects whose recommendations are already independent and time-bound + may go straight from assessment to backlog ([howto]). ### Non-goals (phase one) @@ -199,8 +201,9 @@ them; they may not replace them. Each phase runs the same six steps: -1. Request. Phase A is triggered by the intake issue (section 7); phases B and C - become eligible once the previous phase is merged. +1. Request. Phase A is triggered by the intake issue (section 7); the next phase + becomes eligible once the previous one is merged (phase C directly after A + when phase B is skipped; section 6). 2. Accept. A technical writer triages and explicitly accepts the request, for every phase, not just A; eligibility alone does not start work (P-1). 3. Draft. The drafter produces the deliverable as a draft PR. @@ -222,12 +225,16 @@ end up slower than doing it by hand. ## 6. Phases and deliverables - Phase A: Assessment (`analysis.md`). Detailed first. -- Phase B: Implementation plan (`implementation.md`). -- Phase C: Issue backlog. Each proposed issue is a separate file scoped to - roughly 4 hours for someone experienced with the project and comfortable - writing, plus an umbrella/index file that lists them. Effort estimates are the - agent's first pass and are sanity-checked by the reviewer; agent estimates are - not reliable on their own. +- Phase B: Implementation plan (`implementation.md`). Per the methodology, this + phase may be skipped for a small project whose recommendations are already + independent and time-bound; the reviewer and stakeholders make that call + during phase A's review and record it on the phase A PR before merge. One of + the human baselines (Knative) took this path. +- Phase C: Issue backlog. Each proposed issue is a separate file scoped to the + methodology's time bound (a few hours to a couple of days at most for someone + experienced with the project and comfortable writing), plus an umbrella/index + file that lists them. Effort estimates are the agent's first pass and are + sanity-checked by the reviewer; agent estimates are not reliable on their own. Proposed methodology change. The current method delivers the backlog as one `_PROJECT_-issues.md` file (`howto.md`, `templates/issues-list.md`, @@ -297,16 +304,18 @@ its documented guarantees are specified in Part II (section 11). Every requirement here derives from the existing methodology, so the system automates the current method rather than inventing a new one (P-2): -- The three deliverables (G-7) and the phase order (section 5) follow the - [howto]: analysis, then implementation plan, then issue backlog. +- The deliverables (G-7) and the phase order (section 5) follow the [howto]: + analysis, then implementation plan, then issue backlog, keeping the howto's + own skip path for the implementation plan (section 6). - Assessment content, ratings, and criteria come from `criteria.md`. - Each deliverable's structure comes from the templates in `docs/analysis/templates/` (`analysis.md`, `implementation.md`, `issues-list.md`, `issue.md`). - Scope and maturity framing (section 7) come from the analysis template's "About" and "Scope" sections and the criteria's maturity levels. -- The 4-hour, independent, time-bounded issue scoping (Phase C) follows the - howto and assistance-program guidance on backlog creation. +- The independent, time-bounded issue scoping (Phase C, a few hours to a couple + of days per issue) follows the howto and assistance-program guidance on + backlog creation. Prior prototyping has explored parts of this workflow; that experience will inform implementation. @@ -329,7 +338,8 @@ The system is acceptable when, on a pilot assessment: quiet redefinition. - Safety. Zero writes outside cncf/techdocs, audited from the platform's activity records, and no unmitigated prompt-injection incident. -- Completeness and reproducibility. All three deliverables produced; every +- Completeness and reproducibility. Every phase's deliverable produced (all + three, or two where the implementation plan was skipped per section 6); every quantitative claim reproducible from a committed step (HC-5); AI involvement disclosed (HC-6). @@ -441,8 +451,10 @@ always readable from its issues and pull requests. merge-triggered GitHub Actions workflow (standard `GITHUB_TOKEN` with `issues: write`) then opens the next phase's tracking issue, linking the intake and the merged deliverable, and assigns no one: eligible, not started - (HC-2, P-1). Approver independence cannot be natively enforced by GitHub; it - is verifiable from the public PR record, and an advisory CI check that flags a + (HC-2, P-1). The phase B skip decision (section 6) is recorded as a label on + the phase A PR, which the workflow reads to open phase C's tracking issue + instead. Approver independence cannot be natively enforced by GitHub; it is + verifiable from the public PR record, and an advisory CI check that flags a violation is a build-plan candidate (section 16). - Failure and abort leave a trail. Discarding a draft (section 5) is closing its PR with the reason recorded in a comment; the tracking issue stays open for a @@ -475,8 +487,9 @@ points at the methodology, it never restates it (P-2). is missing or malformed (section 15). - Data collection: `scripts/assessment/`. The deterministic inventory scripts behind HC-5 (section 15). -- Labels: `.github/settings.yml`. An `assessment` label plus per-phase labels, - managed declaratively alongside the repository's existing label set. +- Labels: `.github/settings.yml`. An `assessment` label, per-phase labels, and + the phase B skip marker (section 12), managed declaratively alongside the + repository's existing label set. - Deliverables: `analyses///`. The existing convention: `analysis.md`, `implementation.md`, and the backlog. - Backlog files: `analyses///issues/`. One file per proposed @@ -529,10 +542,11 @@ The three drafting profiles: methodology. Output: a draft `implementation.md` derived from the assessment's findings only; a gap discovered while planning is flagged in the PR, not silently promoted into a new finding. -- Backlog drafter (Phase C). Input: the merged `implementation.md`. Output: the - issue backlog in the layout section 6 settles on, each item scoped to roughly - 4 hours, with effort estimates marked as first-pass for reviewer - sanity-checking. +- Backlog drafter (Phase C). Input: the merged `implementation.md`, or the + merged `analysis.md` where phase B was skipped (section 6). Output: the issue + backlog in the layout section 6 settles on, each item scoped to the + methodology's time bound (section 6), with effort estimates marked as + first-pass for reviewer sanity-checking. The verification profile: @@ -637,7 +651,8 @@ The build steps, in dependency order, each sized to one issue: and the verifier's report correctly flags a planted unsupported claim. 8. Phase-advance workflow (`.github/workflows/assessment-phase.yml`). Done when a merged fixture PR opens the next phase's tracking issue, linked and - unassigned (section 12). + unassigned, on both the default route and the phase B skip route (section + 12). 9. Approver-independence check (advisory; a build-plan candidate from section 12). Comments when a phase's approver also drafted or reviewed it; never blocks. Done when it flags a staged violation, or explicitly deferred. From c2ae7ac631097f49f0e509ab389f2b699abdfaec Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 14:29:26 -0700 Subject: [PATCH 18/47] Report cycle time; do not gate acceptance on it Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index fff8448..c186e70 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -42,7 +42,9 @@ the labor. AI drafts, humans decide, all in the open. ### Goals - G-1: Compress the cycle. Reduce the 6–8 week assessment toward roughly 2 - weeks, a pilot hypothesis to validate rather than a commitment (section 10). + weeks. This is the motivation and a pilot hypothesis, not an acceptance + criterion (section 10): the pilot succeeds or fails on quality and safety, and + the elapsed time simply gets reported. - G-2: Hold quality. Output meets or exceeds the current human baseline (Flatcar, Knative, Helm), verified by the method in section 10, not asserted. AI drafts fail by being fluent, well-formatted, and generically wrong, a mode @@ -332,10 +334,6 @@ The system is acceptable when, on a pilot assessment: the approver (section 4), who was neither the drafter nor a reviewer of that phase, to avoid signing off on one's own work. The bar is parity with the human baselines (Flatcar, Knative, Helm). -- Cycle time. Measured against the time decomposition from G-1 (writer working - time versus waiting on people). The roughly 2-week target is evaluated as a - hypothesis; missing it prompts a look at which gates or waits dominate, not a - quiet redefinition. - Safety. Zero writes outside cncf/techdocs, audited from the platform's activity records, and no unmitigated prompt-injection incident. - Completeness and reproducibility. Every phase's deliverable produced (all @@ -460,10 +458,11 @@ always readable from its issues and pull requests. PR with the reason recorded in a comment; the tracking issue stays open for a restart or a hand-written phase. An administrator/platform owner abort closes the tracking issue, with the rationale recorded there. -- The timeline is the measurement. Gate transitions (opened, accepted, draft PR, - ready, approved, merged) are timestamped in issue and PR history, so the - pilot's cycle-time decomposition (section 10) is harvested from the GitHub - timeline rather than a separate log. +- The timeline records elapsed time. Gate transitions (opened, accepted, draft + PR, ready, approved, merged) are timestamped in issue and PR history, so how + long a pilot took, and where the time went, can be read from the GitHub + timeline after the fact. This reports on G-1 without adding a log or an + acceptance gate. ## 13. Components and repository layout From 95729f769a81a48648d6dc15a38d815af39f1c02 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 14:39:01 -0700 Subject: [PATCH 19/47] State model policy as capability requirements, not model names Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index c186e70..6dc27cb 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -556,6 +556,17 @@ The verification profile: fine), so it does not share the drafter's failure mode (G-2). Its tool list is read-only; it edits nothing. +Model and effort. Profiles state capability requirements, never model names: +models change faster than this spec. Drafting warrants the most capable model +and effort available to the repository; verification does not need the drafter's +configuration, and running the verifier on a different model is preferable where +the platform allows it, since two models are less likely to share one blind +spot. The platform exposes model choice as a per-task picker at delegation time, +so the writer applies this policy when delegating; whether a profile can pin its +own model is a build-time check (section 17). Which models are available to +cncf/techdocs is organization policy, handled with the CNCF GitHub organization +administrators by the administrator/platform owner (section 4). + ## 15. The provenance block The provenance block is where Part I's disclosure and evidence duties converge @@ -667,8 +678,10 @@ chosen, deliberately, per the caveat in section 10. documentation, to be answered by observation during the build (section 16) and recorded: the agent token's effective permission scopes (section 11); the agent's branch naming; whether review-comment revisions require write access - (section 12); whether Actions runs on agent PRs wait for approval; and how a - verifier run is invoked against an existing PR (section 12). + (section 12); whether Actions runs on agent PRs wait for approval; how a + verifier run is invoked against an existing PR (section 12); and whether an + agent profile can pin a model and effort level, or model choice rides entirely + on the per-task picker (section 14). - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase one, and it must preserve HC-1. From 4feb4f6e643237660a8e9426dbb9ab00dfcf6303 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 14:42:05 -0700 Subject: [PATCH 20/47] Correct platform claims: PR cap, firewall scope, secrets vector Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 45 +++++++++++++------ 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 6dc27cb..e1c0336 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -371,13 +371,17 @@ containment model satisfies HC-1 as shipped, rather than requiring credential infrastructure we build and maintain ourselves. - The write boundary (HC-1). The cloud agent can only make changes in the - repository where its task was started, on a single branch, through a single - pull request per task, using a token the platform issues for that run. The - built-in GitHub MCP server connects with read-only access to the current - repository by default. Together these provide the enforced write boundary Part - I requires; we mint and manage no credential of our own. The platform does not - publicly enumerate the agent token's exact permission scopes, so confirming - them empirically is a build-time check (section 17). + repository where its task was started, on a single branch, using a token the + platform issues for that run. It can open at most one pull request per task, + and a run can also end leaving changes on its branch with no pull request at + all, so "every change reaches main through a reviewed PR" is enforced by this + spec's lifecycle (HC-4, section 12), not by the platform alone. The built-in + GitHub MCP server connects with read-only access to the current repository by + default. Together these provide the enforced write boundary Part I requires; + we mint and manage no credential of our own. The platform does not publicly + enumerate the agent token's exact permission scopes, so confirming them + empirically is a build-time check (section 17), as is confirming that + delegation reliably opens a draft pull request. - Outbound network. The agent's internet access is governed by a [default-deny firewall][cloud-agent-firewall] with an allowlist. The default allowlist covers package registries and GitHub content domains, not the general web, so @@ -387,14 +391,26 @@ infrastructure we build and maintain ourselves. them at acceptance, making each assessment's external reads an explicit, auditable, reversible contract. Requests the firewall blocks are disclosed automatically in the pull request, which feeds the safety audit in section 10. -- MCP policy. The firewall does not apply to MCP servers, and MCP tools are the - one documented mechanism that can widen the agent's write reach, so [MCP - configuration][cloud-agent-mcp] is the control surface that matters most. - Policy: the GitHub MCP server stays at its read-only default; any additional - MCP server must be read-only with its tools explicitly allowlisted; no - write-capable MCP tool is permitted. MCP configuration lives in repository + The firewall's scope is the agent's own session: the environment-setup + workflow (see Execution environment below) runs outside it, so what setup + fetches is governed by review of that versioned workflow file, not by the + firewall. +- MCP policy. The firewall does not apply to MCP servers, and MCP tools are one + of the two documented mechanisms that can widen the agent's write reach (a + secret is the other; see the next bullet), so [MCP + configuration][cloud-agent-mcp] is a control surface requiring explicit + policy. Policy: the GitHub MCP server stays at its read-only default; any + additional MCP server must be read-only with its tools explicitly allowlisted; + no write-capable MCP tool is permitted. MCP configuration lives in repository settings rather than in a versioned file, so the administrator/platform owner records the current configuration in the operational docs whenever it changes. +- Secrets policy. Repository or organization secrets can be made available to + the agent's environment during setup and execution. A secret carrying a + credential (a personal access token, for example) would hand the agent write + reach past HC-1's boundary. Policy: the agent's environment gets no secrets + beyond what the platform itself requires; any exception is a recorded + administrator/platform owner decision, like MCP changes, and a write-capable + credential is never acceptable. - Execution environment. The agent runs in an ephemeral [GitHub Actions-based environment][cloud-agent-env] with a hard session cap (currently 59 minutes). That cap shapes the design: deterministic data collection (section 13) runs as @@ -679,7 +695,8 @@ chosen, deliberately, per the caveat in section 10. recorded: the agent token's effective permission scopes (section 11); the agent's branch naming; whether review-comment revisions require write access (section 12); whether Actions runs on agent PRs wait for approval; how a - verifier run is invoked against an existing PR (section 12); and whether an + verifier run is invoked against an existing PR (section 12); whether + delegation reliably opens a draft pull request (section 11); and whether an agent profile can pin a model and effort level, or model choice rides entirely on the per-task picker (section 14). - Filing issues into project repos. A separate, opt-in tool to create the From 3a8e7f21eb78249172d73eab16dc059b6e9eae45 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 14:52:43 -0700 Subject: [PATCH 21/47] Make the disclosure status-aware; pin one methodology per assessment Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 37 ++++++++++++------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index e1c0336..eaf6bbe 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -456,11 +456,14 @@ always readable from its issues and pull requests. require write access, and how a verifier run is invoked against an existing PR, are build-time checks (section 17). - Stakeholder review (step 5): mention, not access. The reviewer marks the PR - ready for review and mentions the stakeholders named in the intake. - cncf/techdocs is public, so stakeholders can review and comment without any - special access; formal review requests are limited to collaborators, which is - why the binding is a mention plus a factual-accuracy confirmation recorded on - the PR, with any disagreement recorded in the deliverable itself (HC-2). + ready for review and mentions the stakeholders named in the intake. The + draft-to-ready flip carries the real-world meaning: draft while the assessing + team is still working the deliverable over, ready when they would put it in + front of the requesting project. cncf/techdocs is public, so stakeholders can + review and comment without any special access; formal review requests are + limited to collaborators, which is why the binding is a mention plus a + factual-accuracy confirmation recorded on the PR, with any disagreement + recorded in the deliverable itself (HC-2). - Merge (step 6): merge plus one workflow. The approver (section 4) merges. A merge-triggered GitHub Actions workflow (standard `GITHUB_TOKEN` with `issues: write`) then opens the next phase's tracking issue, linking the @@ -600,14 +603,19 @@ renderer, and the fixed labels make it machine-checkable. Its fields, each a labeled bullet: -- Disclosure. A fixed sentence: this deliverable was drafted by an AI agent and - reviewed, verified, and approved by humans (HC-6). +- Disclosure. A fixed sentence with two forms matching the deliverable's state + (HC-6). The draft form: drafted by an AI agent, human review pending. The + final form: drafted by an AI agent and reviewed, verified, and approved by + humans. A draft asserting the final form would be claiming a review that has + not happened yet. - Drafter. The agent profile that produced the draft, by name and ref (section 14). - Verifier. The verifier profile, by name and ref, and a link to its report on the PR (section 14). - Methodology. The ref of the methodology corpus the draft was produced against (P-2), so the deliverable pins the criteria and templates it was measured by. + The ref is set at phase A and carried unchanged by every later phase's + deliverable: one assessment is measured against one methodology. - Data. The data-collection commands that were run and the committed paths of their outputs, so every quantitative claim traces to a reproducible step (HC-5). @@ -619,15 +627,18 @@ disclosure, its own profile, the methodology ref, and the data paths. The verifier field and the verification record belong to the review step: the report link is added once the verifier's pass runs, and the record starts as a marked placeholder that only the reviewer fills; a drafter asserting its own work -verified would be a fabrication about fabrication-checking. +verified would be a fabrication about fabrication-checking. The disclosure's +flip from draft form to final form belongs to review for the same reason: it +happens when the PR is marked ready. The provenance check (section 13) enforces the block in CI: a deliverable PR fails if the block is missing, a field is absent or malformed, the methodology -ref does not resolve, or a listed data path is not in the tree. While the PR is -a draft, the verification record may be a placeholder; once it is marked ready -for review, the check runs strict and an unfilled record fails. Judgment stays -human: CI proves the block is present and well-formed, the approver confirms the -verification behind it was real (section 10). +ref does not resolve or differs from the pin set at phase A, or a listed data +path is not in the tree. While the PR is a draft, the verification record may be +a placeholder and the disclosure carries its draft form; once it is marked ready +for review, the check runs strict: an unfilled record or a draft-form disclosure +fails. Judgment stays human: CI proves the block is present and well-formed, the +approver confirms the verification behind it was real (section 10). The block is defined by this spec and layered above the template body, so the methodology corpus and its templates are not modified (P-2, NG-4). From fb6a5f4eadcbf28247a463df62fce8568862dabb Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 15:18:37 -0700 Subject: [PATCH 22/47] Pin the verification floor; snapshot agent config from the API Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 62 ++++++++++++------- 1 file changed, 41 insertions(+), 21 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index eaf6bbe..2eb178b 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -166,11 +166,11 @@ them; they may not replace them. - Verifier. An agent that runs an adversarial fact-check pass over each draft before human review: resolving citations, checking claims against the sources they cite, and flagging anything unsupported. It is prompted to find - unsupported claims, not to confirm the draft, so it does not inherit the - drafter's blind spots. Its report feeds the reviewer's verification (HC-7); it - never substitutes for it. The layering exists because a fabrication that - reaches a project's maintainers wastes their time and costs the system its - credibility. + unsupported claims, not to confirm the draft, making it less likely to inherit + the drafter's blind spots. Its report feeds the reviewer's verification + (HC-7); it never substitutes for it. The layering exists because a fabrication + that reaches a project's maintainers wastes their time and costs the system + its credibility. - Reviewer. Accepts a request to begin work, then reviews and refines each draft in conversation, verifies findings against source (HC-7), and marks it ready. Owns the draft's quality, but does not give its final sign-off; that is @@ -328,12 +328,13 @@ The system is acceptable when, on a pilot assessment: - Quality. The deliverable is scored against an assessment-quality rubric (its definition is an open item; section 17). Verification is not a token sample: - the reviewer checks every rating-bearing finding, or at minimum a set number - per criterion, biased toward the highest-risk claims, and records in the - deliverable which findings were verified (HC-7). Final sign-off is given by - the approver (section 4), who was neither the drafter nor a reviewer of that - phase, to avoid signing off on one's own work. The bar is parity with the - human baselines (Flatcar, Knative, Helm). + for the pilot, the reviewer checks every rating-bearing finding, biased first + toward the highest-risk claims, and records in the deliverable which findings + were verified (HC-7); any cheaper sampling rule is future work, set with the + rubric (section 17). Final sign-off is given by the approver (section 4), who + was neither the drafter nor a reviewer of that phase, to avoid signing off on + one's own work. The bar is parity with the human baselines (Flatcar, Knative, + Helm). - Safety. Zero writes outside cncf/techdocs, audited from the platform's activity records, and no unmitigated prompt-injection incident. - Completeness and reproducibility. Every phase's deliverable produced (all @@ -402,8 +403,9 @@ infrastructure we build and maintain ourselves. policy. Policy: the GitHub MCP server stays at its read-only default; any additional MCP server must be read-only with its tools explicitly allowlisted; no write-capable MCP tool is permitted. MCP configuration lives in repository - settings rather than in a versioned file, so the administrator/platform owner - records the current configuration in the operational docs whenever it changes. + settings rather than in a versioned file, so its effective state is captured + by the agent-configuration snapshot (section 13), and settings changes appear + in the organization audit log. - Secrets policy. Repository or organization secrets can be made available to the agent's environment during setup and execution. A secret carrying a credential (a personal access token, for example) would hand the agent write @@ -508,6 +510,14 @@ points at the methodology, it never restates it (P-2). - Labels: `.github/settings.yml`. An `assessment` label, per-phase labels, and the phase B skip marker (section 12), managed declaratively alongside the repository's existing label set. +- Agent-configuration snapshot: `scripts/assessment/`. The agent's effective + configuration (MCP servers, firewall state, and custom allowlist) is readable + from a documented endpoint, so a deterministic script captures it alongside + each assessment's data outputs (HC-5): what the agent could reach when a draft + was produced is committed evidence. The organization audit log remains the + authoritative history of who changed a setting and when. The endpoint does not + expose organization-level allowlist entries; the administrator/platform owner + records those by hand when they exist. - Deliverables: `analyses///`. The existing convention: `analysis.md`, `implementation.md`, and the backlog. - Backlog files: `analyses///issues/`. One file per proposed @@ -570,10 +580,13 @@ The verification profile: - Verifier (every phase). Input: the draft PR's deliverable and the sources it cites. Output: a report on the PR listing each checked claim as supported, - unsupported, or unverifiable, with the failing ones quoted. Its prompt is - adversarial (find unsupported claims), not confirmatory (check the draft is - fine), so it does not share the drafter's failure mode (G-2). Its tool list is - read-only; it edits nothing. + unsupported, or unverifiable, with the failing ones quoted. The report is + delivered whole: a finding tucked behind an all-clear summary line is G-2's + fluent-but-wrong failure mode reappearing in review clothing, and the reviewer + reads the list, not the headline. Its prompt is adversarial (find unsupported + claims), not confirmatory (check the draft is fine), making it less likely to + share the drafter's failure mode (G-2). Its tool list is read-only; it edits + nothing. Model and effort. Profiles state capability requirements, never model names: models change faster than this spec. Drafting warrants the most capable model @@ -707,12 +720,19 @@ chosen, deliberately, per the caveat in section 10. agent's branch naming; whether review-comment revisions require write access (section 12); whether Actions runs on agent PRs wait for approval; how a verifier run is invoked against an existing PR (section 12); whether - delegation reliably opens a draft pull request (section 11); and whether an - agent profile can pin a model and effort level, or model choice rides entirely - on the per-task picker (section 14). + delegation reliably opens a draft pull request (section 11); whether assigning + an issue to Copilot offers the choice of agent profile, or selection needs the + Agents panel (section 12); whether a full phase A draft fits the session cap + or the drafting needs decomposing (section 11); whether an agent profile can + pin a model and effort level, or model choice rides entirely on the per-task + picker (section 14); and whether the cloud-agent configuration read endpoint, + in public preview as of this writing, returns the fields the snapshot needs + (section 13). - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase - one, and it must preserve HC-1. + one. It would be human-run with its own credential and the project's explicit + opt-in, entirely outside the agent's trust boundary; HC-1 continues to bind + the agent unchanged. - Intake relationship. How the request template fits with the existing CNCF service desk and assistance-program intake (section 7), without a competing front door. Related: the contribute.cncf.io site needs a page for projects on From 9a5e7ac28bdff34baad90a3c4cdf94250b9e48ea Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 15:37:57 -0700 Subject: [PATCH 23/47] Drop the approval claim from the disclosure's final form Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 2eb178b..2b1ea06 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -618,9 +618,10 @@ Its fields, each a labeled bullet: - Disclosure. A fixed sentence with two forms matching the deliverable's state (HC-6). The draft form: drafted by an AI agent, human review pending. The - final form: drafted by an AI agent and reviewed, verified, and approved by - humans. A draft asserting the final form would be claiming a review that has - not happened yet. + final form: drafted by an AI agent, reviewed and verified by humans. Each form + asserts only what has already happened at the moment it may appear; approval + is not claimed in the sentence because it is evidenced by the PR record + itself, which no body text can substitute for. - Drafter. The agent profile that produced the draft, by name and ref (section 14). - Verifier. The verifier profile, by name and ref, and a link to its report on @@ -642,7 +643,8 @@ link is added once the verifier's pass runs, and the record starts as a marked placeholder that only the reviewer fills; a drafter asserting its own work verified would be a fabrication about fabrication-checking. The disclosure's flip from draft form to final form belongs to review for the same reason: it -happens when the PR is marked ready. +happens when the PR is marked ready, at which point the review and verification +it asserts have both occurred. The provenance check (section 13) enforces the block in CI: a deliverable PR fails if the block is missing, a field is absent or malformed, the methodology From 9ef5603be68770e3e12f2408d9ccb47d505b052d Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 15:42:53 -0700 Subject: [PATCH 24/47] Narrow HC-4 to agent content; exempt gate-fired bookkeeping Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 2b1ea06..9ad9040 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -137,8 +137,11 @@ the labor. AI drafts, humans decide, all in the open. a technical writer still starts it (section 5). - HC-3: Deliverables are files in cncf/techdocs. Including the issue backlog (one file per proposed issue). Nothing is filed to an external repo. -- HC-4: No unattended autonomy. Any change to repository state lands through a - human-reviewed PR. The agent never self-merges. +- HC-4: No unattended autonomy. Content the agent produces lands only through a + human-reviewed PR, and the agent never self-merges. Deterministic lifecycle + bookkeeping fired by a human decision, such as the merge-triggered workflow + that opens the next phase's tracking issue (section 12), is not autonomy: no + model runs, and the human act that fires it is an approval already given. - HC-5: Provenance required. Quantitative claims (counts, link status, inventories) must be generated by a committed, re-runnable step and cite their source; no unverifiable figures in a deliverable. From bb51e34b514699ba25de41d304e39a5c237cffa5 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 15:50:30 -0700 Subject: [PATCH 25/47] Pin data sources: repo SHAs and site retrieval dates Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 9ad9040..44d304c 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -635,7 +635,12 @@ Its fields, each a labeled bullet: deliverable: one assessment is measured against one methodology. - Data. The data-collection commands that were run and the committed paths of their outputs, so every quantitative claim traces to a reproducible step - (HC-5). + (HC-5), plus the commit SHA of each assessed repository at collection time and + the retrieval date for each live site. The pins record which state of the + sources each claim describes: the reviewer verifies findings against what the + drafter actually saw (section 10), and when the sources move on, drift reads + as drift rather than error. They promise no re-run of the assessment itself, + which is judgment: two honest runs converge without matching. - Verification record. Which rating-bearing findings the reviewer verified against source, satisfying the section 10 floor (HC-7). From 1a237cce567a369caa33651a0db4cac3f465c5d0 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 15:56:35 -0700 Subject: [PATCH 26/47] Cite the config endpoint; close two deferral loops Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 44d304c..826afe2 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -515,12 +515,13 @@ points at the methodology, it never restates it (P-2). repository's existing label set. - Agent-configuration snapshot: `scripts/assessment/`. The agent's effective configuration (MCP servers, firewall state, and custom allowlist) is readable - from a documented endpoint, so a deterministic script captures it alongside - each assessment's data outputs (HC-5): what the agent could reach when a draft - was produced is committed evidence. The organization audit log remains the - authoritative history of who changed a setting and when. The endpoint does not - expose organization-level allowlist entries; the administrator/platform owner - records those by hand when they exist. + from a [documented endpoint][cloud-agent-config-api], so a deterministic + script captures it alongside each assessment's data outputs (HC-5): what the + agent could reach when a draft was produced is committed evidence. The + organization audit log remains the authoritative history of who changed a + setting and when. The endpoint does not expose organization-level allowlist + entries; the administrator/platform owner records those by hand when they + exist. - Deliverables: `analyses///`. The existing convention: `analysis.md`, `implementation.md`, and the backlog. - Backlog files: `analyses///issues/`. One file per proposed @@ -737,7 +738,7 @@ chosen, deliberately, per the caveat in section 10. pin a model and effort level, or model choice rides entirely on the per-task picker (section 14); and whether the cloud-agent configuration read endpoint, in public preview as of this writing, returns the fields the snapshot needs - (section 13). + and what credential the snapshot script must hold to call it (section 13). - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase one. It would be human-run with its own credential and the project's explicit @@ -753,7 +754,9 @@ chosen, deliberately, per the caveat in section 10. - Assessment-quality rubric. Define the meta-rubric that scores an assessment's quality, distinct from `criteria.md` (which scores a project's docs), and validate it by scoring the Flatcar, Knative, and Helm baselines to set a - reference band (sections 2, 10). + reference band (sections 2, 10). This work also sets any post-pilot sampling + rule for verification, replacing the pilot's check-every-finding floor + (section 10). - Small-team staffing. Sustaining the approver separation (a phase's approver must be neither the drafter nor a reviewer of that phase; sections 4, 10) when the same few writers wear multiple hats. @@ -778,3 +781,5 @@ chosen, deliberately, per the caveat in section 10. https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-cloud-agent/customize-the-agent-environment [cloud-agent-access]: https://docs.github.com/en/copilot/concepts/agents/cloud-agent/access-management +[cloud-agent-config-api]: + https://docs.github.com/en/rest/copilot/copilot-cloud-agent-management From ff954ecc0bccc426038dc1befddfbf87f35aab66 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 16:03:58 -0700 Subject: [PATCH 27/47] Scope HC-4 to repository content Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 826afe2..46f5094 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -137,11 +137,12 @@ the labor. AI drafts, humans decide, all in the open. a technical writer still starts it (section 5). - HC-3: Deliverables are files in cncf/techdocs. Including the issue backlog (one file per proposed issue). Nothing is filed to an external repo. -- HC-4: No unattended autonomy. Content the agent produces lands only through a - human-reviewed PR, and the agent never self-merges. Deterministic lifecycle - bookkeeping fired by a human decision, such as the merge-triggered workflow - that opens the next phase's tracking issue (section 12), is not autonomy: no - model runs, and the human act that fires it is an approval already given. +- HC-4: No unattended autonomy. Repository content the agent produces lands only + through a human-reviewed PR, and the agent never self-merges. Deterministic + lifecycle bookkeeping fired by a human decision, such as the merge-triggered + workflow that opens the next phase's tracking issue (section 12), is not + autonomy: no model runs, and the human act that fires it is an approval + already given. - HC-5: Provenance required. Quantitative claims (counts, link status, inventories) must be generated by a committed, re-runnable step and cite their source; no unverifiable figures in a deliverable. From 359de3f3e25382f064292e2f1e1d4ceeebb5a8a6 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 16:15:37 -0700 Subject: [PATCH 28/47] Profiles can pin a model; ours choose not to Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 27 ++++++++++++------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 46f5094..4827065 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -598,11 +598,15 @@ models change faster than this spec. Drafting warrants the most capable model and effort available to the repository; verification does not need the drafter's configuration, and running the verifier on a different model is preferable where the platform allows it, since two models are less likely to share one blind -spot. The platform exposes model choice as a per-task picker at delegation time, -so the writer applies this policy when delegating; whether a profile can pin its -own model is a build-time check (section 17). Which models are available to -cncf/techdocs is organization policy, handled with the CNCF GitHub organization -administrators by the administrator/platform owner (section 4). +spot. The platform offers model choice in two documented places: a `model` +property in the profile's front matter, which inherits the default when unset, +and a per-task picker at delegation time [custom-agents-config]. Our profiles +leave `model` unset by policy: a pinned name turns model churn into profile +churn, and the writer applies this policy through the picker when delegating. +Whether effort level can be pinned anywhere is a build-time check (section 17). +Which models the picker offers to cncf/techdocs is organization policy, handled +with the CNCF GitHub organization administrators by the administrator/platform +owner (section 4). ## 15. The provenance block @@ -735,11 +739,12 @@ chosen, deliberately, per the caveat in section 10. delegation reliably opens a draft pull request (section 11); whether assigning an issue to Copilot offers the choice of agent profile, or selection needs the Agents panel (section 12); whether a full phase A draft fits the session cap - or the drafting needs decomposing (section 11); whether an agent profile can - pin a model and effort level, or model choice rides entirely on the per-task - picker (section 14); and whether the cloud-agent configuration read endpoint, - in public preview as of this writing, returns the fields the snapshot needs - and what credential the snapshot script must hold to call it (section 13). + or the drafting needs decomposing (section 11); whether effort level can be + pinned in a profile or anywhere else, the one model-choice question the + documentation leaves open (section 14); and whether the cloud-agent + configuration read endpoint, in public preview as of this writing, returns the + fields the snapshot needs and what credential the snapshot script must hold to + call it (section 13). - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase one. It would be human-run with its own credential and the project's explicit @@ -784,3 +789,5 @@ chosen, deliberately, per the caveat in section 10. https://docs.github.com/en/copilot/concepts/agents/cloud-agent/access-management [cloud-agent-config-api]: https://docs.github.com/en/rest/copilot/copilot-cloud-agent-management +[custom-agents-config]: + https://docs.github.com/en/copilot/reference/custom-agents-configuration From 0e691c2337f229afe9052f6c77ffe64bc78bf4ec Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 16:20:33 -0700 Subject: [PATCH 29/47] Pin the methodology by commit SHA, not by ref Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 27 ++++++++++--------- 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 4827065..56b8905 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -553,7 +553,7 @@ Rules common to all profiles, in tension-order with Part I: - Point, do not restate. A profile carries the task shape, output paths, and discipline; the criteria, process, and templates are read from the methodology - corpus at the pinned ref (P-2). No rubric content, criterion text, or template + corpus at the pinned SHA (P-2). No rubric content, criterion text, or template structure is copied into a prompt. - Data, not instructions. Content from the assessed project (repos, sites, issues) is evidence to analyze, never instructions to follow (section 8). @@ -568,7 +568,7 @@ Rules common to all profiles, in tension-order with Part I: The three drafting profiles: - Assessment drafter (Phase A). Input: the intake issue, the methodology at the - pinned ref, and the data-collection outputs for the project. Output: a draft + pinned SHA, and the data-collection outputs for the project. Output: a draft `analysis.md` in the project's deliverables directory, rating each criterion with cited evidence. - Plan drafter (Phase B). Input: the merged `analysis.md` plus the intake and @@ -635,10 +635,12 @@ Its fields, each a labeled bullet: 14). - Verifier. The verifier profile, by name and ref, and a link to its report on the PR (section 14). -- Methodology. The ref of the methodology corpus the draft was produced against - (P-2), so the deliverable pins the criteria and templates it was measured by. - The ref is set at phase A and carried unchanged by every later phase's - deliverable: one assessment is measured against one methodology. +- Methodology. The commit SHA of the methodology corpus the draft was produced + against (P-2), so the deliverable pins the criteria and templates it was + measured by. A resolved SHA, not a branch or tag name: a name can move after + the assessment and take the pin's meaning with it. The SHA is set at phase A + and carried unchanged by every later phase's deliverable: one assessment is + measured against one methodology. - Data. The data-collection commands that were run and the committed paths of their outputs, so every quantitative claim traces to a reproducible step (HC-5), plus the commit SHA of each assessed repository at collection time and @@ -662,12 +664,13 @@ it asserts have both occurred. The provenance check (section 13) enforces the block in CI: a deliverable PR fails if the block is missing, a field is absent or malformed, the methodology -ref does not resolve or differs from the pin set at phase A, or a listed data -path is not in the tree. While the PR is a draft, the verification record may be -a placeholder and the disclosure carries its draft form; once it is marked ready -for review, the check runs strict: an unfilled record or a draft-form disclosure -fails. Judgment stays human: CI proves the block is present and well-formed, the -approver confirms the verification behind it was real (section 10). +pin is not a resolved commit SHA, does not resolve, or differs from the SHA set +at phase A, or a listed data path is not in the tree. While the PR is a draft, +the verification record may be a placeholder and the disclosure carries its +draft form; once it is marked ready for review, the check runs strict: an +unfilled record or a draft-form disclosure fails. Judgment stays human: CI +proves the block is present and well-formed, the approver confirms the +verification behind it was real (section 10). The block is defined by this spec and layered above the template body, so the methodology corpus and its templates are not modified (P-2, NG-4). From ffb31665ae7bf6ac10ca322d1b6b5b48a8155519 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 16:26:59 -0700 Subject: [PATCH 30/47] Run data collection in the session's setup steps Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 35 ++++++++++++------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 56b8905..684d6cd 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -212,7 +212,10 @@ Each phase runs the same six steps: when phase B is skipped; section 6). 2. Accept. A technical writer triages and explicitly accepts the request, for every phase, not just A; eligibility alone does not start work (P-1). -3. Draft. The drafter produces the deliverable as a draft PR. +3. Draft. The drafter produces the deliverable as a draft PR. When the phase's + drafter takes data-collection outputs as input (section 14), delegation + produces them first: they are generated deterministically at session start, + before any model runs, and committed with the draft as its evidence (HC-5). 4. Review. The verifier's fact-check pass runs first. The reviewer then refines the draft in conversation with the drafter, verifies findings against source (HC-7), and marks it ready. @@ -420,8 +423,9 @@ infrastructure we build and maintain ourselves. - Execution environment. The agent runs in an ephemeral [GitHub Actions-based environment][cloud-agent-env] with a hard session cap (currently 59 minutes). That cap shapes the design: deterministic data collection (section 13) runs as - committed scripts rather than inside drafting sessions, so the agent spends - its session on judgment, not inventory. The environment is prepared by + committed scripts in the session's setup steps, before the model starts, so + the agent spends its session on judgment, not inventory. The environment, + those steps included, is prepared by `.github/workflows/copilot-setup-steps.yml`. - Delegation is permission-gated. Only users with [write access to the repository][cloud-agent-access] can delegate work to the agent, which lets the @@ -510,7 +514,10 @@ points at the methodology, it never restates it (P-2). `scripts/assessment/`. CI that fails a deliverable PR whose provenance block is missing or malformed (section 15). - Data collection: `scripts/assessment/`. The deterministic inventory scripts - behind HC-5 (section 15). + behind HC-5 (section 15). The drafting session's setup steps (section 11) run + them and leave the outputs in the workspace; the drafter commits them + unmodified with the draft. Unmodified is checkable, not assumed: re-running + the committed command reproduces the committed outputs or exposes the edit. - Labels: `.github/settings.yml`. An `assessment` label, per-phase labels, and the phase B skip marker (section 12), managed declaratively alongside the repository's existing label set. @@ -568,7 +575,8 @@ Rules common to all profiles, in tension-order with Part I: The three drafting profiles: - Assessment drafter (Phase A). Input: the intake issue, the methodology at the - pinned SHA, and the data-collection outputs for the project. Output: a draft + pinned SHA, and the data-collection outputs for the project. Output: the + collection outputs committed unmodified (section 13), then a draft `analysis.md` in the project's deliverables directory, rating each criterion with cited evidence. - Plan drafter (Phase B). Input: the merged `analysis.md` plus the intake and @@ -710,7 +718,8 @@ The build steps, in dependency order, each sized to one issue: agent session's logs show the prepared environment. 5. Data-collection scripts (`scripts/assessment/`). Done when running the committed command against a sample project twice yields the same committed - outputs (HC-5). + outputs (HC-5), and a delegated session's draft PR contains the outputs its + setup steps produced. 6. Provenance check (workflow plus `scripts/assessment/`). Done when a fixture PR with a malformed or unfilled block fails, a well-formed one passes, and the draft-versus-ready leniency behaves as section 15 specifies. @@ -741,13 +750,13 @@ chosen, deliberately, per the caveat in section 10. verifier run is invoked against an existing PR (section 12); whether delegation reliably opens a draft pull request (section 11); whether assigning an issue to Copilot offers the choice of agent profile, or selection needs the - Agents panel (section 12); whether a full phase A draft fits the session cap - or the drafting needs decomposing (section 11); whether effort level can be - pinned in a profile or anywhere else, the one model-choice question the - documentation leaves open (section 14); and whether the cloud-agent - configuration read endpoint, in public preview as of this writing, returns the - fields the snapshot needs and what credential the snapshot script must hold to - call it (section 13). + Agents panel (section 12); whether a full phase A draft, with its setup-step + collection sharing the envelope, fits the session cap or the drafting needs + decomposing (section 11); whether effort level can be pinned in a profile or + anywhere else, the one model-choice question the documentation leaves open + (section 14); and whether the cloud-agent configuration read endpoint, in + public preview as of this writing, returns the fields the snapshot needs and + what credential the snapshot script must hold to call it (section 13). - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase one. It would be human-run with its own credential and the project's explicit From 2f6ab6fe25e926d85ab899d3caea4ba0085aa50a Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 16:31:28 -0700 Subject: [PATCH 31/47] Disclose to the project before drafting, whoever filed Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 684d6cd..005fee3 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -275,6 +275,10 @@ sections, plus maturity and contacts). Filing a request does not start an assessment: it gathers context, and a technical writer triages and explicitly accepts it to begin Phase A (NG-5, P-1). +A request can also be filed by the assessing team rather than the project, and +HC-6's up-front promise must not depend on who filed: phase A is not delegated +until the project contacts named in the request have been notified on the intake +issue. Open question: how this relates to the existing CNCF service desk and TechDocs assistance-program intake, so we don't create a competing front door. @@ -453,8 +457,11 @@ always readable from its issues and pull requests. eligibility is not work. - Accept (step 2): assignment. A writer accepts by assigning the issue to Copilot, adding themselves as an assignee to record who accepted; they act as - the phase's reviewer. Because only users with write access can delegate to the - agent (section 11), P-1 is enforced by repository permissions, not convention. + the phase's reviewer. For phase A, accepting includes mentioning the named + project contacts on the intake issue, so the AI disclosure reaches the project + before drafting starts, whoever filed the request (HC-6, section 7). Because + only users with write access can delegate to the agent (section 11), P-1 is + enforced by repository permissions, not convention. - Draft (step 3): a draft pull request. The agent works on its own branch in cncf/techdocs and opens a draft PR linked to the tracking issue, carrying the provenance block (section 15). From a0e14dd98310202fa3f5e9bf4482640da275e60e Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 16:34:12 -0700 Subject: [PATCH 32/47] Extend the verification floor to unrated phases Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 28 +++++++++++-------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 005fee3..77d71f0 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -339,11 +339,13 @@ The system is acceptable when, on a pilot assessment: - Quality. The deliverable is scored against an assessment-quality rubric (its definition is an open item; section 17). Verification is not a token sample: - for the pilot, the reviewer checks every rating-bearing finding, biased first - toward the highest-risk claims, and records in the deliverable which findings - were verified (HC-7); any cheaper sampling rule is future work, set with the - rubric (section 17). Final sign-off is given by the approver (section 4), who - was neither the drafter nor a reviewer of that phase, to avoid signing off on + for the pilot, the reviewer checks every rating-bearing finding in the + analysis and, in the phases that carry no ratings, every claim about the + project's current state that a recommendation stands on, biased first toward + the highest-risk claims, and records in the deliverable which findings were + verified (HC-7); any cheaper sampling rule is future work, set with the rubric + (section 17). Final sign-off is given by the approver (section 4), who was + neither the drafter nor a reviewer of that phase, to avoid signing off on one's own work. The bar is parity with the human baselines (Flatcar, Knative, Helm). - Safety. Zero writes outside cncf/techdocs, audited from the platform's @@ -660,15 +662,17 @@ Its fields, each a labeled bullet: their outputs, so every quantitative claim traces to a reproducible step (HC-5), plus the commit SHA of each assessed repository at collection time and the retrieval date for each live site. The pins record which state of the - sources each claim describes: the reviewer verifies findings against what the - drafter actually saw (section 10), and when the sources move on, drift reads - as drift rather than error. They promise no re-run of the assessment itself, - which is judgment: two honest runs converge without matching. -- Verification record. Which rating-bearing findings the reviewer verified - against source, satisfying the section 10 floor (HC-7). + sources each claim describes: a pinned repository lets the reviewer check out + exactly what the drafter saw (section 10), a dated site bounds when the claim + held, and when the sources move on, drift reads as drift rather than error. + They promise no re-run of the assessment itself, which is judgment: two honest + runs converge without matching. +- Verification record. Which findings the reviewer verified against source, + satisfying the section 10 floor (HC-7): the rating-bearing findings in phase + A, the claims grounding the recommendations in phases B and C. The fields have owners. The drafter emits the block with everything it can know: -disclosure, its own profile, the methodology ref, and the data paths. The +disclosure, its own profile, the methodology SHA, and the data paths. The verifier field and the verification record belong to the review step: the report link is added once the verifier's pass runs, and the record starts as a marked placeholder that only the reviewer fills; a drafter asserting its own work From 60d7307583d2f8ef9b651001562ac0b0a422c472 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 16:40:29 -0700 Subject: [PATCH 33/47] Let the methodology, not this spec, set the backlog layout Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 77d71f0..e9331c0 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -135,8 +135,9 @@ the labor. AI drafts, humans decide, all in the open. required, and a project's disagreement is recorded in the deliverable rather than allowed to block or soften it. Merging phase N makes phase N+1 eligible; a technical writer still starts it (section 5). -- HC-3: Deliverables are files in cncf/techdocs. Including the issue backlog - (one file per proposed issue). Nothing is filed to an external repo. +- HC-3: Deliverables are files in cncf/techdocs. Including the issue backlog, in + whatever layout the methodology prescribes (section 6 proposes one file per + issue). Nothing is filed to an external repo. - HC-4: No unattended autonomy. Repository content the agent produces lands only through a human-reviewed PR, and the agent never self-merges. Deterministic lifecycle bookkeeping fired by a human decision, such as the merge-triggered @@ -253,7 +254,10 @@ end up slower than doing it by hand. script (NG-2, section 17), which can batch-create issues with `gh` instead of someone copy-pasting from a large combined file; it also keeps any single file readable. This is raised as a proposal for discussion; the methodology docs - are not changed by this PR. + are not changed by this PR. The order matters (P-2): the agent follows the + methodology as it stands, so until the change is accepted there, the backlog + drafter produces the single combined file, and the split layout waits for the + methodology, never the other way around. Every deliverable carries a header noting it was AI-drafted and human-reviewed (HC-6). @@ -542,7 +546,9 @@ points at the methodology, it never restates it (P-2). - Deliverables: `analyses///`. The existing convention: `analysis.md`, `implementation.md`, and the backlog. - Backlog files: `analyses///issues/`. One file per proposed - issue plus an index, if the section 6 proposal is accepted. + issue plus an index, if the section 6 proposal is accepted into the + methodology first; until then, the combined file the methodology prescribes + (P-2). Notes: From 1bfdcfd4d4f4e055ce0dffd157970579c93870dd Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 16:55:50 -0700 Subject: [PATCH 34/47] Bind P-2's reproducibility pin to a commit SHA Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index e9331c0..f659582 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -101,9 +101,9 @@ the labor. AI drafts, humans decide, all in the open. behavior, and no parallel copy can drift. This addresses the criteria-duplication concern raised in PR #357 review. - Reproducibility pin. For a given assessment run, the agent binds to a - specific ref (commit or tag) of the methodology docs so the run is - reproducible (P-4). "Auto-updates" means each new assessment picks up the - latest canonical docs, not that an in-flight assessment shifts under review. + specific commit SHA of the methodology docs so the run is reproducible + (P-4). "Auto-updates" means each new assessment picks up the latest + canonical docs, not that an in-flight assessment shifts under review. - Consumability is a contract. Because the docs are the interface, an operational layer may add how-to-run guidance (output format, file naming, orchestration) but must reference the canonical content, never restate it. From c14e8788291b71d172afbcd2a94a361208fbb9d6 Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 16:57:37 -0700 Subject: [PATCH 35/47] Profiles run only when a writer selects them Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index f659582..6d0e37d 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -586,6 +586,11 @@ Rules common to all profiles, in tension-order with Part I: - Provenance. Every draft opens with the provenance block (section 15). - Least tools. A profile's tool list is the minimum its phase needs; none may include a write-capable MCP tool (section 11). +- Explicit selection only. By default the platform may auto-select a custom + agent from task context [custom-agents-config], which would let an unrelated + delegation in cncf/techdocs pull in an assessment profile. Every profile sets + `disable-model-invocation`, so a profile runs only when the writer selects it + (P-1, section 12). The three drafting profiles: From 2d7b7797f6a36fcbf7dee7da2a0817f63d822e4b Mon Sep 17 00:00:00 2001 From: Nate W Date: Tue, 11 Aug 2026 17:03:42 -0700 Subject: [PATCH 36/47] Name the setup-targeting unknown; align backlog wording Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 6d0e37d..2a16dd3 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -605,8 +605,8 @@ The three drafting profiles: silently promoted into a new finding. - Backlog drafter (Phase C). Input: the merged `implementation.md`, or the merged `analysis.md` where phase B was skipped (section 6). Output: the issue - backlog in the layout section 6 settles on, each item scoped to the - methodology's time bound (section 6), with effort estimates marked as + backlog in the layout the methodology prescribes (section 6), each item scoped + to the methodology's time bound (section 6), with effort estimates marked as first-pass for reviewer sanity-checking. The verification profile: @@ -772,7 +772,9 @@ chosen, deliberately, per the caveat in section 10. verifier run is invoked against an existing PR (section 12); whether delegation reliably opens a draft pull request (section 11); whether assigning an issue to Copilot offers the choice of agent profile, or selection needs the - Agents panel (section 12); whether a full phase A draft, with its setup-step + Agents panel (section 12); how the assessment's target project reaches the + setup steps that run collection, or whether collection needs a different + trigger (section 13); whether a full phase A draft, with its setup-step collection sharing the envelope, fits the session cap or the drafting needs decomposing (section 11); whether effort level can be pinned in a profile or anywhere else, the one model-choice question the documentation leaves open From 831ed272b2710349e0542b5614f435be130ac97c Mon Sep 17 00:00:00 2001 From: Nate W Date: Wed, 12 Aug 2026 13:06:32 -0700 Subject: [PATCH 37/47] spec: bind lifecycle to slash commands, labels, and an umbrella intake issue Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 229 ++++++++++++------ 1 file changed, 158 insertions(+), 71 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 2a16dd3..c04a4ec 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -217,9 +217,10 @@ Each phase runs the same six steps: drafter takes data-collection outputs as input (section 14), delegation produces them first: they are generated deterministically at session start, before any model runs, and committed with the draft as its evidence (HC-5). -4. Review. The verifier's fact-check pass runs first. The reviewer then refines - the draft in conversation with the drafter, verifies findings against source - (HC-7), and marks it ready. +4. Review. The verifier's fact-check pass runs first; the reviewer may skip it + deliberately, recording why (section 12). The reviewer then refines the draft + in conversation with the drafter, verifies findings against source (HC-7), + and marks it ready. 5. Stakeholder review. Stakeholders confirm factual accuracy (required to advance) and record any disagreement with the conclusions (not required; HC-2). @@ -453,58 +454,113 @@ infrastructure we build and maintain ourselves. ## 12. Lifecycle bindings -Each lifecycle step (section 5) binds to a native GitHub primitive, so the -workflow needs no orchestration layer of its own. An assessment's state is -always readable from its issues and pull requests. +Each lifecycle step (section 5) binds to a native GitHub primitive, and the +workflow keeps no state of its own: an assessment's state is always readable +from its issues and pull requests. + +The human acts that move a phase are issued as slash commands (`/accept`, +`/decline`, `/verify`, `/ready`, `/confirm`, `/discard`, `/skip-implementation`) +written as ordinary comments, following [Kubernetes issue +triage][k8s-issue-triage] practice and the comment-command automation already +running in [cncf/mentoring][mentoring-commands]. An `issue_comment` workflow +parses the command, checks the commenter against the role bindings below, and +performs the corresponding native operations, so a command grants no power the +commenter does not already hold. Commands are deterministic bookkeeping fired by +a recorded human decision (HC-4); the one that starts a model run, `/verify`, is +an explicit delegation by the phase's reviewer (P-1). Labels applied by the +commands and workflows record where each assessment stands: an issue search +filtered by phase label is the portfolio view, with no dashboard to build. - Request (step 1): an issue. Phase A's request is the intake issue (section 7); phase B's and C's tracking issues are opened automatically when the previous - phase merges (see Merge below). An issue may sit unassigned indefinitely; - eligibility is not work. -- Accept (step 2): assignment. A writer accepts by assigning the issue to - Copilot, adding themselves as an assignee to record who accepted; they act as - the phase's reviewer. For phase A, accepting includes mentioning the named - project contacts on the intake issue, so the AI disclosure reaches the project - before drafting starts, whoever filed the request (HC-6, section 7). Because - only users with write access can delegate to the agent (section 11), P-1 is - enforced by repository permissions, not convention. + phase merges (see Merge below). A request issue opens labeled `needs-triage`, + applied by the intake form or by the phase-advance workflow. An issue may sit + unassigned indefinitely; eligibility is not work. The intake issue doubles as + the assessment's umbrella issue, open end to end: every phase's tracking issue + and pull request links back to it, so cross-references accumulate on its + timeline; its labels carry the current phase; and the workflows post each + transition to it (see Merge below). Where an assessment stands, and what has + to happen next, is readable from that one issue. (The backlog's umbrella, + section 6, is the same pattern applied to the resulting doc work; a different + artifact.) +- Accept (step 2): the `/accept` command. A writer accepts by commenting + `/accept` on the request issue: the workflow assigns the issue to them, + recording who accepted and who acts as the phase's reviewer, and swaps + `needs-triage` for `triage/accepted`. For phase A, the same command posts the + mention of the named project contacts on the intake issue, so the AI + disclosure reaches the project before drafting starts, whoever filed the + request, and the step cannot be forgotten (HC-6, section 7). The writer then + delegates to the agent directly, selecting the phase's drafting profile + (section 14); the command does not start the agent. Because only users with + write access can delegate (section 11), P-1 is enforced by repository + permissions, not convention, and the workflow refuses `/accept` from a + commenter without write access. For phases B and C, whose tracking issues are + separate from the intake, the workflow also notes the acceptance on the intake + issue: an unaccepted eligible phase is the stall mode, so the umbrella shows + when work actually started and by whom. The triage verdict has a counterpart: + a writer declines an unaccepted request with `/decline `, under the + same authorization as `/accept`; the workflow labels it `triage/declined`, + posts the reason, and closes the issue as not planned. Declining the intake + means the assessment never starts. Declining a phase B or C tracking issue + ends the assessment there, noted on the intake issue, which the workflow + closes, so an open intake still means work in flight. A project's request to + stop is honored the same way: the writer records a decline rather than leaving + the request to sit unanswered. - Draft (step 3): a draft pull request. The agent works on its own branch in cncf/techdocs and opens a draft PR linked to the tracking issue, carrying the provenance block (section 15). - Review (step 4): PR review comments. The reviewer triggers the verifier's pass - on the draft PR; its report lands as a PR comment, part of the same public - record. The reviewer then refines the draft in conversation, handing revisions - back by mentioning `@copilot` in review comments, and performs the - verification required by section 10. Whether mention-triggered revisions - require write access, and how a verifier run is invoked against an existing - PR, are build-time checks (section 17). -- Stakeholder review (step 5): mention, not access. The reviewer marks the PR - ready for review and mentions the stakeholders named in the intake. The - draft-to-ready flip carries the real-world meaning: draft while the assessing - team is still working the deliverable over, ready when they would put it in - front of the requesting project. cncf/techdocs is public, so stakeholders can - review and comment without any special access; formal review requests are - limited to collaborators, which is why the binding is a mention plus a - factual-accuracy confirmation recorded on the PR, with any disagreement - recorded in the deliverable itself (HC-2). + by commenting `/verify` on the draft PR; its report lands as a PR comment, + part of the same public record, and the workflow applies the `verified` label + to the PR when it does. The label means the verification this PR needs is + satisfied, not that the agent ran: a reviewer who deliberately skips the pass + (section 5), judging the human check sufficient or the verifier unavailable, + applies the label by hand with the reason in a comment, and the timeline + records who and when. The label is the gate: `/ready` refuses without it, and + the strict provenance check fails a ready PR that lacks it (section 15). The + reviewer refines the draft in conversation, handing revisions back by + mentioning `@copilot` in review comments, and performs the verification + required by section 10. Whether mention-triggered revisions require write + access, and whether a comment-triggered workflow can start the verifier + against an existing PR, as `/verify` requires, are build-time checks (section + 17). +- Stakeholder review (step 5): mention, not access. The reviewer comments + `/ready`: the workflow marks the PR ready for review and mentions the + stakeholders named in the intake, one act instead of two. The draft-to-ready + flip carries the real-world meaning: draft while the assessing team is still + working the deliverable over, ready when they would put it in front of the + requesting project. cncf/techdocs is public, so stakeholders can review and + comment without any special access; formal review requests are limited to + collaborators, which is why the binding is a mention plus a factual-accuracy + confirmation, recorded by the stakeholder commenting `/confirm`, which the + workflow checks against the names in the intake, no repository access + required. Any disagreement with the conclusions is recorded in the deliverable + itself (HC-2). - Merge (step 6): merge plus one workflow. The approver (section 4) merges. A merge-triggered GitHub Actions workflow (standard `GITHUB_TOKEN` with - `issues: write`) then opens the next phase's tracking issue, linking the - intake and the merged deliverable, and assigns no one: eligible, not started - (HC-2, P-1). The phase B skip decision (section 6) is recorded as a label on - the phase A PR, which the workflow reads to open phase C's tracking issue - instead. Approver independence cannot be natively enforced by GitHub; it is - verifiable from the public PR record, and an advisory CI check that flags a + `issues: write`) then opens the next phase's tracking issue, labeled for its + phase and `needs-triage`, linking the intake and the merged deliverable, and + assigns no one: eligible, not started (HC-2, P-1). The phase B skip decision + (section 6) is recorded with `/skip-implementation` on the phase A PR, + applying the label the workflow reads to open phase C's tracking issue + instead. The same workflow keeps the umbrella current: it posts the transition + on the intake issue, what merged and what is now eligible with the command + that starts it, and swaps the intake's phase label. The final phase's merge + closes the intake issue: an open intake is an assessment in flight, a closed + one is done. Approver independence cannot be natively enforced by GitHub; it + is verifiable from the public PR record, and an advisory CI check that flags a violation is a build-plan candidate (section 16). -- Failure and abort leave a trail. Discarding a draft (section 5) is closing its - PR with the reason recorded in a comment; the tracking issue stays open for a - restart or a hand-written phase. An administrator/platform owner abort closes - the tracking issue, with the rationale recorded there. +- Failure and abort leave a trail. The reviewer discards a draft (section 5) + with `/discard `, which closes the PR with the reason recorded in the + command comment and notes the discard on the intake issue; the tracking issue + stays open for a restart or a hand-written phase. An administrator/platform + owner abort closes the tracking issue, with the rationale recorded there. - The timeline records elapsed time. Gate transitions (opened, accepted, draft PR, ready, approved, merged) are timestamped in issue and PR history, so how long a pilot took, and where the time went, can be read from the GitHub - timeline after the fact. This reports on G-1 without adding a log or an - acceptance gate. + timeline after the fact. The intake issue aggregates the phase boundaries, so + the reading starts from one place. This reports on G-1 without adding a log or + an acceptance gate. ## 13. Components and repository layout @@ -523,6 +579,12 @@ points at the methodology, it never restates it (P-2). agent's ephemeral environment (section 11). - Phase-advance workflow: `.github/workflows/assessment-phase.yml`. On merge, opens the next phase's tracking issue (section 12). +- Command workflow: `.github/workflows/assessment-commands.yml`, backed by a + parser in `scripts/assessment/`. The `issue_comment` glue behind the slash + commands (section 12): parse the comment, authorize the commenter against the + role bindings, act. The parser is a plain unit-tested function and the + workflow is thin glue over it, the pattern proven by the [cncf/mentoring + automation][mentoring-commands]. - Provenance check: a workflow in `.github/workflows/` backed by `scripts/assessment/`. CI that fails a deliverable PR whose provenance block is missing or malformed (section 15). @@ -531,9 +593,13 @@ points at the methodology, it never restates it (P-2). them and leave the outputs in the workspace; the drafter commits them unmodified with the draft. Unmodified is checkable, not assumed: re-running the committed command reproduces the committed outputs or exposes the edit. -- Labels: `.github/settings.yml`. An `assessment` label, per-phase labels, and - the phase B skip marker (section 12), managed declaratively alongside the - repository's existing label set. +- Labels: `.github/settings.yml`. The repository's existing `Docs analysis` + label marks assessment work, unchanged from the human-run analyses; AI + involvement is indicated by the provenance block and the disclosure, not a + label. Added for the workflow: per-phase labels, the + `needs-triage`/`triage/accepted`/`triage/declined` triage set, the `verified` + gate (section 12), and the phase B skip marker (section 12), managed + declaratively alongside the existing label set. - Agent-configuration snapshot: `scripts/assessment/`. The agent's effective configuration (MCP servers, firewall state, and custom allowlist) is readable from a [documented endpoint][cloud-agent-config-api], so a deterministic @@ -653,16 +719,22 @@ renderer, and the fixed labels make it machine-checkable. Its fields, each a labeled bullet: -- Disclosure. A fixed sentence with two forms matching the deliverable's state +- Disclosure. A fixed sentence with three forms matching the deliverable's state (HC-6). The draft form: drafted by an AI agent, human review pending. The final form: drafted by an AI agent, reviewed and verified by humans. Each form asserts only what has already happened at the moment it may appear; approval is not claimed in the sentence because it is evidenced by the PR record - itself, which no body text can substitute for. + itself, which no body text can substitute for. The third form covers a phase + the reviewer wrote by hand after discarding the draft (sections 5, 12): + written by humans, with `none` in the agent fields. The block rides every + deliverable either way, so the reader always learns how the document was + produced, including that no agent was involved. - Drafter. The agent profile that produced the draft, by name and ref (section 14). - Verifier. The verifier profile, by name and ref, and a link to its report on - the PR (section 14). + the PR (section 14); `none` when the reviewer deliberately skipped the pass, + with the reason recorded on the PR where the `verified` label was applied by + hand (section 12). - Methodology. The commit SHA of the methodology corpus the draft was produced against (P-2), so the deliverable pins the criteria and templates it was measured by. A resolved SHA, not a branch or tag name: a name can move after @@ -698,9 +770,13 @@ pin is not a resolved commit SHA, does not resolve, or differs from the SHA set at phase A, or a listed data path is not in the tree. While the PR is a draft, the verification record may be a placeholder and the disclosure carries its draft form; once it is marked ready for review, the check runs strict: an -unfilled record or a draft-form disclosure fails. Judgment stays human: CI -proves the block is present and well-formed, the approver confirms the -verification behind it was real (section 10). +unfilled record, a draft-form disclosure, or a missing `verified` label +(section 12) fails. A hand-written deliverable carries the human-written +disclosure form with `none` in the agent fields, and the check accepts that +shape: the provenance regime describes agent work, it does not block the failure +path's hand-written phase (section 12). Judgment stays human: CI proves the +block is present and well-formed, the approver confirms the verification behind +it was real (section 10). The block is defined by this spec and layered above the template body, so the methodology corpus and its templates are not modified (P-2, NG-4). @@ -751,13 +827,20 @@ The build steps, in dependency order, each sized to one issue: and the verifier's report correctly flags a planted unsupported claim. 8. Phase-advance workflow (`.github/workflows/assessment-phase.yml`). Done when a merged fixture PR opens the next phase's tracking issue, linked and - unassigned, on both the default route and the phase B skip route (section - 12). -9. Approver-independence check (advisory; a build-plan candidate from section - 12). Comments when a phase's approver also drafted or reviewed it; never - blocks. Done when it flags a staged violation, or explicitly deferred. - -The build is complete when steps 1 through 8 are merged, the section 17 + unassigned, on both the default route and the phase B skip route, posts the + transition on the intake issue and swaps its phase label, and the final + phase's merge closes the intake (section 12). +9. Command workflow (`.github/workflows/assessment-commands.yml` plus parser in + `scripts/assessment/`). Done when each command performs its section 12 + actions for an authorized commenter, refuses an unauthorized one with a + visible reply, `/ready` refuses without the `verified` label, a manually + applied label is honored, and the parser's unit tests cover recognition, + argument handling, and authorization decisions. +10. Approver-independence check (advisory; a build-plan candidate from section + 12). Comments when a phase's approver also drafted or reviewed it; never + blocks. Done when it flags a staged violation, or explicitly deferred. + +The build is complete when steps 1 through 9 are merged, the section 17 build-time checks have recorded answers, and one end-to-end walkthrough of the section 5 lifecycle on a fixture project has run clean. Then the pilot is chosen, deliberately, per the caveat in section 10. @@ -768,19 +851,20 @@ chosen, deliberately, per the caveat in section 10. documentation, to be answered by observation during the build (section 16) and recorded: the agent token's effective permission scopes (section 11); the agent's branch naming; whether review-comment revisions require write access - (section 12); whether Actions runs on agent PRs wait for approval; how a - verifier run is invoked against an existing PR (section 12); whether - delegation reliably opens a draft pull request (section 11); whether assigning - an issue to Copilot offers the choice of agent profile, or selection needs the - Agents panel (section 12); how the assessment's target project reaches the - setup steps that run collection, or whether collection needs a different - trigger (section 13); whether a full phase A draft, with its setup-step - collection sharing the envelope, fits the session cap or the drafting needs - decomposing (section 11); whether effort level can be pinned in a profile or - anywhere else, the one model-choice question the documentation leaves open - (section 14); and whether the cloud-agent configuration read endpoint, in - public preview as of this writing, returns the fields the snapshot needs and - what credential the snapshot script must hold to call it (section 13). + (section 12); whether Actions runs on agent PRs wait for approval; whether a + comment-triggered workflow can start the verifier against an existing PR, as + `/verify` requires (section 12); whether delegation reliably opens a draft + pull request (section 11); whether assigning an issue to Copilot offers the + choice of agent profile, or selection needs the Agents panel (section 12); how + the assessment's target project reaches the setup steps that run collection, + or whether collection needs a different trigger (section 13); whether a full + phase A draft, with its setup-step collection sharing the envelope, fits the + session cap or the drafting needs decomposing (section 11); whether effort + level can be pinned in a profile or anywhere else, the one model-choice + question the documentation leaves open (section 14); and whether the + cloud-agent configuration read endpoint, in public preview as of this writing, + returns the fields the snapshot needs and what credential the snapshot script + must hold to call it (section 13). - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase one. It would be human-run with its own credential and the project's explicit @@ -827,3 +911,6 @@ chosen, deliberately, per the caveat in section 10. https://docs.github.com/en/rest/copilot/copilot-cloud-agent-management [custom-agents-config]: https://docs.github.com/en/copilot/reference/custom-agents-configuration +[k8s-issue-triage]: https://www.kubernetes.dev/docs/guide/issue-triage/ +[mentoring-commands]: + https://github.com/cncf/mentoring/blob/main/.github/workflows/lfx-proposal-approvals.yml From 1e83675896b7050a799caea52d3f9a18309bac67 Mon Sep 17 00:00:00 2001 From: Nate W Date: Wed, 12 Aug 2026 13:50:58 -0700 Subject: [PATCH 38/47] spec: give /confirm teeth and close command-binding gaps from review Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 139 ++++++++++-------- 1 file changed, 78 insertions(+), 61 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index c04a4ec..55553a3 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -463,10 +463,13 @@ The human acts that move a phase are issued as slash commands (`/accept`, written as ordinary comments, following [Kubernetes issue triage][k8s-issue-triage] practice and the comment-command automation already running in [cncf/mentoring][mentoring-commands]. An `issue_comment` workflow -parses the command, checks the commenter against the role bindings below, and -performs the corresponding native operations, so a command grants no power the -commenter does not already hold. Commands are deterministic bookkeeping fired by -a recorded human decision (HC-4); the one that starts a model run, `/verify`, is +parses the command, authorizes the commenter against the role bindings below, +and performs the corresponding native operations: a command is honored only from +the role the lifecycle assigns that act, so the write-gated commands grant no +power the commenter does not already hold, and `/confirm` is authorized by the +stakeholder names in the intake rather than by repository access (see +Stakeholder review below). Commands are deterministic bookkeeping fired by a +recorded human decision (HC-4); the one that starts a model run, `/verify`, is an explicit delegation by the phase's reviewer (P-1). Labels applied by the commands and workflows record where each assessment stands: an issue search filtered by phase label is the portfolio view, with no dashboard to build. @@ -491,21 +494,22 @@ filtered by phase label is the portfolio view, with no dashboard to build. disclosure reaches the project before drafting starts, whoever filed the request, and the step cannot be forgotten (HC-6, section 7). The writer then delegates to the agent directly, selecting the phase's drafting profile - (section 14); the command does not start the agent. Because only users with - write access can delegate (section 11), P-1 is enforced by repository - permissions, not convention, and the workflow refuses `/accept` from a - commenter without write access. For phases B and C, whose tracking issues are - separate from the intake, the workflow also notes the acceptance on the intake - issue: an unaccepted eligible phase is the stall mode, so the umbrella shows - when work actually started and by whom. The triage verdict has a counterpart: - a writer declines an unaccepted request with `/decline `, under the - same authorization as `/accept`; the workflow labels it `triage/declined`, - posts the reason, and closes the issue as not planned. Declining the intake - means the assessment never starts. Declining a phase B or C tracking issue - ends the assessment there, noted on the intake issue, which the workflow - closes, so an open intake still means work in flight. A project's request to - stop is honored the same way: the writer records a decline rather than leaving - the request to sit unanswered. + (section 14); the command does not start the agent, because delegation carries + choices a comment cannot: the profile and the model picker (section 14). + Because only users with write access can delegate (section 11), P-1 is + enforced by repository permissions, not convention, and the workflow refuses + `/accept` from a commenter without write access. For phases B and C, whose + tracking issues are separate from the intake, the workflow also notes the + acceptance on the intake issue: an unaccepted eligible phase is the stall + mode, so the umbrella shows when work actually started and by whom. The triage + verdict has a counterpart: a writer declines an unaccepted request with + `/decline `, under the same authorization as `/accept`; the workflow + labels it `triage/declined`, posts the reason, and closes the issue as not + planned. Declining the intake means the assessment never starts. Declining a + phase B or C tracking issue ends the assessment there, noted on the intake + issue, which the workflow closes, so an open intake still means work in + flight. A project's request to stop is honored the same way: the writer + records a decline rather than leaving the request to sit unanswered. - Draft (step 3): a draft pull request. The agent works on its own branch in cncf/techdocs and opens a draft PR linked to the tracking issue, carrying the provenance block (section 15). @@ -534,27 +538,33 @@ filtered by phase label is the portfolio view, with no dashboard to build. collaborators, which is why the binding is a mention plus a factual-accuracy confirmation, recorded by the stakeholder commenting `/confirm`, which the workflow checks against the names in the intake, no repository access - required. Any disagreement with the conclusions is recorded in the deliverable - itself (HC-2). -- Merge (step 6): merge plus one workflow. The approver (section 4) merges. A - merge-triggered GitHub Actions workflow (standard `GITHUB_TOKEN` with - `issues: write`) then opens the next phase's tracking issue, labeled for its - phase and `needs-triage`, linking the intake and the merged deliverable, and - assigns no one: eligible, not started (HC-2, P-1). The phase B skip decision - (section 6) is recorded with `/skip-implementation` on the phase A PR, - applying the label the workflow reads to open phase C's tracking issue - instead. The same workflow keeps the umbrella current: it posts the transition - on the intake issue, what merged and what is now eligible with the command - that starts it, and swaps the intake's phase label. The final phase's merge - closes the intake issue: an open intake is an assessment in flight, a closed - one is done. Approver independence cannot be natively enforced by GitHub; it - is verifiable from the public PR record, and an advisory CI check that flags a - violation is a build-plan candidate (section 16). + required, and records by applying the `confirmed` label to the PR. Any + disagreement with the conclusions is recorded in the deliverable itself + (HC-2). +- Merge (step 6): merge plus one workflow. The approver (section 4) merges, + checking the `confirmed` label is present first: stakeholder confirmation is + required to advance (step 5, HC-2), and because a ready PR awaiting + stakeholders is a normal state rather than a failure, the check is the + approver's duty, not CI's. A merge-triggered GitHub Actions workflow (standard + `GITHUB_TOKEN` with `issues: write`) then opens the next phase's tracking + issue, labeled for its phase and `needs-triage`, linking the intake and the + merged deliverable, and assigns no one: eligible, not started (HC-2, P-1). The + reviewer records the phase B skip decision, the joint call with stakeholders + (section 6), with `/skip-implementation` on the phase A PR, applying the label + the workflow reads to open phase C's tracking issue instead. The same workflow + keeps the umbrella current: it posts the transition on the intake issue, what + merged and what is now eligible with the command that starts it, and swaps the + intake's phase label. The final phase's merge closes the intake issue: an open + intake is an assessment in flight, a closed one is done. Approver independence + cannot be natively enforced by GitHub; it is verifiable from the public PR + record, and an advisory CI check that flags a violation is a build-plan + candidate (section 16). - Failure and abort leave a trail. The reviewer discards a draft (section 5) with `/discard `, which closes the PR with the reason recorded in the command comment and notes the discard on the intake issue; the tracking issue stays open for a restart or a hand-written phase. An administrator/platform - owner abort closes the tracking issue, with the rationale recorded there. + owner abort closes the tracking issue and the intake with it, the rationale + recorded on both, so an open intake still means work in flight. - The timeline records elapsed time. Gate transitions (opened, accepted, draft PR, ready, approved, merged) are timestamped in issue and PR history, so how long a pilot took, and where the time went, can be read from the GitHub @@ -598,8 +608,8 @@ points at the methodology, it never restates it (P-2). involvement is indicated by the provenance block and the disclosure, not a label. Added for the workflow: per-phase labels, the `needs-triage`/`triage/accepted`/`triage/declined` triage set, the `verified` - gate (section 12), and the phase B skip marker (section 12), managed - declaratively alongside the existing label set. + and `confirmed` gates (section 12), and the phase B skip marker (section 12), + managed declaratively alongside the existing label set. - Agent-configuration snapshot: `scripts/assessment/`. The agent's effective configuration (MCP servers, firewall state, and custom allowlist) is readable from a [documented endpoint][cloud-agent-config-api], so a deterministic @@ -697,10 +707,12 @@ property in the profile's front matter, which inherits the default when unset, and a per-task picker at delegation time [custom-agents-config]. Our profiles leave `model` unset by policy: a pinned name turns model churn into profile churn, and the writer applies this policy through the picker when delegating. -Whether effort level can be pinned anywhere is a build-time check (section 17). -Which models the picker offers to cncf/techdocs is organization policy, handled -with the CNCF GitHub organization administrators by the administrator/platform -owner (section 4). +Whether `/verify`'s workflow invocation can carry the same model choice is a +build-time check (section 17); if it cannot, the different-model preference +survives only where the verifier is delegated by hand. Whether effort level can +be pinned anywhere is a build-time check (section 17). Which models the picker +offers to cncf/techdocs is organization policy, handled with the CNCF GitHub +organization administrators by the administrator/platform owner (section 4). ## 15. The provenance block @@ -773,10 +785,13 @@ draft form; once it is marked ready for review, the check runs strict: an unfilled record, a draft-form disclosure, or a missing `verified` label (section 12) fails. A hand-written deliverable carries the human-written disclosure form with `none` in the agent fields, and the check accepts that -shape: the provenance regime describes agent work, it does not block the failure -path's hand-written phase (section 12). Judgment stays human: CI proves the -block is present and well-formed, the approver confirms the verification behind -it was real (section 10). +shape in the agent fields only: the methodology pin, the data provenance, and +the verification record still apply and are checked, because the evidence rules +bind human work the same (HC-5, HC-7). The provenance regime describes agent +work; what it does not do is block the failure path's hand-written phase for +lacking an agent (section 12). Judgment stays human: CI proves the block is +present and well-formed, the approver confirms the verification behind it was +real (section 10). The block is defined by this spec and layered above the template body, so the methodology corpus and its templates are not modified (P-2, NG-4). @@ -833,9 +848,10 @@ The build steps, in dependency order, each sized to one issue: 9. Command workflow (`.github/workflows/assessment-commands.yml` plus parser in `scripts/assessment/`). Done when each command performs its section 12 actions for an authorized commenter, refuses an unauthorized one with a - visible reply, `/ready` refuses without the `verified` label, a manually - applied label is honored, and the parser's unit tests cover recognition, - argument handling, and authorization decisions. + visible reply, `/ready` refuses without the `verified` label, `/confirm` from + an intake-named stakeholder applies the `confirmed` label, a manually applied + label is honored, and the parser's unit tests cover recognition, argument + handling, and authorization decisions. 10. Approver-independence check (advisory; a build-plan candidate from section 12). Comments when a phase's approver also drafted or reviewed it; never blocks. Done when it flags a staged violation, or explicitly deferred. @@ -853,18 +869,19 @@ chosen, deliberately, per the caveat in section 10. agent's branch naming; whether review-comment revisions require write access (section 12); whether Actions runs on agent PRs wait for approval; whether a comment-triggered workflow can start the verifier against an existing PR, as - `/verify` requires (section 12); whether delegation reliably opens a draft - pull request (section 11); whether assigning an issue to Copilot offers the - choice of agent profile, or selection needs the Agents panel (section 12); how - the assessment's target project reaches the setup steps that run collection, - or whether collection needs a different trigger (section 13); whether a full - phase A draft, with its setup-step collection sharing the envelope, fits the - session cap or the drafting needs decomposing (section 11); whether effort - level can be pinned in a profile or anywhere else, the one model-choice - question the documentation leaves open (section 14); and whether the - cloud-agent configuration read endpoint, in public preview as of this writing, - returns the fields the snapshot needs and what credential the snapshot script - must hold to call it (section 13). + `/verify` requires, and whether that invocation can carry the model choice the + section 14 policy prefers (section 12); whether delegation reliably opens a + draft pull request (section 11); whether assigning an issue to Copilot offers + the choice of agent profile, or selection needs the Agents panel (section 12); + how the assessment's target project reaches the setup steps that run + collection, or whether collection needs a different trigger (section 13); + whether a full phase A draft, with its setup-step collection sharing the + envelope, fits the session cap or the drafting needs decomposing (section 11); + whether effort level can be pinned in a profile or anywhere else, the one + model-choice question the documentation leaves open (section 14); and whether + the cloud-agent configuration read endpoint, in public preview as of this + writing, returns the fields the snapshot needs and what credential the + snapshot script must hold to call it (section 13). - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase one. It would be human-run with its own credential and the project's explicit From a58556b0dbaf3b88582310c6426a69e51cef462a Mon Sep 17 00:00:00 2001 From: Nate W Date: Wed, 12 Aug 2026 14:08:41 -0700 Subject: [PATCH 39/47] spec: bring concurrent assessments into scope Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 30 +++++++++++++++---- 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 55553a3..deb377c 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -572,6 +572,21 @@ filtered by phase label is the portfolio view, with no dashboard to build. the reading starts from one place. This reports on G-1 without adding a log or an acceptance gate. +Assessments run concurrently; the lifecycle is sequential within an assessment, +never across assessments. The bindings above need no coordination between two +assessments in flight because each one resolves its assessment from the artifact +it fires on: a command from the issue or PR it is commented on, the +phase-advance workflow from the tracking issue the merged PR links. Each +assessment keeps its own namespace, the intake issue and the deliverable +directory (`analyses///`, section 13), one assessment per project +per year, with the tracking-issue link authoritative and the path the +human-readable key. Admission control is `/accept` itself: eligible issues wait +until a writer with review capacity takes one, so how many assessments run at +once is bounded by the humans available, not by a mechanism (P-1). What +concurrent assessments do share is the credit pool, watched by the +administrator/platform owner (section 11), and the small team sustaining +approver independence across them (section 17). + ## 13. Components and repository layout Everything the system is made of lives in cncf/techdocs, versioned and @@ -843,8 +858,10 @@ The build steps, in dependency order, each sized to one issue: 8. Phase-advance workflow (`.github/workflows/assessment-phase.yml`). Done when a merged fixture PR opens the next phase's tracking issue, linked and unassigned, on both the default route and the phase B skip route, posts the - transition on the intake issue and swaps its phase label, and the final - phase's merge closes the intake (section 12). + transition on the intake issue and swaps its phase label, closes the intake + on the final phase's merge, and, with two fixture assessments in flight, + lands every action on the intake the merged PR links, never the other + (section 12). 9. Command workflow (`.github/workflows/assessment-commands.yml` plus parser in `scripts/assessment/`). Done when each command performs its section 12 actions for an authorized commenter, refuses an unauthorized one with a @@ -858,8 +875,9 @@ The build steps, in dependency order, each sized to one issue: The build is complete when steps 1 through 9 are merged, the section 17 build-time checks have recorded answers, and one end-to-end walkthrough of the -section 5 lifecycle on a fixture project has run clean. Then the pilot is -chosen, deliberately, per the caveat in section 10. +section 5 lifecycle on a fixture project has run clean with a second fixture +assessment in flight, proving assessments do not cross (section 12). Then the +pilot is chosen, deliberately, per the caveat in section 10. ## 17. Open questions and future work @@ -902,8 +920,8 @@ chosen, deliberately, per the caveat in section 10. (section 10). - Small-team staffing. Sustaining the approver separation (a phase's approver must be neither the drafter nor a reviewer of that phase; sections 4, 10) when - the same few writers wear multiple hats. -- Scaling. Running assessments for several projects concurrently. + the same few writers wear multiple hats, felt sooner now that assessments run + concurrently (section 12). --- From 619189b87a150185e8f795bbaf25251a7dee4924 Mon Sep 17 00:00:00 2001 From: Nate W Date: Wed, 12 Aug 2026 15:58:01 -0700 Subject: [PATCH 40/47] spec: authorize /confirm against a stakeholder set frozen at acceptance Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 71 +++++++++++-------- 1 file changed, 40 insertions(+), 31 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index deb377c..21d5723 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -467,12 +467,13 @@ parses the command, authorizes the commenter against the role bindings below, and performs the corresponding native operations: a command is honored only from the role the lifecycle assigns that act, so the write-gated commands grant no power the commenter does not already hold, and `/confirm` is authorized by the -stakeholder names in the intake rather than by repository access (see -Stakeholder review below). Commands are deterministic bookkeeping fired by a -recorded human decision (HC-4); the one that starts a model run, `/verify`, is -an explicit delegation by the phase's reviewer (P-1). Labels applied by the -commands and workflows record where each assessment stands: an issue search -filtered by phase label is the portfolio view, with no dashboard to build. +stakeholder set validated and frozen at acceptance rather than by repository +access (see Stakeholder review below). Commands are deterministic bookkeeping +fired by a recorded human decision (HC-4); the one that starts a model run, +`/verify`, is an explicit delegation by the phase's reviewer (P-1). Labels +applied by the commands and workflows record where each assessment stands: an +issue search filtered by phase label is the portfolio view, with no dashboard to +build. - Request (step 1): an issue. Phase A's request is the intake issue (section 7); phase B's and C's tracking issues are opened automatically when the previous @@ -492,17 +493,23 @@ filtered by phase label is the portfolio view, with no dashboard to build. `needs-triage` for `triage/accepted`. For phase A, the same command posts the mention of the named project contacts on the intake issue, so the AI disclosure reaches the project before drafting starts, whoever filed the - request, and the step cannot be forgotten (HC-6, section 7). The writer then - delegates to the agent directly, selecting the phase's drafting profile - (section 14); the command does not start the agent, because delegation carries - choices a comment cannot: the profile and the model picker (section 14). - Because only users with write access can delegate (section 11), P-1 is - enforced by repository permissions, not convention, and the workflow refuses - `/accept` from a commenter without write access. For phases B and C, whose - tracking issues are separate from the intake, the workflow also notes the - acceptance on the intake issue: an unaccepted eligible phase is the stall - mode, so the umbrella shows when work actually started and by whom. The triage - verdict has a counterpart: a writer declines an unaccepted request with + request, and the step cannot be forgotten (HC-6, section 7). Accepting also + freezes who may later confirm: the intake's contact list is requester-supplied + and its body stays editable, so the writer validates the named contacts + against the project's own public records (maintainer lists, governance files) + before accepting, and the workflow's acceptance comment records the validated + set, which `/confirm` authorizes against; a later phase's acceptance may + revalidate and update the set the same way (HC-2). The writer then delegates + to the agent directly, selecting the phase's drafting profile (section 14); + the command does not start the agent, because delegation carries choices a + comment cannot: the profile and the model picker (section 14). Because only + users with write access can delegate (section 11), P-1 is enforced by + repository permissions, not convention, and the workflow refuses `/accept` + from a commenter without write access. For phases B and C, whose tracking + issues are separate from the intake, the workflow also notes the acceptance on + the intake issue: an unaccepted eligible phase is the stall mode, so the + umbrella shows when work actually started and by whom. The triage verdict has + a counterpart: a writer declines an unaccepted request with `/decline `, under the same authorization as `/accept`; the workflow labels it `triage/declined`, posts the reason, and closes the issue as not planned. Declining the intake means the assessment never starts. Declining a @@ -530,17 +537,18 @@ filtered by phase label is the portfolio view, with no dashboard to build. 17). - Stakeholder review (step 5): mention, not access. The reviewer comments `/ready`: the workflow marks the PR ready for review and mentions the - stakeholders named in the intake, one act instead of two. The draft-to-ready - flip carries the real-world meaning: draft while the assessing team is still - working the deliverable over, ready when they would put it in front of the - requesting project. cncf/techdocs is public, so stakeholders can review and - comment without any special access; formal review requests are limited to - collaborators, which is why the binding is a mention plus a factual-accuracy - confirmation, recorded by the stakeholder commenting `/confirm`, which the - workflow checks against the names in the intake, no repository access - required, and records by applying the `confirmed` label to the PR. Any - disagreement with the conclusions is recorded in the deliverable itself - (HC-2). + stakeholders from the set frozen at acceptance, one act instead of two. The + draft-to-ready flip carries the real-world meaning: draft while the assessing + team is still working the deliverable over, ready when they would put it in + front of the requesting project. cncf/techdocs is public, so stakeholders can + review and comment without any special access; formal review requests are + limited to collaborators, which is why the binding is a mention plus a + factual-accuracy confirmation, recorded by the stakeholder commenting + `/confirm`, which the workflow checks against the stakeholder set frozen at + acceptance (see Accept above), never the requester-editable intake body, no + repository access required, and records by applying the `confirmed` label to + the PR. Any disagreement with the conclusions is recorded in the deliverable + itself (HC-2). - Merge (step 6): merge plus one workflow. The approver (section 4) merges, checking the `confirmed` label is present first: stakeholder confirmation is required to advance (step 5, HC-2), and because a ready PR awaiting @@ -866,9 +874,10 @@ The build steps, in dependency order, each sized to one issue: `scripts/assessment/`). Done when each command performs its section 12 actions for an authorized commenter, refuses an unauthorized one with a visible reply, `/ready` refuses without the `verified` label, `/confirm` from - an intake-named stakeholder applies the `confirmed` label, a manually applied - label is honored, and the parser's unit tests cover recognition, argument - handling, and authorization decisions. + a stakeholder in the acceptance-frozen set applies the `confirmed` label and + from a name outside it is refused, a manually applied label is honored, and + the parser's unit tests cover recognition, argument handling, and + authorization decisions. 10. Approver-independence check (advisory; a build-plan candidate from section 12). Comments when a phase's approver also drafted or reviewed it; never blocks. Done when it flags a staged violation, or explicitly deferred. From 761709b1d1eb34350701f440925466377c254967 Mon Sep 17 00:00:00 2001 From: Nate W Date: Wed, 12 Aug 2026 16:47:02 -0700 Subject: [PATCH 41/47] spec: test deterministically in CI, live on the consenting pilot Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 72 ++++++++++++------- 1 file changed, 48 insertions(+), 24 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 21d5723..e78c81a 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -836,6 +836,24 @@ cncf/techdocs, MCP left at its read-only default, writer access confirmed, and the credit cap reviewed. These are organization and repository settings, not files, so this step is recorded in its tracking issue rather than a PR. +Test discipline, for every step: the done-when below is the step's acceptance +test. Deterministic components, the parser, the collection scripts, the checks +(section 13), are developed test-first, the failing test written before the +component that must pass it, and their tests run in CI on every pull request. +The workflows stay thin glue over tested scripts because logic in a script can +be asserted while logic in workflow YAML can only be exercised live. What CI +cannot assert, agent behavior and the workflows' live wiring, is tested in the +live environment, deliberately: during the build, each step's own delegations +and draft PRs are the test bed, closed unmerged and linked to the step's +tracking issue so they read as what they are; the first assessment is the +end-to-end test, run with a pilot project that agreed to exactly that role +(section 10). There is no separate test repository and no fixture assessment: +the workflows automate bookkeeping a human can do by hand (HC-4, section 12), so +a wiring fault found live is recovered by hand and fixed forward, never +rehearsed against fake assessments in the production tracker. Agent runs draw on +the credit pool (section 11), so a profile change is exercised deliberately, +before it merges, never per push. + The build steps, in dependency order, each sized to one issue: 1. Labels (`.github/settings.yml`). Deliberately trivial first delegation: its @@ -856,37 +874,43 @@ The build steps, in dependency order, each sized to one issue: committed command against a sample project twice yields the same committed outputs (HC-5), and a delegated session's draft PR contains the outputs its setup steps produced. -6. Provenance check (workflow plus `scripts/assessment/`). Done when a fixture - PR with a malformed or unfilled block fails, a well-formed one passes, and - the draft-versus-ready leniency behaves as section 15 specifies. +6. Provenance check (workflow plus `scripts/assessment/`). Done when the check's + unit tests cover the malformed, unfilled, and well-formed block shapes and + the draft-versus-ready leniency as section 15 specifies, and the step's own + draft PR exercises the workflow wiring live. 7. Agent profiles (`.github/agents/`, one PR per profile). The three drafters, - then the verifier (section 14). Done when each profile, run against a - fixture, writes only its declared outputs with the provenance block present, - and the verifier's report correctly flags a planted unsupported claim. + then the verifier (section 14). Done when each profile, delegated during the + build against a sample target and its draft closed unmerged, writes only its + declared outputs with the provenance block present, and the verifier's report + correctly flags an unsupported claim planted in that draft. 8. Phase-advance workflow (`.github/workflows/assessment-phase.yml`). Done when - a merged fixture PR opens the next phase's tracking issue, linked and - unassigned, on both the default route and the phase B skip route, posts the - transition on the intake issue and swaps its phase label, closes the intake - on the final phase's merge, and, with two fixture assessments in flight, - lands every action on the intake the merged PR links, never the other - (section 12). + its logic lives in a script whose unit tests cover opening the next phase's + tracking issue, linked and unassigned, on both the default route and the + phase B skip route, posting the transition on the intake issue and swapping + its phase label, closing the intake on the final phase's merge, and, with two + assessments in flight, landing every action on the intake the merged PR + links, never the other (section 12). The live wiring is verified on the + pilot's first phase transition, with manual bookkeeping as the fallback + (HC-4). 9. Command workflow (`.github/workflows/assessment-commands.yml` plus parser in - `scripts/assessment/`). Done when each command performs its section 12 - actions for an authorized commenter, refuses an unauthorized one with a - visible reply, `/ready` refuses without the `verified` label, `/confirm` from - a stakeholder in the acceptance-frozen set applies the `confirmed` label and - from a name outside it is refused, a manually applied label is honored, and - the parser's unit tests cover recognition, argument handling, and - authorization decisions. + `scripts/assessment/`). Done when the parser's unit tests cover recognition, + argument handling, and every section 12 action and authorization decision: an + unauthorized commenter refused with a visible reply, `/ready` refusing + without the `verified` label, `/confirm` from a stakeholder in the + acceptance-frozen set applying the `confirmed` label and from a name outside + it refused, and a manually applied label honored. The live wiring is verified + on each command's first real use, with manual bookkeeping as the fallback + (HC-4). 10. Approver-independence check (advisory; a build-plan candidate from section 12). Comments when a phase's approver also drafted or reviewed it; never blocks. Done when it flags a staged violation, or explicitly deferred. -The build is complete when steps 1 through 9 are merged, the section 17 -build-time checks have recorded answers, and one end-to-end walkthrough of the -section 5 lifecycle on a fixture project has run clean with a second fixture -assessment in flight, proving assessments do not cross (section 12). Then the -pilot is chosen, deliberately, per the caveat in section 10. +The build is complete when steps 1 through 9 are merged and the section 17 +build-time checks have recorded answers. There is no fixture walkthrough: the +pilot, chosen deliberately per the caveat in section 10 and run with a project +that agreed to be the live test, is the end-to-end test of the section 5 +lifecycle, and the concurrency claim is verified the first time a second +assessment overlaps it (section 12). ## 17. Open questions and future work From 9aa7dd10f26817d4e871aa68594ea8353fdc7ce8 Mon Sep 17 00:00:00 2001 From: Nate W Date: Wed, 12 Aug 2026 16:57:49 -0700 Subject: [PATCH 42/47] spec: close label-trust, terminal-state, and site-pin gaps Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 63 ++++++++++--------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index e78c81a..365551b 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -514,9 +514,9 @@ build. labels it `triage/declined`, posts the reason, and closes the issue as not planned. Declining the intake means the assessment never starts. Declining a phase B or C tracking issue ends the assessment there, noted on the intake - issue, which the workflow closes, so an open intake still means work in - flight. A project's request to stop is honored the same way: the writer - records a decline rather than leaving the request to sit unanswered. + issue, which the workflow closes as not planned, so an open intake still means + work in flight. A project's request to stop is honored the same way: the + writer records a decline rather than leaving the request to sit unanswered. - Draft (step 3): a draft pull request. The agent works on its own branch in cncf/techdocs and opens a draft PR linked to the tracking issue, carrying the provenance block (section 15). @@ -550,29 +550,33 @@ build. the PR. Any disagreement with the conclusions is recorded in the deliverable itself (HC-2). - Merge (step 6): merge plus one workflow. The approver (section 4) merges, - checking the `confirmed` label is present first: stakeholder confirmation is - required to advance (step 5, HC-2), and because a ready PR awaiting - stakeholders is a normal state rather than a failure, the check is the - approver's duty, not CI's. A merge-triggered GitHub Actions workflow (standard - `GITHUB_TOKEN` with `issues: write`) then opens the next phase's tracking - issue, labeled for its phase and `needs-triage`, linking the intake and the - merged deliverable, and assigns no one: eligible, not started (HC-2, P-1). The - reviewer records the phase B skip decision, the joint call with stakeholders - (section 6), with `/skip-implementation` on the phase A PR, applying the label - the workflow reads to open phase C's tracking issue instead. The same workflow - keeps the umbrella current: it posts the transition on the intake issue, what - merged and what is now eligible with the command that starts it, and swaps the - intake's phase label. The final phase's merge closes the intake issue: an open - intake is an assessment in flight, a closed one is done. Approver independence - cannot be natively enforced by GitHub; it is verifiable from the public PR - record, and an advisory CI check that flags a violation is a build-plan - candidate (section 16). + first checking the confirmation behind the `confirmed` label: the label is the + record, mutable by any collaborator with triage access, so what the approver + reads is the `/confirm` on the PR timeline from a stakeholder in the + acceptance-frozen set (HC-2). Stakeholder confirmation is required to advance + (step 5), and because a ready PR awaiting stakeholders is a normal state + rather than a failure, the check is the approver's duty, not CI's. A + merge-triggered GitHub Actions workflow (standard `GITHUB_TOKEN` with + `issues: write`) then opens the next phase's tracking issue, labeled for its + phase and `needs-triage`, linking the intake and the merged deliverable, and + assigns no one: eligible, not started (HC-2, P-1). The reviewer records the + phase B skip decision, the joint call with stakeholders (section 6), with + `/skip-implementation` on the phase A PR, applying the label the workflow + reads to open phase C's tracking issue instead. The same workflow keeps the + umbrella current: it posts the transition on the intake issue, what merged and + what is now eligible with the command that starts it, and swaps the intake's + phase label. The final phase's merge closes the intake issue as completed: an + open intake is an assessment in flight, a closed one has ended, and the close + reason and labels say how it ended, completed at the final merge, not planned + on a decline or an abort. Approver independence cannot be natively enforced by + GitHub; it is verifiable from the public PR record, and an advisory CI check + that flags a violation is a build-plan candidate (section 16). - Failure and abort leave a trail. The reviewer discards a draft (section 5) with `/discard `, which closes the PR with the reason recorded in the command comment and notes the discard on the intake issue; the tracking issue stays open for a restart or a hand-written phase. An administrator/platform - owner abort closes the tracking issue and the intake with it, the rationale - recorded on both, so an open intake still means work in flight. + owner abort closes the tracking issue and the intake with it as not planned, + the rationale recorded on both, so an open intake still means work in flight. - The timeline records elapsed time. Gate transitions (opened, accepted, draft PR, ready, approved, merged) are timestamped in issue and PR history, so how long a pilot took, and where the time went, can be read from the GitHub @@ -779,12 +783,15 @@ Its fields, each a labeled bullet: - Data. The data-collection commands that were run and the committed paths of their outputs, so every quantitative claim traces to a reproducible step (HC-5), plus the commit SHA of each assessed repository at collection time and - the retrieval date for each live site. The pins record which state of the - sources each claim describes: a pinned repository lets the reviewer check out - exactly what the drafter saw (section 10), a dated site bounds when the claim - held, and when the sources move on, drift reads as drift rather than error. - They promise no re-run of the assessment itself, which is judgment: two honest - runs converge without matching. + the retrieval date for each live site. A dated site is not a bare reference: + what collection fetched from it is among the committed outputs, so the + snapshot the drafter saw is in the tree and the date labels it (P-4). The pins + record which state of the sources each claim describes: a pinned repository + lets the reviewer check out exactly what the drafter saw (section 10), a dated + site's claims check against its committed fetch, and when the sources move on, + drift reads as drift rather than error. They promise no re-run of the + assessment itself, which is judgment: two honest runs converge without + matching. - Verification record. Which findings the reviewer verified against source, satisfying the section 10 floor (HC-7): the rating-bearing findings in phase A, the claims grounding the recommendations in phases B and C. From 0bdb6024b36afa683b796fd511068ac4312a812b Mon Sep 17 00:00:00 2001 From: Nate W Date: Wed, 12 Aug 2026 17:34:57 -0700 Subject: [PATCH 43/47] spec: add a participant guide so the system documents itself Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 45 +++++++++++++------ 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 365551b..bb111a8 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -538,17 +538,19 @@ build. - Stakeholder review (step 5): mention, not access. The reviewer comments `/ready`: the workflow marks the PR ready for review and mentions the stakeholders from the set frozen at acceptance, one act instead of two. The - draft-to-ready flip carries the real-world meaning: draft while the assessing - team is still working the deliverable over, ready when they would put it in - front of the requesting project. cncf/techdocs is public, so stakeholders can - review and comment without any special access; formal review requests are - limited to collaborators, which is why the binding is a mention plus a - factual-accuracy confirmation, recorded by the stakeholder commenting - `/confirm`, which the workflow checks against the stakeholder set frozen at - acceptance (see Accept above), never the requester-editable intake body, no - repository access required, and records by applying the `confirmed` label to - the PR. Any disagreement with the conclusions is recorded in the deliverable - itself (HC-2). + mention carries a link to the participant guide (section 13), so what is being + asked and how to answer it arrive with the ask. The draft-to-ready flip + carries the real-world meaning: draft while the assessing team is still + working the deliverable over, ready when they would put it in front of the + requesting project. cncf/techdocs is public, so stakeholders can review and + comment without any special access; formal review requests are limited to + collaborators, which is why the binding is a mention plus a factual-accuracy + confirmation, recorded by the stakeholder commenting `/confirm`, which the + workflow checks against the stakeholder set frozen at acceptance (see Accept + above), never the requester-editable intake body, no repository access + required, and records by applying the `confirmed` label to the PR. Any + disagreement with the conclusions is recorded in the deliverable itself + (HC-2). - Merge (step 6): merge plus one workflow. The approver (section 4) merges, first checking the confirmation behind the `confirmed` label: the label is the record, mutable by any collaborator with triage access, so what the approver @@ -652,12 +654,20 @@ points at the methodology, it never restates it (P-2). issue plus an index, if the section 6 proposal is accepted into the methodology first; until then, the combined file the methodology prescribes (P-2). +- Participant guide: `docs/assessment-guide.md`. The lifecycle from the outside, + one page for the three audiences the system asks something of: how a project + requests an assessment and what AI involvement it consents to (HC-6), what + `/confirm` asks of a stakeholder and who may give it, and the operator's + reference for the commands, the labels, and the by-hand bookkeeping that + stands in when a workflow fails (HC-4). Command replies and mentions link to + it, so the explanation arrives with the ask. How-to-run guidance only: it + references the methodology, never restates it (P-2). Notes: - The methodology corpus at [docs/analysis/][analysis-dir] is deliberately absent from this list: the system reads it, but owns no file in it (P-2, - NG-4). This spec likewise lives outside the corpus. + NG-4). This spec and the participant guide likewise live outside the corpus. - The backlog file naming across existing analyses already varies (`issues.md`, `issues-list.md`); whatever the section 6 decision, the layout above pins one convention going forward. @@ -908,11 +918,18 @@ The build steps, in dependency order, each sized to one issue: it refused, and a manually applied label honored. The live wiring is verified on each command's first real use, with manual bookkeeping as the fallback (HC-4). -10. Approver-independence check (advisory; a build-plan candidate from section +10. Participant guide (`docs/assessment-guide.md`, section 13). Late in the + order because it documents the system that now exists. Done when review + confirms every ask the lifecycle makes of a requester, stakeholder, or + operator appears with the section 12 behavior it triggers, including the + by-hand procedure a failed workflow falls back to (HC-4), referencing the + methodology rather than restating it (P-2). The pilot is its live test: a + participant who needs an ad hoc explanation has found a guide defect. +11. Approver-independence check (advisory; a build-plan candidate from section 12). Comments when a phase's approver also drafted or reviewed it; never blocks. Done when it flags a staged violation, or explicitly deferred. -The build is complete when steps 1 through 9 are merged and the section 17 +The build is complete when steps 1 through 10 are merged and the section 17 build-time checks have recorded answers. There is no fixture walkthrough: the pilot, chosen deliberately per the caveat in section 10 and run with a project that agreed to be the live test, is the end-to-end test of the section 5 From e192802dee69f3a2504ae40a851d32cd80462cb4 Mon Sep 17 00:00:00 2001 From: Nate W Date: Thu, 13 Aug 2026 11:46:08 -0700 Subject: [PATCH 44/47] spec: name budget as the constraint and experts as the loop Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 48 +++++++++++-------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index bb111a8..106e3f4 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -19,15 +19,16 @@ tooling, and the plan for building the system. A CNCF TechDocs assessment evaluates a project's documentation against the TechDocs [criteria] and produces a prioritized improvement plan. It is valuable and in demand, but each assessment takes 6–8 weeks of a skilled technical -writer's time, and that expertise is scarce. Throughput is the constraint: more -projects want assessments than the expert pool can serve. +writer's time, and that time is what the program pays for. The constraint is +budget, which can only fund a few assessments a year, so more projects want +assessments than the program can serve. Two things make now the moment to change that: -- The method is sound; the throughput isn't. The existing methodology +- The method is sound; the economics aren't. The existing methodology ([criteria], the [howto], and the analysis, implementation, and issues - deliverables) is well-established. The problem to solve is speed and scale, - not what a good assessment is. + deliverables) is well-established. The problem to solve is what an assessment + costs in expert hours, not what a good assessment is. - Maintainers now have agentic AI. GitHub has granted CNCF maintainers [Copilot Enterprise][copilot-blog]. That lets us put an AI agent on the mechanical and first-draft work, supervised by a human expert, to compress the cycle. We @@ -42,15 +43,18 @@ the labor. AI drafts, humans decide, all in the open. ### Goals - G-1: Compress the cycle. Reduce the 6–8 week assessment toward roughly 2 - weeks. This is the motivation and a pilot hypothesis, not an acceptance - criterion (section 10): the pilot succeeds or fails on quality and safety, and - the elapsed time simply gets reported. + weeks, so the budget that funds assessments serves more projects (section 1). + This is the motivation and a pilot hypothesis, not an acceptance criterion + (section 10): the pilot succeeds or fails on quality and safety, and the + elapsed time simply gets reported. - G-2: Hold quality. Output meets or exceeds the current human baseline (Flatcar, Knative, Helm), verified by the method in section 10, not asserted. AI drafts fail by being fluent, well-formatted, and generically wrong, a mode that survives a readability pass, so catching it is an explicit requirement. -- G-3: Human in the loop throughout. The agent proposes; a technical writer and - project stakeholders dispose. Every phase is reviewed before it advances. +- G-3: Expert in the loop throughout. The agent proposes; an experienced + technical writer and project stakeholders dispose. The reviewer has to know + what good looks like, which is why the role takes an experienced writer, not + just any human (section 4). Every phase is reviewed before it advances. - G-4: Work in the open. The whole process happens on cncf/techdocs via issues and PRs, and the deliverables are public. Assessments can be candid about a project's documentation gaps; publishing in the open is deliberate and shapes @@ -128,13 +132,13 @@ the labor. AI drafts, humans decide, all in the open. so a write elsewhere is impossible, not merely disallowed. Full design in section 8. - HC-2: Phase gating. Each phase (assessment, implementation, backlog) is - reviewed by a technical writer, then goes to stakeholders before the next - begins. Sign-off separates two things. A factual-accuracy check: stakeholders - confirm they have read the deliverable and had the chance to correct factual - errors, which is required to advance. Agreement with the conclusions: not - required, and a project's disagreement is recorded in the deliverable rather - than allowed to block or soften it. Merging phase N makes phase N+1 eligible; - a technical writer still starts it (section 5). + reviewed by an experienced technical writer, then goes to stakeholders before + the next begins. Sign-off separates two things. A factual-accuracy check: + stakeholders confirm they have read the deliverable and had the chance to + correct factual errors, which is required to advance. Agreement with the + conclusions: not required, and a project's disagreement is recorded in the + deliverable rather than allowed to block or soften it. Merging phase N makes + phase N+1 eligible; a technical writer still starts it (section 5). - HC-3: Deliverables are files in cncf/techdocs. Including the issue backlog, in whatever layout the methodology prescribes (section 6 proposes one file per issue). Nothing is filed to an external repo. @@ -179,9 +183,11 @@ them; they may not replace them. - Reviewer. Accepts a request to begin work, then reviews and refines each draft in conversation, verifies findings against source (HC-7), and marks it ready. Owns the draft's quality, but does not give its final sign-off; that is - independent (see Approver). A human role: it starts from the verifier's report - and the machine checks (provenance lint, template conformance), but the - verification and the readiness call are the reviewer's own. + independent (see Approver). An expert role, filled by an experienced technical + writer: the reviewer must know what a good assessment looks like. The reviewer + starts from the verifier's report and the machine checks (provenance lint, + template conformance), but the verification and the readiness call are the + reviewer's own. - Stakeholders. The project party with a stake in its documentation and direction: maintainers, tech leads, trusted community members, and others. They set priorities, provide project-level context, and confirm factual @@ -195,7 +201,7 @@ them; they may not replace them. - Writers. The audience for the issue backlog: community members who take up the resulting issues and do the documentation work. - Approver. Gives each phase its independent quality sign-off and merges the PR. - A qualified technical writer who was neither the drafter nor a reviewer of + An experienced technical writer who was neither the drafter nor a reviewer of that phase, so no one signs off on work they produced or refined, and the role spreads across the team instead of bottlenecking. Confirms the verification in section 10 was done. From 9e91a8a623583da6013ae126aa159ced1219476f Mon Sep 17 00:00:00 2001 From: Nate W Date: Thu, 13 Aug 2026 12:07:40 -0700 Subject: [PATCH 45/47] answer the pass-eight review: reassessment, manifests, fallbacks Signed-off-by: Nate W Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> --- specs/ai-assisted-techdocs-assessment-spec.md | 87 +++++++++++-------- 1 file changed, 53 insertions(+), 34 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 106e3f4..23e41da 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -74,10 +74,11 @@ the labor. AI drafts, humans decide, all in the open. file, PR, issue, comment, or setting outside cncf/techdocs. (Hard constraint; see HC-1 in section 3.) - NG-2: No auto-filing of issues. The backlog is delivered as files in - cncf/techdocs. Filing them as real issues in project repos is future work (a - separate, opt-in script; see section 17), explicitly out of scope now. The - per-file backlog format proposed in section 6 is chosen partly to make that - future script straightforward. + cncf/techdocs. The methodology's follow-up, a human filing those issues in the + project's repository ([howto]), remains as it is today, human-run and outside + this system's boundary; what is out of scope is automating it (a separate, + opt-in script; see section 17). The per-file backlog format proposed in + section 6 is chosen partly to make that future script straightforward. - NG-3: Not a replacement for human judgment. No unattended, end-to-end autonomous assessments; no bypassing writer review or stakeholder sign-off. - NG-4: Not a change to the criteria or evaluation method. We automate the @@ -173,8 +174,9 @@ them; they may not replace them. - Drafter. Produces the first-pass draft of each phase's deliverable as a draft PR, then refines it in conversation with the reviewer. - Verifier. An agent that runs an adversarial fact-check pass over each draft - before human review: resolving citations, checking claims against the sources - they cite, and flagging anything unsupported. It is prompted to find + before human review (a reviewer may skip a phase's pass deliberately, + recording why; section 5): resolving citations, checking claims against the + sources they cite, and flagging anything unsupported. It is prompted to find unsupported claims, not to confirm the draft, making it less likely to inherit the drafter's blind spots. Its report feeds the reviewer's verification (HC-7); it never substitutes for it. The layering exists because a fabrication @@ -266,8 +268,9 @@ end up slower than doing it by hand. drafter produces the single combined file, and the split layout waits for the methodology, never the other way around. -Every deliverable carries a header noting it was AI-drafted and human-reviewed -(HC-6). +Every deliverable carries a provenance header recording how it was produced: +AI-drafted and human-reviewed in the normal path, hand-written when a reviewer +restarts by hand (section 15, HC-6). Scope note: phase one produces plans (assessment, implementation, backlog), not documentation changes, and the backlog lands as files in cncf/techdocs rather @@ -418,8 +421,11 @@ infrastructure we build and maintain ourselves. automatically in the pull request, which feeds the safety audit in section 10. The firewall's scope is the agent's own session: the environment-setup workflow (see Execution environment below) runs outside it, so what setup - fetches is governed by review of that versioned workflow file, not by the - firewall. + fetches is governed not by the firewall but by review of that versioned + workflow file and by where the collection scripts are pointed: the + documentation domains recorded at acceptance, the same human-reviewed list the + allowlist is built from, with every fetch's output committed as evidence + (HC-5). - MCP policy. The firewall does not apply to MCP servers, and MCP tools are one of the two documented mechanisms that can widen the agent's write reach (a secret is the other; see the next bullet), so [MCP @@ -598,14 +604,17 @@ assessments in flight because each one resolves its assessment from the artifact it fires on: a command from the issue or PR it is commented on, the phase-advance workflow from the tracking issue the merged PR links. Each assessment keeps its own namespace, the intake issue and the deliverable -directory (`analyses///`, section 13), one assessment per project -per year, with the tracking-issue link authoritative and the path the -human-readable key. Admission control is `/accept` itself: eligible issues wait -until a writer with review capacity takes one, so how many assessments run at -once is bounded by the humans available, not by a mechanism (P-1). What -concurrent assessments do share is the credit pool, watched by the -administrator/platform owner (section 11), and the small team sustaining -approver independence across them (section 17). +directory (`analyses///`, section 13), with the tracking-issue +link authoritative and the path the human-readable key. A completed assessment +never blocks a new request: a project reassessed within the same year, say for a +maturity move, takes a distinct directory named at acceptance (a suffix on the +default path). What `/accept` refuses is a duplicate request for a project whose +assessment is still in flight, pointing at the open intake instead. Admission +control is `/accept` itself: eligible issues wait until a writer with review +capacity takes one, so how many assessments run at once is bounded by the humans +available, not by a mechanism (P-1). What concurrent assessments do share is the +credit pool, watched by the administrator/platform owner (section 11), and the +small team sustaining approver independence across them (section 17). ## 13. Components and repository layout @@ -636,8 +645,12 @@ points at the methodology, it never restates it (P-2). - Data collection: `scripts/assessment/`. The deterministic inventory scripts behind HC-5 (section 15). The drafting session's setup steps (section 11) run them and leave the outputs in the workspace; the drafter commits them - unmodified with the draft. Unmodified is checkable, not assumed: re-running - the committed command reproduces the committed outputs or exposes the edit. + unmodified with the draft. Unmodified is checkable, not assumed: the scripts + write a content-hash manifest alongside the outputs, so an edit after + collection breaks the manifest, and re-running the committed command + reproduces the outputs drawn from pinned sources; a live site's fetch is + vouched by the manifest alone, since a later refetch reads ordinary drift, not + tampering (P-4). - Labels: `.github/settings.yml`. The repository's existing `Docs analysis` label marks assessment work, unchanged from the human-run analyses; AI involvement is indicated by the provenance block and the disclosure, not a @@ -918,12 +931,13 @@ The build steps, in dependency order, each sized to one issue: 9. Command workflow (`.github/workflows/assessment-commands.yml` plus parser in `scripts/assessment/`). Done when the parser's unit tests cover recognition, argument handling, and every section 12 action and authorization decision: an - unauthorized commenter refused with a visible reply, `/ready` refusing - without the `verified` label, `/confirm` from a stakeholder in the - acceptance-frozen set applying the `confirmed` label and from a name outside - it refused, and a manually applied label honored. The live wiring is verified - on each command's first real use, with manual bookkeeping as the fallback - (HC-4). + unauthorized commenter refused with a visible reply, a second `/accept` for a + project whose assessment is still in flight refused with a pointer to the + open intake, `/ready` refusing without the `verified` label, `/confirm` from + a stakeholder in the acceptance-frozen set applying the `confirmed` label and + from a name outside it refused, and a manually applied label honored. The + live wiring is verified on each command's first real use, with manual + bookkeeping as the fallback (HC-4). 10. Participant guide (`docs/assessment-guide.md`, section 13). Late in the order because it documents the system that now exists. Done when review confirms every ask the lifecycle makes of a requester, stakeholder, or @@ -936,11 +950,15 @@ The build steps, in dependency order, each sized to one issue: blocks. Done when it flags a staged violation, or explicitly deferred. The build is complete when steps 1 through 10 are merged and the section 17 -build-time checks have recorded answers. There is no fixture walkthrough: the -pilot, chosen deliberately per the caveat in section 10 and run with a project -that agreed to be the live test, is the end-to-end test of the section 5 -lifecycle, and the concurrency claim is verified the first time a second -assessment overlaps it (section 12). +build-time checks have recorded answers, any negative answer landing on its +documented fallback rather than an unbuilt path. One thing more must exist +before the pilot's first deliverable is scored: the assessment-quality rubric +(sections 10, 17), defined and validated against the human baselines; a pilot +scored without it would leave G-2 asserted, not verified. There is no fixture +walkthrough: the pilot, chosen deliberately per the caveat in section 10 and run +with a project that agreed to be the live test, is the end-to-end test of the +section 5 lifecycle, and the concurrency claim is verified the first time a +second assessment overlaps it (section 12). ## 17. Open questions and future work @@ -978,9 +996,10 @@ assessment overlaps it (section 12). - Assessment-quality rubric. Define the meta-rubric that scores an assessment's quality, distinct from `criteria.md` (which scores a project's docs), and validate it by scoring the Flatcar, Knative, and Helm baselines to set a - reference band (sections 2, 10). This work also sets any post-pilot sampling - rule for verification, replacing the pilot's check-every-finding floor - (section 10). + reference band (sections 2, 10), sequenced before the pilot's first + deliverable is scored (section 16). This work also sets any post-pilot + sampling rule for verification, replacing the pilot's check-every-finding + floor (section 10). - Small-team staffing. Sustaining the approver separation (a phase's approver must be neither the drafter nor a reviewer of that phase; sections 4, 10) when the same few writers wear multiple hats, felt sooner now that assessments run From 2f872d5414bc45966e218b9e5275e0a4bbf418ac Mon Sep 17 00:00:00 2001 From: Nate W Date: Thu, 13 Aug 2026 13:00:59 -0700 Subject: [PATCH 46/47] name the per-phase and skip labels in section 13 Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Nate W --- specs/ai-assisted-techdocs-assessment-spec.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 23e41da..67007a7 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -654,9 +654,11 @@ points at the methodology, it never restates it (P-2). - Labels: `.github/settings.yml`. The repository's existing `Docs analysis` label marks assessment work, unchanged from the human-run analyses; AI involvement is indicated by the provenance block and the disclosure, not a - label. Added for the workflow: per-phase labels, the + label. Added for the workflow: the per-phase labels `phase/assessment`, + `phase/implementation`, and `phase/backlog`, the `needs-triage`/`triage/accepted`/`triage/declined` triage set, the `verified` - and `confirmed` gates (section 12), and the phase B skip marker (section 12), + and `confirmed` gates (section 12), and the phase B skip marker + `skip-implementation`, named for the command that applies it (section 12), managed declaratively alongside the existing label set. - Agent-configuration snapshot: `scripts/assessment/`. The agent's effective configuration (MCP servers, firewall state, and custom allowlist) is readable From 09a92bfdfc47b63c2027ccec4062e4946d7d61c3 Mon Sep 17 00:00:00 2001 From: Nate W Date: Thu, 13 Aug 2026 17:20:50 -0700 Subject: [PATCH 47/47] build components locally with Copilot CLI; move platform observation to a probe delegation Assisted-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Nate W --- specs/ai-assisted-techdocs-assessment-spec.md | 109 ++++++++++-------- 1 file changed, 62 insertions(+), 47 deletions(-) diff --git a/specs/ai-assisted-techdocs-assessment-spec.md b/specs/ai-assisted-techdocs-assessment-spec.md index 67007a7..4801d2e 100644 --- a/specs/ai-assisted-techdocs-assessment-spec.md +++ b/specs/ai-assisted-techdocs-assessment-spec.md @@ -859,20 +859,37 @@ methodology corpus and its templates are not modified (P-2, NG-4). ## 16. Build plan -The system is built the way it runs. Each component in section 13 starts as a -tracking issue in cncf/techdocs, is drafted by the cloud agent where a file is -the deliverable, and lands through a human-reviewed PR: the same -issue-to-draft-to-review loop the assessments will use (section 5). Building the -system with its own loop is the point: by the time the first project is -assessed, every role has rehearsed and the platform's behavior has been observed -rather than assumed. The earliest steps run before the instructions and -environment components exist; the loop holds anyway, just with less scaffolding. - -Preflight, before any build step: the administrator/platform owner works through -the preconditions in section 11: cloud agent and MCP policies enabled for -cncf/techdocs, MCP left at its read-only default, writer access confirmed, and -the credit cap reviewed. These are organization and repository settings, not -files, so this step is recorded in its tracking issue rather than a PR. +The system runs on the cloud agent (section 11) but is not built by it. Where a +file is the deliverable, the component is drafted locally with GitHub Copilot +CLI, which keeps model choice and effort under the operator's direct control, +and lands through a human-reviewed PR. Each component in section 13 starts as a +tracking issue in cncf/techdocs: the same issue-to-draft-to-review loop the +assessments will use (section 5), rehearsed by the human roles on every step. +The CLI performs no GitHub writes; the operator executes every push, pull +request, issue, label, and delegation. Build PRs also get Copilot code review, +requested when the PR opens: with drafting done locally, it restores an +independent automated reader. Its findings are input to the human review; +approval stays with humans (P-1). The cloud agent's own behavior is observed +rather than assumed: a probe delegation (below) and the delegated exercises in +later steps put it on record before the first project is assessed. + +Preflight, before any delegation to the cloud agent: the administrator/platform +owner works through the preconditions in section 11: cloud agent and MCP +policies enabled for cncf/techdocs, MCP left at its read-only default, writer +access confirmed, and the credit cap reviewed. These are organization and +repository settings, not files, so this step is recorded in its tracking issue +rather than a PR. Steps whose deliverable is a locally drafted file do not wait +on preflight; the probe and every delegated exercise do. + +Probe delegation, once preflight closes: the first delegation to the cloud agent +is a deliberately disposable task whose real product is observed platform +behavior: the agent's branch naming, whether delegation reliably opens a draft +pull request, what access review-comment revisions require, whether Actions runs +on agent PRs wait for approval, and the token's effective scopes. The draft +closes unmerged, linked to the probe's tracking issue, and the answers are +recorded against the section 17 checks. The probe runs early because its answers +parameterize later steps, and it may run before the instructions and environment +components exist: the loop holds anyway, just with less scaffolding. Test discipline, for every step: the done-when below is the step's acceptance test. Deterministic components, the parser, the collection scripts, the checks @@ -881,24 +898,21 @@ component that must pass it, and their tests run in CI on every pull request. The workflows stay thin glue over tested scripts because logic in a script can be asserted while logic in workflow YAML can only be exercised live. What CI cannot assert, agent behavior and the workflows' live wiring, is tested in the -live environment, deliberately: during the build, each step's own delegations -and draft PRs are the test bed, closed unmerged and linked to the step's -tracking issue so they read as what they are; the first assessment is the -end-to-end test, run with a pilot project that agreed to exactly that role -(section 10). There is no separate test repository and no fixture assessment: -the workflows automate bookkeeping a human can do by hand (HC-4, section 12), so -a wiring fault found live is recovered by hand and fixed forward, never -rehearsed against fake assessments in the production tracker. Agent runs draw on -the credit pool (section 11), so a profile change is exercised deliberately, -before it merges, never per push. +live environment, deliberately: during the build, the probe delegation and the +delegated exercises later steps require are the test bed, their drafts closed +unmerged and linked to the owning step's tracking issue so they read as what +they are; the first assessment is the end-to-end test, run with a pilot project +that agreed to exactly that role (section 10). There is no separate test +repository and no fixture assessment: the workflows automate bookkeeping a human +can do by hand (HC-4, section 12), so a wiring fault found live is recovered by +hand and fixed forward, never rehearsed against fake assessments in the +production tracker. Agent runs draw on the credit pool (section 11), so a +profile change is exercised deliberately, before it merges, never per push. The build steps, in dependency order, each sized to one issue: -1. Labels (`.github/settings.yml`). Deliberately trivial first delegation: its - real product is observed platform behavior: the agent's branch naming, what - access review-comment revisions require, whether Actions runs on agent PRs - wait for approval, and the token's effective scopes. Done when the labels - exist and the observations are recorded against the section 17 checks. +1. Labels (`.github/settings.yml`). Done when the labels exist on the + repository. 2. Repository instructions (`.github/copilot-instructions.md`). Ground rules for any agent work in this repository, carrying the section 8 discipline. Done when merged after review against sections 8 and 14. @@ -965,24 +979,25 @@ second assessment overlaps it (section 12). ## 17. Open questions and future work - Build-time checks. Platform behavior we could not verify from public - documentation, to be answered by observation during the build (section 16) and - recorded: the agent token's effective permission scopes (section 11); the - agent's branch naming; whether review-comment revisions require write access - (section 12); whether Actions runs on agent PRs wait for approval; whether a - comment-triggered workflow can start the verifier against an existing PR, as - `/verify` requires, and whether that invocation can carry the model choice the - section 14 policy prefers (section 12); whether delegation reliably opens a - draft pull request (section 11); whether assigning an issue to Copilot offers - the choice of agent profile, or selection needs the Agents panel (section 12); - how the assessment's target project reaches the setup steps that run - collection, or whether collection needs a different trigger (section 13); - whether a full phase A draft, with its setup-step collection sharing the - envelope, fits the session cap or the drafting needs decomposing (section 11); - whether effort level can be pinned in a profile or anywhere else, the one - model-choice question the documentation leaves open (section 14); and whether - the cloud-agent configuration read endpoint, in public preview as of this - writing, returns the fields the snapshot needs and what credential the - snapshot script must hold to call it (section 13). + documentation, to be answered by the probe delegation and the delegated + exercises during the build (section 16) and recorded: the agent token's + effective permission scopes (section 11); the agent's branch naming; whether + review-comment revisions require write access (section 12); whether Actions + runs on agent PRs wait for approval; whether a comment-triggered workflow can + start the verifier against an existing PR, as `/verify` requires, and whether + that invocation can carry the model choice the section 14 policy prefers + (section 12); whether delegation reliably opens a draft pull request (section + 11); whether assigning an issue to Copilot offers the choice of agent profile, + or selection needs the Agents panel (section 12); how the assessment's target + project reaches the setup steps that run collection, or whether collection + needs a different trigger (section 13); whether a full phase A draft, with its + setup-step collection sharing the envelope, fits the session cap or the + drafting needs decomposing (section 11); whether effort level can be pinned in + a profile or anywhere else, the one model-choice question the documentation + leaves open (section 14); and whether the cloud-agent configuration read + endpoint, in public preview as of this writing, returns the fields the + snapshot needs and what credential the snapshot script must hold to call it + (section 13). - Filing issues into project repos. A separate, opt-in tool to create the backlog issues in a project's own repository (NG-2). Out of scope for phase one. It would be human-run with its own credential and the project's explicit