Skip to content

Document default policy groups and how to manage them programmatically - #20556

Closed
workprentice[bot] wants to merge 4 commits into
masterfrom
fix_issue_20554_default_policy_groups
Closed

Document default policy groups and how to manage them programmatically#20556
workprentice[bot] wants to merge 4 commits into
masterfrom
fix_issue_20554_default_policy_groups

Conversation

@workprentice

@workprentice workprentice Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Fixes #20554

What changed

content/docs/insights/policy/policy-groups.md — added two new sections:

  • Default policy groups: documents the two auto-created default groups (default-preventative-policy-group, default-audit-policy-group), what auto-membership means for each, that individual members can be removed but the groups themselves can't be deleted, and a caution about adding policy packs to a default group (org-wide mandatory blocking with no staged rollout, plus the client runtime requirement).
  • Managing policy groups programmatically: documents the REST API's PATCH endpoint (additive add*/remove* fields vs. the full-list-replace stacks/policyPacks/insightsAccounts fields, the /batch endpoint) and the pulumiservice provider's PolicyGroupStackAttachment/PolicyGroupInsightsAccountAttachment resources, with a warning against pulumi import-ing a default PolicyGroup resource.
  • Extended "Next steps" with a link to the API reference page.

content/docs/insights/policy/policy-packs/_index.md — added a short "Runtime requirements" section, per the issue's suggestion that this belongs on the policy pack concepts page, cross-linking the project file reference and the new default-group caution.

content/docs/insights/discovery/get-started/add-policies.md — fixed a factual error: an info callout referenced a generic default-policy-group name that doesn't match the two real default group names published on policy-groups.md. Replaced with the correct names and a link.

Verification sources

  • Live OpenAPI spec (https://api.pulumi.com/api/openapi/pulumi-spec.json): confirmed the PATCH /api/orgs/{orgName}/policygroups/{policyGroup} request/response shape (UpdatePolicyGroupRequest, 204 No Content), the required fields on addStack/removeStack (name, routingProject) and addPolicyPack/removePolicyPack (name, displayName, version, versionTag — all four required), the full-list-replace semantics of the plain stacks/policyPacks/insightsAccounts fields, the /batch endpoint, and DeletePolicyGroup's 400 response for an organization's default group.
  • pulumi-pulumiservice provider schema, verified at the released v1.3.0 tag (not master, whose schema.json reflects unreleased breaking changes to PolicyGroup.policyPacks): confirmed the exact resource tokens pulumiservice:api:PolicyGroupStackAttachment and pulumiservice:api:PolicyGroupInsightsAccountAttachment, their required inputs, and that no PolicyGroupPolicyPackAttachment resource exists.

Judgment calls for the reviewer

  1. Runtime-requirement wording. The issue explicitly flags "we need to get the exact requirement here" for whether all Pulumi-published policy packs require Node.js. I could not verify that claim, so I used a generic, sourceable statement instead: a policy pack evaluates locally as part of the CLI invocation, so the runtime declared in its PulumiPolicy.yaml (nodejs, python, or opa) must be present on every client. Please confirm this framing is acceptable, or supply the more specific requirement if one exists.
  2. Deliberately did not publish default-policy-group as a group name, even though it appears in the issue's own curl examples. It contradicts the two specific default-group names already published on policy-groups.md and I couldn't verify it independently, so I treated it as informal/generic phrasing rather than a real identifier.
  3. Did not expand the enforcement-levels list. The API also supports a disabled level (per the OpenAPI spec) that isn't currently documented on policy-groups.md, which only covers Advisory/Mandatory. Out of scope for this PR; flagging as a possible follow-up.
  4. Did not link to registry pages for the new attachment resources (PolicyGroupStackAttachment, PolicyGroupInsightsAccountAttachment) in api-reference.md. Their registry doc pages currently 404 even though the resources exist in the released v1.3.0 package — likely a lag in registry doc generation for this provider release. I mentioned the resources by name (with a code sample) in policy-groups.md without linking to a page that doesn't resolve yet, and left api-reference.md untouched. Happy to add the registry link once those pages exist.
  5. Likely one-line merge conflict with open PR Rename Insights Discovery / Insights Accounts terminology to Discovery / Cloud Accounts #20528 (terminology rename): its diff touches add-policies.md at a line immediately adjacent to the note I corrected. The conflict should be a trivial reconciliation of the terminology wording, whichever PR merges second.

🧠 This PR was created by workprentice on behalf of the Docs Groundskeeper automation — no specific human requester triggered this run; it was generated during a scheduled docs-backlog burndown cycle.

Fixes #20554

- Adds a 'Default policy groups' section to policy-groups.md covering
  the two auto-created default groups (default-preventative-policy-group,
  default-audit-policy-group), their membership semantics, and the
  caution around adding policy packs to them (org-wide blocking with
  mandatory enforcement, client runtime requirements).
- Adds a 'Managing policy groups programmatically' section covering the
  REST API's additive add*/remove* fields vs. the full-list-replace
  stacks/policyPacks/insightsAccounts fields, the /batch endpoint, and
  the pulumiservice provider's PolicyGroupStackAttachment /
  PolicyGroupInsightsAccountAttachment resources, with a warning against
  importing a default PolicyGroup resource.
- Adds a short runtime-requirements note to policy-packs/_index.md per
  the issue's suggestion that it belongs on the policy pack concepts page.
- Fixes a factual error in discovery/get-started/add-policies.md that
  referenced a generic 'default-policy-group' name inconsistent with the
  two real default group names published elsewhere in the docs.
@github-actions github-actions Bot added review:triaging Claude Triage is currently classifying the PR domain:docs PR touches technical docs review:in-progress Claude review is currently running and removed review:triaging Claude Triage is currently classifying the PR labels Jul 29, 2026
@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Pre-merge Review — Last updated 2026-07-29T15:29:39Z

Tip

Summary: This PR adds two sections to the Insights policy-groups reference page — Default policy groups and Managing policy groups programmatically — plus a Runtime requirements section on the policy-packs index and a corrected default-group note on the discovery add-policies page. It documents that every organization gets two auto-created default groups, the risk of enabling a mandatory pack org-wide, the policy-group REST API (PATCH//batch fields), and the pulumiservice provider's PolicyGroupStackAttachment/PolicyGroupInsightsAccountAttachment resources. The kind of wrongness that would block a reader is an incorrect API or resource surface — wrong field names, a non-existent endpoint or resource, or a mis-described import behavior that leads a reader to disable policies. The review ran claim extraction + verification (27 claims; 19 verified, 7 unverifiable, 0 contradicted), a cross-sibling read of the API-reference and project-file pages, internal-anchor-link checks, a code-example pass, and style linting. All internal anchors resolve and the code example matches the provider schema. The two most consequential unverifiable claims — the two attachment resources and their inputs — were independently confirmed against the pulumiservice provider schema; the remaining unverifiable items are REST API specifics (the exact endpoint/field set, the 204 status, the /batch endpoint) that read as internally consistent but should be confirmed against the Policy Groups REST API reference. No blockers.

Review confidence:

Dimension Level Notes
mechanics HIGH
facts MEDIUM 19/27 verified, 0 contradicted; 7 REST-API/provider specifics left unverifiable (2 attachment resources since confirmed against the provider schema).
cross-sibling consistency MEDIUM Read api-reference.md + project-file.md; no contradictions. The API-reference sibling's provider table still lists only PolicyGroup, not the two attachment resources this PR now recommends — a pre-existing completeness gap worth closing in a follow-up.
code correctness HIGH TypeScript example's inputs match the PolicyGroupStackAttachment schema exactly.
Investigation log
  • Cross-sibling reads: 2 of 13 siblings (read api-reference.md and policy-packs/project-file.md — the two bearing on the new REST API and runtime content)
  • External claim verification: 19 of 27 claims verified (7 unverifiable, 0 contradicted) · 4 specialists (numerical, cross-reference, capability, framing); 0 cross-specialist corroborations · routed: 0 inline, 25 Pass 1, 0 Pass 2, 2 Pass 3 (verified 2, contradicted 0, unverifiable 0).
  • Cited-claim spot-checks: not run (no cited claims)
  • Frontmatter sweep: ran on body + meta_desc
  • Temporal-trigger sweep: ran (recency words present in diff; spot-check in-review)
  • Code execution: not run (no static/programs/ change)
  • Code-examples checks: ran (3 specialists: structural, existence, body-code-coverage); 0 findings
  • Editorial-balance pass: not run (not under content/blog/)
🚨 Outstanding ⚠️ Low-confidence 💡 Pre-existing ✅ Resolved
0 7 0 0

🔍 Verification trail

27 claims extracted · 19 verified · 7 unverifiable · 0 contradicted
  • L90 in content/docs/insights/discovery/get-started/add-policies.md "The page /docs/insights/policy/policy-groups/#default-policy-groups exists and contains details about default policy groups." → ✅ verified (evidence: content/docs/insights/policy/policy-groups.md contains a heading "## Default policy groups" at line 89, which generates the anchor #default-policy-groups, and the section (lines 90-96) details the two default policy groups (default-prevent…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L46 in content/docs/insights/policy/policy-groups.md "There is a documentation section titled 'Default policy groups' (linked via anchor #default-policy-groups) that describes how default policy groups behave and…" → ✅ verified (evidence: The same file contains a heading ## Default policy groups (line 89) which describes default group behavior (auto-creation, membership) and a subsection 'Adding policy packs to a default policy group' (line 98) describing considerations b…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L46 in content/docs/insights/policy/policy-groups.md "When you enable Pulumi Policies for your organization, default policy groups are created automatically: default-preventative-policy-group for stacks and def…" → ✅ verified (evidence: The same document states: "Every organization has two default policy groups, created automatically the first time you enable Pulumi Policies: **default-preventative-policy-group`** automatically includes every new stack in the organizatio…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L91 in content/docs/insights/policy/policy-groups.md "Every organization has exactly two default policy groups, created automatically the first time you enable Pulumi Policies." → ✅ verified (evidence: The Pulumi docs page states: "When you enable Pulumi Policies for your organization, default policy groups are created automatically: default-preventative-policy-group for stacks and default-audit-policy-group for stacks and cloud accounts…; source: https://www.pulumi.com/docs/insights/policy/policy-groups/)
  • L93-94 in content/docs/insights/policy/policy-groups.md "The default-audit-policy-group automatically includes every newly connected cloud account, alongside all stacks." → ✅ verified (evidence: The linked accounts page confirms newly connected accounts get policy evaluation enabled by default during setup: "review the defaults: scheduled scans and policy evaluation are both enabled, with a default policy pack pre-selected" — cons…; source: content/docs/insights/discovery/accounts.md)
  • L96 in content/docs/insights/policy/policy-groups.md "You can remove an individual stack or account from a default policy group the same way you would from any other group." → ✅ verified (evidence: The same doc states you can add/remove stacks, cloud accounts, and policy packs from a policy group via the REST API or pulumiservice provider "including the default policy groups described above," and details removeStack/`removeInsights…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L100 in content/docs/insights/policy/policy-groups.md "Enabling a policy pack with mandatory enforcement on the default preventative policy group starts blocking pulumi up and pulumi preview for every team im…" → ✅ verified (evidence: The same doc states preventative policy groups "act as guardrails during pulumi up and pulumi preview, blocking non-compliant deployments," that "Mandatory: Blocks deployments when violations are detected," and that "`default-preventat…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L102 in content/docs/insights/policy/policy-groups.md "A nodejs policy pack requires Node.js on the client." → ✅ verified (evidence: The sibling project-file reference confirms nodejs policy packs declare runtime: nodejs ("Must be nodejs") and policy-groups.md states policies "evaluate locally, as part of the pulumi CLI invocation," so a nodejs pack requires…; source: repo:content/docs/insights/policy/policy-packs/project-file.md)
  • L102 in content/docs/insights/policy/policy-groups.md "Because a mandatory pack on the default preventative group applies to every stack, a missing runtime shows up as unexpected pulumi up failures across the org…" → ✅ verified (evidence: The same doc establishes that "the default preventative policy group covers every stack in the organization" (L100) and that policy packs "evaluate locally, as part of the pulumi CLI invocation," requiring the declared runtime on the cli…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L102 in content/docs/insights/policy/policy-groups.md "Before enabling any policy pack, every environment where pulumi preview or pulumi up runs (developer machines, CI/CD workers) must have the runtime that th…" → ✅ verified (evidence: project-file.md confirms PulumiPolicy.yaml requires a runtime field (nodejs, python, or opa), e.g. "runtime | required | Must be nodejs." and "runtime | required | Must be python.", which supports the policy-groups.md cla…; source: repo:content/docs/insights/policy/policy-packs/project-file.md)
  • L106 in content/docs/insights/policy/policy-groups.md "The REST API and the pulumiservice provider can be used to manage default policy groups, including adding/removing stacks, cloud accounts, and policy packs." → ✅ verified (evidence: The same doc states: "In addition to the console, you can add and remove stacks, cloud accounts, and policy packs from a policy group using the Pulumi Cloud REST API or the pulumiservice Pulumi pro…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L110 in content/docs/insights/policy/policy-groups.md "A single PATCH request to a policy group's endpoint can add or remove a stack, a cloud account, or a policy pack, or rename the group, through one or more fi…" → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L112-114 in content/docs/insights/policy/policy-groups.md "The addPolicyPack and removePolicyPack API fields take the pack's name, displayName, version, and versionTag, and all four fields are required." → ✅ verified (evidence: (escalated from pass1) The Pulumi Go SDK apitype source shows type PolicyPackMetadata struct { Name string json:"name"DisplayName stringjson:"displayName"Version intjson:"version"VersionTag stringjson:"versionTag" ... } use…; source: https://pkg.go.dev/github.com/pulumi/pulumi/sdk/v3/go/common/apitype)
  • L116 in content/docs/insights/policy/policy-groups.md "A successful PATCH request to the policy group endpoint returns 204 No Content." → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L116 in content/docs/insights/policy/policy-groups.md "The add/remove API fields are additive: sending addStack doesn't touch the group's existing stacks or policy packs." → ➖ not-a-claim (evidence: This is the doc author's own description of the API design they're documenting, and it's internally consistent with the immediately following warning note: "The stacks, policyPacks, and insightsAccounts fields, as opposed to their `a…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L119 in content/docs/insights/policy/policy-groups.md "The stacks, policyPacks, and insightsAccounts fields, as opposed to their add/remove counterparts, replace the group's entire membership list wheneve…" → 🤷 unverifiable (evidence: The claim describes API field semantics for the Pulumi Cloud policy-groups PATCH endpoint (stacks/policyPacks/insightsAccounts as full-replace vs add/remove as additive). This is internally consistent with the same doc's adjacent…; source: repo:content/docs/insights/policy/policy-groups.md; intuition: Claim is plausible and internally consistent with sibling text on the same page (additive add/remove vs. presumably non…)
  • L119 in content/docs/insights/policy/policy-groups.md "Omitting a field in the request leaves it unchanged, while including it as an empty array clears it." → ✅ verified (evidence: The doc itself states in the warning box at line 119: "The stacks, policyPacks, and insightsAccounts fields, as opposed to their add/remove counterparts, replace the group's entire membership list whenever you include them in a…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L122 in content/docs/insights/policy/policy-groups.md "Sending multiple change requests to a policy group's /batch endpoint, as an array of request bodies, applies several changes to the group in one round trip." → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L126 in content/docs/insights/policy/policy-groups.md "The pulumiservice provider's PolicyGroupInsightsAccountAttachment resource manages a single cloud account's membership in a policy group declaratively, wit…" → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L128-137 in content/docs/insights/policy/policy-groups.md "The pulumiservice provider's PolicyGroupStackAttachment resource accepts orgName, policyGroup, name, and routingProject properties, as shown in the…" → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L139 in content/docs/insights/policy/policy-groups.md "Because each attachment resource manages one membership edge, multiple stacks or teams can attach themselves to the same policy group in parallel without one P…" → ✅ verified (evidence: The same doc states the attachment resources "each manage a single stack's or cloud account's membership in a policy group declaratively, without taking ownership of the group's other members," which directly supports the claim that separa…; source: repo:content/docs/insights/policy/policy-groups.md (L119-142))
  • L142 in content/docs/insights/policy/policy-groups.md "Because import only populates name, orgName, and entityType, the next pulumi up after importing a default policy group would apply an empty `policyPack…" → 🤷 unverifiable (evidence: verification did not converge within 8 turns)
  • L142 in content/docs/insights/policy/policy-groups.md "pulumi import adds the protect: true option by default." → ✅ verified (evidence: The auto-generated pulumi_import CLI docs (from pulumi/pulumi source) state: "The resources are protected from deletion by default." and the --protect flag doc reads "Allow resources to be imported with protection from deletion enabled (…; source: repo:content/docs/iac/cli/commands/pulumi_import.md)
  • L156 in content/docs/insights/policy/policy-groups.md "There is a documentation page titled 'Policy API reference' at /docs/insights/policy/api-reference/." → ✅ verified (evidence: The docs repo contains content/docs/insights/policy/api-reference.md (html_url: github.com/pulumi/docs/blob/master/content/docs/insights/policy/api-reference.md), which maps to the page at /docs/insights/policy/api-reference/, titled 'Poli…; source: gh api repos/pulumi/docs/contents/content/docs/insights/policy)
  • L48 in content/docs/insights/policy/policy-packs/_index.md "The runtime requirement matters most for policy packs enabled with mandatory enforcement on a preventative policy group." → 🤝 matches (evidence: Sibling page policy-groups.md confirms: preventative groups "block deployments" with "mandatory enforcement" while audit groups "Cannot block deployments (reporting only)"; and explicitly states "Because a mandatory pack on the default pre…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L48 in content/docs/insights/policy/policy-packs/_index.md "Pulumi documents 'adding policy packs to a default policy group' as a topic describing the risk of enabling a policy pack organization-wide." → ✅ verified (evidence: The linked page content/docs/insights/policy/policy-groups.md has a section "### Adding policy packs to a default policy group" that states: "Because the default preventative policy group covers every stack in the organization, enabling a…; source: repo:content/docs/insights/policy/policy-groups.md)
  • L48 in content/docs/insights/policy/policy-packs/_index.md "The runtime requirement for a policy pack applies to every developer machine and CI/CD worker that runs pulumi preview or pulumi up against a stack the pac…" → ✅ verified (evidence: The doc itself states: "that runtime must be installed wherever pulumi preview or pulumi up runs against a stack the pack applies to, including every developer machine and CI/CD worker, not just where you authored or published the pack…; source: repo:content/docs/insights/policy/policy-packs/_index.md)

🚨 Outstanding in this PR

No outstanding findings in this PR.

⚠️ Low-confidence

Review each and resolve as appropriate — these don't block the PR.

  • [L110] content/docs/insights/policy/policy-groups.md"A single PATCH request to a policy group's endpoint can add or remove a stack, a cloud account, or a policy pack, or rename the group, through one or more fi…" — verdict: unverifiable; evidence: verification did not converge within 8 turns. Author check: confirm against the Policy Groups REST API reference that one PATCH supports all of these operations, including rename via a field. Plausible and internally consistent with the field list that follows; not a blocker.

  • [L116] content/docs/insights/policy/policy-groups.md"A successful PATCH request to the policy group endpoint returns 204 No Content." — verdict: unverifiable; evidence: verification did not converge within 8 turns. Author check: confirm the success status is 204 No Content (not 200) against the Policy Groups REST API reference. Not a blocker.

  • [L119] content/docs/insights/policy/policy-groups.md"The stacks, policyPacks, and insightsAccounts fields, as opposed to their add/remove counterparts, replace the group's entire membership list wheneve…" — verdict: unverifiable; evidence: The claim describes API field semantics for the Pulumi Cloud policy-groups PATCH endpoint (stacks/policyPacks/insightsAccounts as full-replace vs add/remove as additive), internally consistent with the doc's adjacent warning note; source: repo:content/docs/insights/policy/policy-groups.md. Author check: confirm the full-replace-vs-additive semantics against the Policy Groups REST API reference. Not a blocker.

  • [L122] content/docs/insights/policy/policy-groups.md"Sending multiple change requests to a policy group's /batch endpoint, as an array of request bodies, applies several changes to the group in one round trip." — verdict: unverifiable; evidence: verification did not converge within 8 turns. Author check: confirm the /batch endpoint exists and accepts an array of the same request bodies, against the Policy Groups REST API reference. Not a blocker.

  • [L126] content/docs/insights/policy/policy-groups.md"The pulumiservice provider's PolicyGroupInsightsAccountAttachment resource manages a single cloud account's membership in a policy group declaratively, wit…" — verdict: unverifiable; evidence: verification did not converge within 8 turns. Confirmed against the pulumiservice provider schema — a PolicyGroupInsightsAccountAttachment resource exists with orgName, policyGroup, and name inputs, matching this description. Left in low-confidence only because it wasn't machine-verified during the automated pass.

  • [L128-137] content/docs/insights/policy/policy-groups.md"The pulumiservice provider's PolicyGroupStackAttachment resource accepts orgName, policyGroup, name, and routingProject properties, as shown in the…" — verdict: unverifiable; evidence: verification did not converge within 8 turns. Confirmed against the pulumiservice provider schemaPolicyGroupStackAttachment accepts exactly orgName, policyGroup, name, and routingProject (all four required), matching the TypeScript example. Left in low-confidence only because it wasn't machine-verified during the automated pass.

  • [L142] content/docs/insights/policy/policy-groups.md — *"Because import only populates name, orgName, and entityType, the next pulumi up after importing a default policy group would apply an empty policyPack…"* — verdict: unverifiable; evidence: verification did not converge within 8 turns. **Author check:** this is a consequential warning (an incorrect version could lead a reader to disable every pack on a group). Confirm that importing a PolicyGroupreally populates onlyname/orgName/entityTypeand that a subsequentpulumi upwould clearpolicyPacks, against current pulumiserviceprovider behavior. Not a merge blocker, but worth an explicit sign-off. (The related claim thatpulumi importaddsprotect: true` by default is separately ✅ verified.)

💡 Pre-existing issues in touched files (optional)

No pre-existing issues in touched files.

✅ Resolved since last review

No items resolved since the last review.

📜 Review history

  • 2026-07-29T14:31:46Z — New default-policy-groups + programmatic-management docs; no blockers. 19/27 claims verified, 0 contradicted; all internal anchors resolve and the code example matches the provider schema. 7 REST-API/provider specifics left low-confidence (the 2 attachment resources confirmed against the pulumiservice schema). (81a91f6)
  • 2026-07-29T15:29:39Z — Re-reviewed after fix push (2 new commits, 5364c52). @workprentice[bot] reported the pre-merge review's 4 style findings (wordiness "In addition" → "Beyond the console"; weasel word "several" → "multiple"; two difficulty-qualifier instances of "just") fixed. Diff confirmed: all 4 wording changes present, each is a substance-preserving rewording of an already-verified or already-unverifiable claim, so no re-verification or new claim extraction was needed. Fresh Vale pass against HEAD (5364c52) returns 0 findings. Outstanding remains 0; the 7 low-confidence items are unchanged (same claims, same evidence).

Need a re-review? Want to dispute a finding? Mention @claude and include #update-review.
(For ad-hoc questions or fixes, just @claude — no hashtag.)

@github-actions github-actions Bot added review:no-blockers Claude review completed cleanly; outstanding is empty and removed review:in-progress Claude review is currently running labels Jul 29, 2026
@pulumi-bot

pulumi-bot commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

@github-actions github-actions Bot added review:stale New commits since last Claude review; refresh on next ready-transition or @claude mention and removed review:no-blockers Claude review completed cleanly; outstanding is empty labels Jul 29, 2026
@joeduffy

Copy link
Copy Markdown
Member

@workprentice Address the feedback.

@workprentice

workprentice Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

@joeduffy Addressed. The pre-merge review's 4 style findings (wordiness, weasel word, two difficulty-qualifier instances of "just") are fixed across the last two commits. @claude #update-review

@github-actions github-actions Bot added review:in-progress Claude review is currently running and removed review:stale New commits since last Claude review; refresh on next ready-transition or @claude mention labels Jul 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Review updated on @workprentice[bot]'s request.

@github-actions github-actions Bot added review:no-blockers Claude review completed cleanly; outstanding is empty and removed review:in-progress Claude review is currently running labels Jul 29, 2026
@CamSoper

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

Copilot AI commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

Merge conflicts resolved in commit b8cf2dc (see diff):

  • policy-groups.md: Kept master's canonical group names (default-policy-group / default-accounts-policy-group) and structure, which already incorporated the programmatic-management content from this PR. Removed the duplicate "Default policy groups" section that this branch had placed under "Best practices."
  • add-policies.md: Took master's corrected wording for the default group names in the info callout.

@CamSoper

Copy link
Copy Markdown
Contributor

Closing this as superseded by #20563, which landed first and covered #20554 comprehensively.

For the record, since the diff here no longer reflects what this PR was: resolving the conflicts against #20563 reduced it from +74/-0 across three files to +5 lines across two, and both remaining pieces are now problems rather than improvements.

  • The ## Runtime requirements section added to policy-packs/_index.md duplicates a heading master already has — and master's version is fuller, with a per-runtime table, the Bun caveat, and the OPA plugin note.
  • The _index.md paragraph links to #adding-policy-packs-to-a-default-policy-group and #preventative-policy-groups, neither of which exists on the policy-groups page. Master's headings are "Adding policy packs to the default preventative group" and "Types of policy groups".

No fault of the author's — two PRs were aimed at the same issue and this one lost the race.

The one thing worth keeping was the pulumi import warning, which master doesn't cover. Filed as #20575, including a note that the behavior needs confirming against the provider before anyone documents it.

@CamSoper CamSoper closed this Jul 30, 2026
CamSoper pushed a commit to CamSoper/pulumi.docs that referenced this pull request Jul 30, 2026
…p escalation, SHA-based review freshness

Gap 1 — no verdict for "value correct, meaning drifted" (PR pulumi#20550: three
stacked framing distortions on an accurate 66% CNCF figure rendered as
verified, with the drift observation buried in evidence prose):
- New `framing-drift` (🌀) verdict across verify-claims.py, compose-review.py,
  validate-pinned.py (schema v18→v19), splicer.py, validator-fix.py,
  record-claims.py (kept), reverify-claims.py (stale), compose-pr-body.py.
- Structured `framing` enum on verify_claim (exact-match / entailed-narrower /
  overclaim-broader / shifted / none); _finalize_verdict deterministically
  coerces a verified verdict carrying a drift shape to framing-drift, so the
  observation can no longer soft-pedal into verified prose.
- Framing taxonomy recast in entailment terms ("does the source, as quoted,
  prove the claim as written?") and reconciled across claim-extraction.md,
  fact-check.md (whose fallback section previously mapped the same labels to
  opposite verdicts), and the verifier prompt.
- Bucket: ⚠️ Low-confidence by default; reviewer promotes to 🚨 when the
  drifted phrasing rides social.* frontmatter (auto-posted on merge).
- New source-discipline rule: the content under review never verifies its own
  technical claims (the AES256-GCM circular-verification case on PR pulumi#20371).

Gap 2 — pass1 turn-cap and routing failures reported as unverifiable
(PR pulumi#20556: six REST-API claims died at the 8-turn cap; PR pulumi#20371: a pure
CLI-behavior claim routed to web search, which cannot read product source):
- pass1 turn cap 8→12; cap exhaustion now auto-escalates once to pass3
  instead of returning bare unverifiable.
- Symmetric route escalation: pass3 verifiers may emit
  route_escalation: "pass1" for Pulumi-product-behavior claims that carry no
  pulumi-shaped token (one hop, ping-pong guarded).
- Terminal turn-cap records carry `turn_cap_exhausted: true` and retryable
  phrasing; compose-review stubs them with an explicit "never call this out
  of scope" TODO so budget failures and architectural limits can no longer
  render identically downstream.

Gap 3 — stale pinned review kept review:no-blockers (PR pulumi#20556: a Copilot
conflict-resolution push fired no pull_request:synchronize event, so
mark-stale never ran; App pushes like workprentice[bot] do fire it — the
suppressed class is Copilot-agent and GITHUB_TOKEN pushes):
- compose-review.py stamps a machine-readable
  `<!-- CLAUDE_REVIEW_HEAD <sha> -->` sentinel under the review header
  (the workflow already passes --head-sha); update.md requires the
  re-entrant path to refresh it.
- pr-review SKILL Step 2 now runs a SHA freshness check before trusting
  CURRENT labels, with a review-history fallback for pre-sentinel reviews.
- New scheduled review-label-reconcile.yml workflow compares each labeled
  open PR's reviewed SHA against its head and applies the same stale
  transition mark-stale would have applied — the polling-side net for
  pushes that suppress webhook events.

Tests: new test_framing_drift.py (36 checks: constant sync, coercion,
escalation hops, cap marker, stub placement, validator acceptance/violation
directions, header sentinel); all existing docs-review and content-review
suites pass; make lint clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KcNQVWHUyvbbe5Go9MGxwA
CamSoper pushed a commit that referenced this pull request Jul 30, 2026
…p escalation, SHA-based review freshness

Gap 1 — no verdict for "value correct, meaning drifted" (PR #20550: three
stacked framing distortions on an accurate 66% CNCF figure rendered as
verified, with the drift observation buried in evidence prose):
- New `framing-drift` (🌀) verdict across verify-claims.py, compose-review.py,
  validate-pinned.py (schema v18→v19), splicer.py, validator-fix.py,
  record-claims.py (kept), reverify-claims.py (stale), compose-pr-body.py.
- Structured `framing` enum on verify_claim (exact-match / entailed-narrower /
  overclaim-broader / shifted / none); _finalize_verdict deterministically
  coerces a verified verdict carrying a drift shape to framing-drift, so the
  observation can no longer soft-pedal into verified prose.
- Framing taxonomy recast in entailment terms ("does the source, as quoted,
  prove the claim as written?") and reconciled across claim-extraction.md,
  fact-check.md (whose fallback section previously mapped the same labels to
  opposite verdicts), and the verifier prompt.
- Bucket: ⚠️ Low-confidence by default; reviewer promotes to 🚨 when the
  drifted phrasing rides social.* frontmatter (auto-posted on merge).
- New source-discipline rule: the content under review never verifies its own
  technical claims (the AES256-GCM circular-verification case on PR #20371).

Gap 2 — pass1 turn-cap and routing failures reported as unverifiable
(PR #20556: six REST-API claims died at the 8-turn cap; PR #20371: a pure
CLI-behavior claim routed to web search, which cannot read product source):
- pass1 turn cap 8→12; cap exhaustion now auto-escalates once to pass3
  instead of returning bare unverifiable.
- Symmetric route escalation: pass3 verifiers may emit
  route_escalation: "pass1" for Pulumi-product-behavior claims that carry no
  pulumi-shaped token (one hop, ping-pong guarded).
- Terminal turn-cap records carry `turn_cap_exhausted: true` and retryable
  phrasing; compose-review stubs them with an explicit "never call this out
  of scope" TODO so budget failures and architectural limits can no longer
  render identically downstream.

Gap 3 — stale pinned review kept review:no-blockers (PR #20556: a Copilot
conflict-resolution push fired no pull_request:synchronize event, so
mark-stale never ran; App pushes like workprentice[bot] do fire it — the
suppressed class is Copilot-agent and GITHUB_TOKEN pushes):
- compose-review.py stamps a machine-readable
  `<!-- CLAUDE_REVIEW_HEAD <sha> -->` sentinel under the review header
  (the workflow already passes --head-sha); update.md requires the
  re-entrant path to refresh it.
- pr-review SKILL Step 2 now runs a SHA freshness check before trusting
  CURRENT labels, with a review-history fallback for pre-sentinel reviews.
- New scheduled review-label-reconcile.yml workflow compares each labeled
  open PR's reviewed SHA against its head and applies the same stale
  transition mark-stale would have applied — the polling-side net for
  pushes that suppress webhook events.

Tests: new test_framing_drift.py (36 checks: constant sync, coercion,
escalation hops, cap marker, stub placement, validator acceptance/violation
directions, header sentinel); all existing docs-review and content-review
suites pass; make lint clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KcNQVWHUyvbbe5Go9MGxwA
CamSoper added a commit that referenced this pull request Jul 30, 2026
…p escalation, SHA-based review freshness (#20577)

* Close three docs-review pipeline gaps: framing-drift verdict, turn-cap escalation, SHA-based review freshness

Gap 1 — no verdict for "value correct, meaning drifted" (PR #20550: three
stacked framing distortions on an accurate 66% CNCF figure rendered as
verified, with the drift observation buried in evidence prose):
- New `framing-drift` (🌀) verdict across verify-claims.py, compose-review.py,
  validate-pinned.py (schema v18→v19), splicer.py, validator-fix.py,
  record-claims.py (kept), reverify-claims.py (stale), compose-pr-body.py.
- Structured `framing` enum on verify_claim (exact-match / entailed-narrower /
  overclaim-broader / shifted / none); _finalize_verdict deterministically
  coerces a verified verdict carrying a drift shape to framing-drift, so the
  observation can no longer soft-pedal into verified prose.
- Framing taxonomy recast in entailment terms ("does the source, as quoted,
  prove the claim as written?") and reconciled across claim-extraction.md,
  fact-check.md (whose fallback section previously mapped the same labels to
  opposite verdicts), and the verifier prompt.
- Bucket: ⚠️ Low-confidence by default; reviewer promotes to 🚨 when the
  drifted phrasing rides social.* frontmatter (auto-posted on merge).
- New source-discipline rule: the content under review never verifies its own
  technical claims (the AES256-GCM circular-verification case on PR #20371).

Gap 2 — pass1 turn-cap and routing failures reported as unverifiable
(PR #20556: six REST-API claims died at the 8-turn cap; PR #20371: a pure
CLI-behavior claim routed to web search, which cannot read product source):
- pass1 turn cap 8→12; cap exhaustion now auto-escalates once to pass3
  instead of returning bare unverifiable.
- Symmetric route escalation: pass3 verifiers may emit
  route_escalation: "pass1" for Pulumi-product-behavior claims that carry no
  pulumi-shaped token (one hop, ping-pong guarded).
- Terminal turn-cap records carry `turn_cap_exhausted: true` and retryable
  phrasing; compose-review stubs them with an explicit "never call this out
  of scope" TODO so budget failures and architectural limits can no longer
  render identically downstream.

Gap 3 — stale pinned review kept review:no-blockers (PR #20556: a Copilot
conflict-resolution push fired no pull_request:synchronize event, so
mark-stale never ran; App pushes like workprentice[bot] do fire it — the
suppressed class is Copilot-agent and GITHUB_TOKEN pushes):
- compose-review.py stamps a machine-readable
  `<!-- CLAUDE_REVIEW_HEAD <sha> -->` sentinel under the review header
  (the workflow already passes --head-sha); update.md requires the
  re-entrant path to refresh it.
- pr-review SKILL Step 2 now runs a SHA freshness check before trusting
  CURRENT labels, with a review-history fallback for pre-sentinel reviews.
- New scheduled review-label-reconcile.yml workflow compares each labeled
  open PR's reviewed SHA against its head and applies the same stale
  transition mark-stale would have applied — the polling-side net for
  pushes that suppress webhook events.

Tests: new test_framing_drift.py (36 checks: constant sync, coercion,
escalation hops, cap marker, stub placement, validator acceptance/violation
directions, header sentinel); all existing docs-review and content-review
suites pass; make lint clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KcNQVWHUyvbbe5Go9MGxwA

* Count framing-drift separately from contradicted in the headline tallies

The first cut folded framing-drift into CONTRADICTION_FAMILY, which feeds the
🔍 trail <details> header and the investigation log's parenthetical — both of
which label that number `contradicted`. But framing-drift defaults to ⚠️, not
🚨, so the headline reported contradictions the body didn't contain: a fork run
produced `2 verified · 0 unverifiable · 2 contradicted` above a trail with zero
❌ lines, and another read `4 contradicted` for 1 real ❌ plus 3 🌀.

CONTRADICTION_FAMILY now means what its name says (contradicted + mismatch) and
drift gets its own `· W framing-drift` segment in both places, omitted entirely
when the count is zero so ordinary reviews are byte-identical. The per-lane
Pass 2/3 V/C/U triple keeps drift in the C column — that triple's shape is
pinned by validate-pinned.py's PASS2_OUTCOME_RE/PASS3_OUTCOME_RE, and it's a
routing diagnostic rather than a reader-facing count.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain:docs PR touches technical docs review:no-blockers Claude review completed cleanly; outstanding is empty

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document the default policy groups

5 participants