Document default policy groups and how to manage them programmatically - #20556
Document default policy groups and how to manage them programmatically#20556workprentice[bot] wants to merge 4 commits into
Conversation
Fixes #20554 - Adds a 'Default policy groups' section to policy-groups.md covering the two auto-created default groups (default-preventative-policy-group, default-audit-policy-group), their membership semantics, and the caution around adding policy packs to them (org-wide blocking with mandatory enforcement, client runtime requirements). - Adds a 'Managing policy groups programmatically' section covering the REST API's additive add*/remove* fields vs. the full-list-replace stacks/policyPacks/insightsAccounts fields, the /batch endpoint, and the pulumiservice provider's PolicyGroupStackAttachment / PolicyGroupInsightsAccountAttachment resources, with a warning against importing a default PolicyGroup resource. - Adds a short runtime-requirements note to policy-packs/_index.md per the issue's suggestion that it belongs on the policy pack concepts page. - Fixes a factual error in discovery/get-started/add-policies.md that referenced a generic 'default-policy-group' name inconsistent with the two real default group names published elsewhere in the docs.
Pre-merge Review — Last updated 2026-07-29T15:29:39ZTip Summary: This PR adds two sections to the Insights policy-groups reference page — Default policy groups and Managing policy groups programmatically — plus a Runtime requirements section on the policy-packs index and a corrected default-group note on the discovery add-policies page. It documents that every organization gets two auto-created default groups, the risk of enabling a Review confidence:
Investigation log
🔍 Verification trail27 claims extracted · 19 verified · 7 unverifiable · 0 contradicted
🚨 Outstanding in this PRNo outstanding findings in this PR.
|
|
Your site preview for commit 5364c52 is ready! 🎉 http://www-testing-pulumi-docs-origin-pr-20556-5364c52d.s3-website.us-west-2.amazonaws.com Changed pages: |
|
@workprentice Address the feedback. |
|
🤖 Review updated on @workprentice[bot]'s request. |
|
@copilot resolve the merge conflicts in this pull request |
Merge conflicts resolved in commit
|
|
Closing this as superseded by #20563, which landed first and covered #20554 comprehensively. For the record, since the diff here no longer reflects what this PR was: resolving the conflicts against #20563 reduced it from +74/-0 across three files to +5 lines across two, and both remaining pieces are now problems rather than improvements.
No fault of the author's — two PRs were aimed at the same issue and this one lost the race. The one thing worth keeping was the |
…p escalation, SHA-based review freshness Gap 1 — no verdict for "value correct, meaning drifted" (PR pulumi#20550: three stacked framing distortions on an accurate 66% CNCF figure rendered as verified, with the drift observation buried in evidence prose): - New `framing-drift` (🌀) verdict across verify-claims.py, compose-review.py, validate-pinned.py (schema v18→v19), splicer.py, validator-fix.py, record-claims.py (kept), reverify-claims.py (stale), compose-pr-body.py. - Structured `framing` enum on verify_claim (exact-match / entailed-narrower / overclaim-broader / shifted / none); _finalize_verdict deterministically coerces a verified verdict carrying a drift shape to framing-drift, so the observation can no longer soft-pedal into verified prose. - Framing taxonomy recast in entailment terms ("does the source, as quoted, prove the claim as written?") and reconciled across claim-extraction.md, fact-check.md (whose fallback section previously mapped the same labels to opposite verdicts), and the verifier prompt. - Bucket:⚠️ Low-confidence by default; reviewer promotes to 🚨 when the drifted phrasing rides social.* frontmatter (auto-posted on merge). - New source-discipline rule: the content under review never verifies its own technical claims (the AES256-GCM circular-verification case on PR pulumi#20371). Gap 2 — pass1 turn-cap and routing failures reported as unverifiable (PR pulumi#20556: six REST-API claims died at the 8-turn cap; PR pulumi#20371: a pure CLI-behavior claim routed to web search, which cannot read product source): - pass1 turn cap 8→12; cap exhaustion now auto-escalates once to pass3 instead of returning bare unverifiable. - Symmetric route escalation: pass3 verifiers may emit route_escalation: "pass1" for Pulumi-product-behavior claims that carry no pulumi-shaped token (one hop, ping-pong guarded). - Terminal turn-cap records carry `turn_cap_exhausted: true` and retryable phrasing; compose-review stubs them with an explicit "never call this out of scope" TODO so budget failures and architectural limits can no longer render identically downstream. Gap 3 — stale pinned review kept review:no-blockers (PR pulumi#20556: a Copilot conflict-resolution push fired no pull_request:synchronize event, so mark-stale never ran; App pushes like workprentice[bot] do fire it — the suppressed class is Copilot-agent and GITHUB_TOKEN pushes): - compose-review.py stamps a machine-readable `<!-- CLAUDE_REVIEW_HEAD <sha> -->` sentinel under the review header (the workflow already passes --head-sha); update.md requires the re-entrant path to refresh it. - pr-review SKILL Step 2 now runs a SHA freshness check before trusting CURRENT labels, with a review-history fallback for pre-sentinel reviews. - New scheduled review-label-reconcile.yml workflow compares each labeled open PR's reviewed SHA against its head and applies the same stale transition mark-stale would have applied — the polling-side net for pushes that suppress webhook events. Tests: new test_framing_drift.py (36 checks: constant sync, coercion, escalation hops, cap marker, stub placement, validator acceptance/violation directions, header sentinel); all existing docs-review and content-review suites pass; make lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcNQVWHUyvbbe5Go9MGxwA
…p escalation, SHA-based review freshness Gap 1 — no verdict for "value correct, meaning drifted" (PR #20550: three stacked framing distortions on an accurate 66% CNCF figure rendered as verified, with the drift observation buried in evidence prose): - New `framing-drift` (🌀) verdict across verify-claims.py, compose-review.py, validate-pinned.py (schema v18→v19), splicer.py, validator-fix.py, record-claims.py (kept), reverify-claims.py (stale), compose-pr-body.py. - Structured `framing` enum on verify_claim (exact-match / entailed-narrower / overclaim-broader / shifted / none); _finalize_verdict deterministically coerces a verified verdict carrying a drift shape to framing-drift, so the observation can no longer soft-pedal into verified prose. - Framing taxonomy recast in entailment terms ("does the source, as quoted, prove the claim as written?") and reconciled across claim-extraction.md, fact-check.md (whose fallback section previously mapped the same labels to opposite verdicts), and the verifier prompt. - Bucket:⚠️ Low-confidence by default; reviewer promotes to 🚨 when the drifted phrasing rides social.* frontmatter (auto-posted on merge). - New source-discipline rule: the content under review never verifies its own technical claims (the AES256-GCM circular-verification case on PR #20371). Gap 2 — pass1 turn-cap and routing failures reported as unverifiable (PR #20556: six REST-API claims died at the 8-turn cap; PR #20371: a pure CLI-behavior claim routed to web search, which cannot read product source): - pass1 turn cap 8→12; cap exhaustion now auto-escalates once to pass3 instead of returning bare unverifiable. - Symmetric route escalation: pass3 verifiers may emit route_escalation: "pass1" for Pulumi-product-behavior claims that carry no pulumi-shaped token (one hop, ping-pong guarded). - Terminal turn-cap records carry `turn_cap_exhausted: true` and retryable phrasing; compose-review stubs them with an explicit "never call this out of scope" TODO so budget failures and architectural limits can no longer render identically downstream. Gap 3 — stale pinned review kept review:no-blockers (PR #20556: a Copilot conflict-resolution push fired no pull_request:synchronize event, so mark-stale never ran; App pushes like workprentice[bot] do fire it — the suppressed class is Copilot-agent and GITHUB_TOKEN pushes): - compose-review.py stamps a machine-readable `<!-- CLAUDE_REVIEW_HEAD <sha> -->` sentinel under the review header (the workflow already passes --head-sha); update.md requires the re-entrant path to refresh it. - pr-review SKILL Step 2 now runs a SHA freshness check before trusting CURRENT labels, with a review-history fallback for pre-sentinel reviews. - New scheduled review-label-reconcile.yml workflow compares each labeled open PR's reviewed SHA against its head and applies the same stale transition mark-stale would have applied — the polling-side net for pushes that suppress webhook events. Tests: new test_framing_drift.py (36 checks: constant sync, coercion, escalation hops, cap marker, stub placement, validator acceptance/violation directions, header sentinel); all existing docs-review and content-review suites pass; make lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcNQVWHUyvbbe5Go9MGxwA
…p escalation, SHA-based review freshness (#20577) * Close three docs-review pipeline gaps: framing-drift verdict, turn-cap escalation, SHA-based review freshness Gap 1 — no verdict for "value correct, meaning drifted" (PR #20550: three stacked framing distortions on an accurate 66% CNCF figure rendered as verified, with the drift observation buried in evidence prose): - New `framing-drift` (🌀) verdict across verify-claims.py, compose-review.py, validate-pinned.py (schema v18→v19), splicer.py, validator-fix.py, record-claims.py (kept), reverify-claims.py (stale), compose-pr-body.py. - Structured `framing` enum on verify_claim (exact-match / entailed-narrower / overclaim-broader / shifted / none); _finalize_verdict deterministically coerces a verified verdict carrying a drift shape to framing-drift, so the observation can no longer soft-pedal into verified prose. - Framing taxonomy recast in entailment terms ("does the source, as quoted, prove the claim as written?") and reconciled across claim-extraction.md, fact-check.md (whose fallback section previously mapped the same labels to opposite verdicts), and the verifier prompt. - Bucket:⚠️ Low-confidence by default; reviewer promotes to 🚨 when the drifted phrasing rides social.* frontmatter (auto-posted on merge). - New source-discipline rule: the content under review never verifies its own technical claims (the AES256-GCM circular-verification case on PR #20371). Gap 2 — pass1 turn-cap and routing failures reported as unverifiable (PR #20556: six REST-API claims died at the 8-turn cap; PR #20371: a pure CLI-behavior claim routed to web search, which cannot read product source): - pass1 turn cap 8→12; cap exhaustion now auto-escalates once to pass3 instead of returning bare unverifiable. - Symmetric route escalation: pass3 verifiers may emit route_escalation: "pass1" for Pulumi-product-behavior claims that carry no pulumi-shaped token (one hop, ping-pong guarded). - Terminal turn-cap records carry `turn_cap_exhausted: true` and retryable phrasing; compose-review stubs them with an explicit "never call this out of scope" TODO so budget failures and architectural limits can no longer render identically downstream. Gap 3 — stale pinned review kept review:no-blockers (PR #20556: a Copilot conflict-resolution push fired no pull_request:synchronize event, so mark-stale never ran; App pushes like workprentice[bot] do fire it — the suppressed class is Copilot-agent and GITHUB_TOKEN pushes): - compose-review.py stamps a machine-readable `<!-- CLAUDE_REVIEW_HEAD <sha> -->` sentinel under the review header (the workflow already passes --head-sha); update.md requires the re-entrant path to refresh it. - pr-review SKILL Step 2 now runs a SHA freshness check before trusting CURRENT labels, with a review-history fallback for pre-sentinel reviews. - New scheduled review-label-reconcile.yml workflow compares each labeled open PR's reviewed SHA against its head and applies the same stale transition mark-stale would have applied — the polling-side net for pushes that suppress webhook events. Tests: new test_framing_drift.py (36 checks: constant sync, coercion, escalation hops, cap marker, stub placement, validator acceptance/violation directions, header sentinel); all existing docs-review and content-review suites pass; make lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KcNQVWHUyvbbe5Go9MGxwA * Count framing-drift separately from contradicted in the headline tallies The first cut folded framing-drift into CONTRADICTION_FAMILY, which feeds the 🔍 trail <details> header and the investigation log's parenthetical — both of which label that number `contradicted`. But framing-drift defaults to⚠️ , not 🚨, so the headline reported contradictions the body didn't contain: a fork run produced `2 verified · 0 unverifiable · 2 contradicted` above a trail with zero ❌ lines, and another read `4 contradicted` for 1 real ❌ plus 3 🌀. CONTRADICTION_FAMILY now means what its name says (contradicted + mismatch) and drift gets its own `· W framing-drift` segment in both places, omitted entirely when the count is zero so ordinary reviews are byte-identical. The per-lane Pass 2/3 V/C/U triple keeps drift in the C column — that triple's shape is pinned by validate-pinned.py's PASS2_OUTCOME_RE/PASS3_OUTCOME_RE, and it's a routing diagnostic rather than a reader-facing count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
Fixes #20554
What changed
content/docs/insights/policy/policy-groups.md— added two new sections:default-preventative-policy-group,default-audit-policy-group), what auto-membership means for each, that individual members can be removed but the groups themselves can't be deleted, and a caution about adding policy packs to a default group (org-widemandatoryblocking with no staged rollout, plus the client runtime requirement).PATCHendpoint (additiveadd*/remove*fields vs. the full-list-replacestacks/policyPacks/insightsAccountsfields, the/batchendpoint) and thepulumiserviceprovider'sPolicyGroupStackAttachment/PolicyGroupInsightsAccountAttachmentresources, with a warning againstpulumi import-ing a defaultPolicyGroupresource.content/docs/insights/policy/policy-packs/_index.md— added a short "Runtime requirements" section, per the issue's suggestion that this belongs on the policy pack concepts page, cross-linking the project file reference and the new default-group caution.content/docs/insights/discovery/get-started/add-policies.md— fixed a factual error: an info callout referenced a genericdefault-policy-groupname that doesn't match the two real default group names published onpolicy-groups.md. Replaced with the correct names and a link.Verification sources
https://api.pulumi.com/api/openapi/pulumi-spec.json): confirmed thePATCH /api/orgs/{orgName}/policygroups/{policyGroup}request/response shape (UpdatePolicyGroupRequest, 204 No Content), the required fields onaddStack/removeStack(name,routingProject) andaddPolicyPack/removePolicyPack(name,displayName,version,versionTag— all four required), the full-list-replace semantics of the plainstacks/policyPacks/insightsAccountsfields, the/batchendpoint, andDeletePolicyGroup's 400 response for an organization's default group.pulumi-pulumiserviceprovider schema, verified at the releasedv1.3.0tag (notmaster, whoseschema.jsonreflects unreleased breaking changes toPolicyGroup.policyPacks): confirmed the exact resource tokenspulumiservice:api:PolicyGroupStackAttachmentandpulumiservice:api:PolicyGroupInsightsAccountAttachment, their required inputs, and that noPolicyGroupPolicyPackAttachmentresource exists.Judgment calls for the reviewer
PulumiPolicy.yaml(nodejs,python, oropa) must be present on every client. Please confirm this framing is acceptable, or supply the more specific requirement if one exists.default-policy-groupas a group name, even though it appears in the issue's own curl examples. It contradicts the two specific default-group names already published onpolicy-groups.mdand I couldn't verify it independently, so I treated it as informal/generic phrasing rather than a real identifier.disabledlevel (per the OpenAPI spec) that isn't currently documented onpolicy-groups.md, which only covers Advisory/Mandatory. Out of scope for this PR; flagging as a possible follow-up.PolicyGroupStackAttachment,PolicyGroupInsightsAccountAttachment) inapi-reference.md. Their registry doc pages currently 404 even though the resources exist in the releasedv1.3.0package — likely a lag in registry doc generation for this provider release. I mentioned the resources by name (with a code sample) inpolicy-groups.mdwithout linking to a page that doesn't resolve yet, and leftapi-reference.mduntouched. Happy to add the registry link once those pages exist.add-policies.mdat a line immediately adjacent to the note I corrected. The conflict should be a trivial reconciliation of the terminology wording, whichever PR merges second.🧠 This PR was created by workprentice on behalf of the Docs Groundskeeper automation — no specific human requester triggered this run; it was generated during a scheduled docs-backlog burndown cycle.