Close three docs-review pipeline gaps: framing-drift verdict, turn-cap escalation, SHA-based review freshness

[CamSoper]

Proposed changes

Three gaps in the docs-review pipeline produced clean pinned reviews on PRs with real problems (adjudicated 2026-07-29), making them invisible to the outcome metrics. Each gap's root cause was confirmed against live evidence before changing anything.

Gap 1 — new framing-drift (🌀) verdict: "value correct, meaning drifted"

On PR #20550, the verifier confirmed the 66% CNCF figure, wrote the framing distortion into evidence prose, and returned verified — because the vocabulary offered only verified (nothing wrong) or contradicted (feels false when the number is right), and a framing_note without a matching verdict has no bucket effect.

• New verdict framing-drift across the pipeline: verify-claims.py, compose-review.py, validate-pinned.py (schema v18→v19, following the flagged v15→v16 precedent), splicer.py, validator-fix.py, record-claims.py, reverify-claims.py, compose-pr-body.py, plus output-format.md / fact-check.md / claim-extraction.md / ci.md and the workflow prompt.
• Structured framing enum on the verify_claim tool (exact-match | entailed-narrower | overclaim-broader | shifted | none). The harness, not the model, places the observation: _finalize_verdict() coerces verified + drift-shaped framing to framing-drift, closing the soft-pedal path mechanically.
• The framing taxonomy is recast in entailment terms ("does the source, as quoted, prove the claim as written?") and reconciled across the two reference docs, which previously mapped the same labels to opposite verdicts (fact-check.md §Mandatory evidence-line format said strengthened → contradicted; claim-extraction.md and the verifier prompt said strengthened → verified).
• Bucketing: ⚠️ Low-confidence by default; promoted to 🚨 when the drifted phrasing also rides social.* frontmatter (auto-posts on merge — exactly Add "How to Run AI Agents on Kubernetes with Pulumi" blog post #20550's exposure). The validator accepts ⚠️ for framing-drift the same way it does for flagged.
• Counting: framing-drift is not part of the Z contradicted headline tally (it defaults to ⚠️, so counting it there made the trail header and the investigation log report contradictions the body didn't contain — caught on a fork battery, where a header read 2 contradicted above a trail with zero ❌ lines). It gets its own · W framing-drift segment in both places, omitted when the count is zero so ordinary reviews render byte-identically. The per-lane Pass 2/3 V/C/U triple keeps drift in the C column; that triple's shape is pinned by the validator and it's a routing diagnostic, not a headline.
• New source-discipline rule: the content under review never verifies its own technical claims (on PR add blog post for automatic logging #20371, "encrypted using AES256-GCM" was ✅-verified citing the blog post itself, while pkg/logging/encrypted.go says encryption is conditional).

Gap 2 — turn-cap escalation + a machine-readable "retryable" marker

Six of PR #20556's seven unverifiables read literally "verification did not converge within 8 turns" (the cap's own comment predicted this failure mode); PR #20371's log-rotation claim (pure CLI behavior, no pulumi-shaped token) routed to web search, which cannot read Go source. Both then surfaced downstream as generic unverifiable — narrated by the sweep as "out of scope."

• pass1 turn cap 8→12; exhausting it now auto-escalates once to pass3 instead of returning bare unverifiable (the explicit-escalation valve existed but was unreachable once capped).
• Symmetric reverse escalation: a pass3 verifier may emit route_escalation: "pass1" for Pulumi-product-behavior claims, one hop, ping-pong guarded.
• Terminal turn-cap records carry turn_cap_exhausted: true plus retryable phrasing; the composer stubs them with an explicit "never call this out of scope" TODO, so a budget failure and an architectural limit can no longer render identically to any consumer.

Gap 3 — SHA-based review freshness (labels alone can't be trusted)

PR #20556's Copilot conflict-resolution push at 2026-07-30T00:47Z fired no pull_request: synchronize event (confirmed from the Actions run history), so mark-stale never ran and the PR closed at review:no-blockers with a pinned review describing replaced content. Ordinary App pushes (workprentice[bot]) do fire the event — the suppressed class is Copilot-agent and GITHUB_TOKEN pushes.

• compose-review.py stamps a machine-readable <!-- CLAUDE_REVIEW_HEAD <sha> --> sentinel under the review header (the workflow already passes --head-sha); update.md requires the re-entrant path to refresh it.
• /pr-review Step 2 runs a SHA freshness check before trusting CURRENT labels, with a last-reviewed-sha review-history fallback for pre-sentinel reviews.
• New scheduled workflow review-label-reconcile.yml (every 2h, deterministic, label transitions only): compares each labeled open PR's reviewed SHA against its live head and applies the same transition mark-stale would have. Skips when it can't justify a flip.

Tests

New test_framing_drift.py (36 checks: cross-script constant sync, coercion, both escalation hops, the cap marker, stub placement, validator acceptance/violation in both directions, header sentinel). All existing docs-review and content-review suites pass; make lint clean.

Out of scope

The pr-review-sweep punt-template wording lives in the work-ops plugin (a personal plugin repo unreachable from this session). The turn_cap_exhausted marker makes the conflation impossible at the source regardless; the template fix should still land separately.

Unreleased product version (optional)

N/A — pipeline/tooling only; no product docs content changed.

Related issues (optional)

Follow-up from the 2026-07-29 punted-PR adjudication of #20371, #20550, #20552, #20556.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KcNQVWHUyvbbe5Go9MGxwA

---

Generated by Claude Code

[pulumi-bot]

Your site preview for commit f0480b5 is ready! 🎉

http://www-testing-pulumi-docs-origin-pr-20577-f0480b57.s3-website.us-west-2.amazonaws.com