ci: decide the changeset allowlist before validating the queue - #721
Conversation
The gate ran `changeset status` before it worked out whether the PR was allowlisted. That command errors with "Some packages have been changed but no changesets were found" for any diff inside a package, and a docs-only PR is such a diff, so the check failed on exactly the PRs the allowlist exists to exempt and the exemption step never ran. PR #715 changes markdown only and fails this way today. The allowlist verdict now runs first and publishes what it decided. The Changesets queue validation still runs whenever the PR needs a changeset, and also when an allowlisted PR carries one anyway: a `.changeset/*.md` file matches the markdown allowlist, so a changeset-only PR is exempt from the gate and would otherwise ship its entry unvalidated. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9479932846
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if: >- | ||
| steps.version_pr.outputs.skip != 'true' | ||
| && steps.label.outputs.skip != 'true' | ||
| && (steps.gate.outputs.exempt != 'true' || steps.gate.outputs.has_changesets == 'true') |
There was a problem hiding this comment.
Update the metadata test for the folded guard
On every PR targeting master, the unconditional Validate package and lockfile metadata step runs web/release-metadata.test.mjs, whose check at lines 172–181 searches for the exact single-line if: guard. Splitting that guard across this folded YAML block makes the assertion fail before the changeset gate executes, so all such workflow runs remain red. Update the test to recognize the expanded condition or preserve the expected guard contiguously.
AGENTS.md reference: AGENTS.md:L99-L100
Useful? React with 👍 / 👎.
The workflow-structure test asserted that each Changesets step carried the label guard by matching the exact concatenation of both short-circuits. Adding a third condition to a step therefore read as a removed guard. It now checks the two short-circuits separately, which is as strict about the property it names and lets a step carry a further guard of its own. Adds the assertion this change is actually about: the allowlist verdict must be resolved before Changesets validation runs. Restoring the old order fails the test, so it guards the regression rather than describing it. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
Second commit: the workflow-structure test needed updating, and that is worth reading rather than skimming.
I also added the assertion this PR is actually about — the allowlist verdict must be resolved before Changesets validation runs. I checked it is a real guard and not a rubber stamp: restoring the old step order fails it with Two notes:
|
The changeset gate fails documentation-only PRs. #715 changes markdown only — every path matches the gate's own allowlist — and the check is red.
Cause
The job ran its steps in the wrong order.
npx changeset statuscame before the allowlist decision:changeset statuserrors with "Some packages have been changed but no changesets were found" for any diff inside a package. A docs-only PR is such a diff, so the command failed on exactly the PRs the allowlist exists to exempt, and the exemption step it would have reached was skipped. Evidence: run 30456460013, job 90591164178 on #715 — the step order above is the job's own step list, and the error text is the last line before the failure.The workflow header already documented the intended order ("3. All changed files are in the path allowlist → skip. 4. Otherwise: require a changeset"). The implementation did not follow it.
Change
The allowlist verdict runs first and publishes what it decided as step outputs. The Changesets queue validation still runs, gated on that verdict.
It runs when the PR needs a changeset — unchanged — and also when an allowlisted PR carries one anyway. That second case matters and is easy to miss:
.changeset/*.mdmatches the markdown allowlist, so a PR consisting of a changeset plus docs is exempt from the gate. Skipping validation on "exempt" alone would let such an entry reach master unvalidated, which is the failure the validation step was added to prevent.No change to what is allowlisted, to the
no-changesetlabel escape, or to the Version PR short-circuit.Verification
The allowlist filter was run against the real file lists of two open PRs:
.changeset/foo.md-only diff → exempt by path, but detected as carrying a changeset, so validation still runs.make verifypasses locally. This PR touches only.github/, so it is itself allowlisted and exercises the new path (exempt, no changeset, validation skipped).Unblocks #715.