Skip to content

ci: decide the changeset allowlist before validating the queue - #721

Merged
frahlg merged 2 commits into
masterfrom
ci/changeset-gate-order
Jul 30, 2026
Merged

ci: decide the changeset allowlist before validating the queue#721
frahlg merged 2 commits into
masterfrom
ci/changeset-gate-order

Conversation

@frahlg

@frahlg frahlg commented Jul 30, 2026

Copy link
Copy Markdown
Member

The changeset gate fails documentation-only PRs. #715 changes markdown only — every path matches the gate's own allowlist — and the check is red.

Cause

The job ran its steps in the wrong order. npx changeset status came before the allowlist decision:

Validate pending Changesets              ← failed here
Verify changeset present (or allowlisted) ← never ran

changeset status errors with "Some packages have been changed but no changesets were found" for any diff inside a package. A docs-only PR is such a diff, so the command failed on exactly the PRs the allowlist exists to exempt, and the exemption step it would have reached was skipped. Evidence: run 30456460013, job 90591164178 on #715 — the step order above is the job's own step list, and the error text is the last line before the failure.

The workflow header already documented the intended order ("3. All changed files are in the path allowlist → skip. 4. Otherwise: require a changeset"). The implementation did not follow it.

Change

The allowlist verdict runs first and publishes what it decided as step outputs. The Changesets queue validation still runs, gated on that verdict.

It runs when the PR needs a changeset — unchanged — and also when an allowlisted PR carries one anyway. That second case matters and is easy to miss: .changeset/*.md matches the markdown allowlist, so a PR consisting of a changeset plus docs is exempt from the gate. Skipping validation on "exempt" alone would let such an entry reach master unvalidated, which is the failure the validation step was added to prevent.

No change to what is allowlisted, to the no-changeset label escape, or to the Version PR short-circuit.

Verification

The allowlist filter was run against the real file lists of two open PRs:

make verify passes locally. This PR touches only .github/, so it is itself allowlisted and exercises the new path (exempt, no changeset, validation skipped).

Unblocks #715.

The gate ran `changeset status` before it worked out whether the PR was
allowlisted. That command errors with "Some packages have been changed but
no changesets were found" for any diff inside a package, and a docs-only PR
is such a diff, so the check failed on exactly the PRs the allowlist exists
to exempt and the exemption step never ran. PR #715 changes markdown only
and fails this way today.

The allowlist verdict now runs first and publishes what it decided. The
Changesets queue validation still runs whenever the PR needs a changeset,
and also when an allowlisted PR carries one anyway: a `.changeset/*.md`
file matches the markdown allowlist, so a changeset-only PR is exempt from
the gate and would otherwise ship its entry unvalidated.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9479932846

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +178 to +181
if: >-
steps.version_pr.outputs.skip != 'true'
&& steps.label.outputs.skip != 'true'
&& (steps.gate.outputs.exempt != 'true' || steps.gate.outputs.has_changesets == 'true')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update the metadata test for the folded guard

On every PR targeting master, the unconditional Validate package and lockfile metadata step runs web/release-metadata.test.mjs, whose check at lines 172–181 searches for the exact single-line if: guard. Splitting that guard across this folded YAML block makes the assertion fail before the changeset gate executes, so all such workflow runs remain red. Update the test to recognize the expanded condition or preserve the expected guard contiguously.

AGENTS.md reference: AGENTS.md:L99-L100

Useful? React with 👍 / 👎.

The workflow-structure test asserted that each Changesets step carried the
label guard by matching the exact concatenation of both short-circuits.
Adding a third condition to a step therefore read as a removed guard. It now
checks the two short-circuits separately, which is as strict about the
property it names and lets a step carry a further guard of its own.

Adds the assertion this change is actually about: the allowlist verdict must
be resolved before Changesets validation runs. Restoring the old order fails
the test, so it guards the regression rather than describing it.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@frahlg
frahlg requested a review from erikarenhill as a code owner July 30, 2026 09:01
@frahlg frahlg added the no-changeset PR intentionally exempt from the changeset requirement (dev tooling / non-shipping) label Jul 30, 2026
@frahlg

frahlg commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

Second commit: the workflow-structure test needed updating, and that is worth reading rather than skimming.

web/release-metadata.test.mjs asserted that each Changesets step carried the label guard by matching one exact string — the concatenation of both short-circuits. Adding a third condition to a step therefore read as a removed guard, and the test failed on this PR. It now checks the two short-circuits separately: as strict about the property it names, while letting a step carry a further guard of its own.

I also added the assertion this PR is actually about — the allowlist verdict must be resolved before Changesets validation runs. I checked it is a real guard and not a rubber stamp: restoring the old step order fails it with the allowlist verdict must be resolved before Changesets validation runs.

Two notes:

  • no-changeset label applied. The test file lives under web/ because that is where the repo keeps its Node tests, so it falls outside the path allowlist even though nothing here ships to a user. This is the "manual override for cases the allowlist does not anticipate" the workflow documents.
  • @erikarenhill — this touches web/ and so lands in your CODEOWNERS area, but only release-metadata.test.mjs, which is a CI workflow-structure test rather than dashboard code. Flagging it so it is not a surprise.

@frahlg
frahlg merged commit 9542965 into master Jul 30, 2026
13 of 14 checks passed
@frahlg
frahlg deleted the ci/changeset-gate-order branch July 30, 2026 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changeset PR intentionally exempt from the changeset requirement (dev tooling / non-shipping)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant