Build/Test Tools: add a plugin compatibility workflow to catch fatals with popular plugins before release.

[adamsilverstein]

Claude drafted this issue after we talked through the WP Rocket breakage:

What happened

WordPress 7.1 shipped and sites running WP Rocket started fataling on every request. Austin Ginder reported that all of his WP Rocket customer sites went offline until he manually patched the plugin: https://x.com/austinginder/status/2090199834787541074

The error is Uncaught TypeError: substr(): Argument #1 ($string) must be of type string, int given in Cloudflare.php:562 - see https://wordify.com/blog/fixed-wp-rocket-fatal-error-wordpress-7-1-uncaught-typeerror-substr-argument-1-string-int-given-cloudflare-php-562/. The plugin's assumptions about a core value's type stopped holding, and the result was a white screen for a lot of real sites.

Why core CI should catch this class of failure

Core's CI is thorough about testing core itself - PHPUnit, e2e, upgrade paths, install paths - but nothing checks that a new version of WordPress can boot with popular plugins active. The ecosystem check happens informally (beta testers, hosts, plugin authors), and this incident shows the gap: a fatal that takes down every request on a top-tier plugin made it through an entire release cycle.

A cheap automated smoke test - install WordPress at the version under test, then install and activate each of the top 100 plugins and confirm nothing fatals - would catch this class of breakage before release, when there is still time to fix core or notify the plugin author. It would not have caught WP Rocket specifically (premium plugin, not in the .org directory), but the same type-change fatal could just as easily land in any of the free plugins with 1M+ installs, and the check gives release leads a concrete pre-release signal that is currently missing.

Proposed approach

Add a new GitHub Actions workflow, plugin-compatibility.yml (plus a reusable companion, following the existing upgrade-testing.yml / reusable-upgrade-testing.yml pattern), that:

1. Fetches the current top 100 plugins by popularity from https://api.wordpress.org/plugins/info/1.2/?action=query_plugins&request[browse]=popular&request[per_page]=100 at run time - no hardcoded list to go stale.
2. Installs WordPress via WP-CLI at the version under test - latest, nightly, a specific version, or a zip built from the current branch, matching how reusable-upgrade-testing.yml handles develop.
3. For each plugin: wp plugin install <slug> --activate, then verifies WordPress still runs - activation exit code, a wp eval smoke check (loads all of core plus active plugins), and front end + /wp-admin/ requests against php -S asserting no 5xx and no Fatal error in the response or debug log.
4. Deactivates and deletes the plugin before moving to the next, so each plugin is tested in isolation and one broken plugin can't mask another.
5. Writes a per-plugin results table to the workflow summary and fails the run if any plugin fatals - while still testing the remaining plugins.
6. Shards the plugin list across a small matrix (eg. 5 shards of 20) to keep wall time reasonable.

Runs would trigger on workflow_dispatch (so release leads can point it at an RC), and on a weekly schedule against nightly, following the install-testing.yml precedent.

Acceptance criteria

• A workflow_dispatch trigger accepts a WordPress version (latest, nightly, or specific version) and a plugin count (default 100), so it can be run on demand against a beta/RC before release.
• A weekly scheduled run tests nightly automatically.
• The workflow can be triggered manually as needed, for example as part of the pre-release checklist.
• The plugin list is fetched from the WordPress.org API at run time, sorted by popularity; there is no hardcoded plugin list.
• Every plugin in the list is installed and activated against the target WordPress version, one at a time, in isolation.
• A fatal is detected whether it happens during activation, during a WP-CLI load, or on a front end / admin request (WSOD with display_errors off must still be caught, via the debug log and HTTP status).
• One plugin's failure does not stop the remaining plugins from being tested; the workflow reports all failures at the end.
• The workflow summary shows a table of plugin slug, plugin version, and pass/fail, so a failing run is actionable without digging through logs.
• The overall run fails if and only if at least one plugin fatals (plugins that fail to download, eg. network flakes, are reported as skipped, not failed).
• Total wall time stays under ~20 minutes via sharding.
• The workflow follows repo conventions: top-of-file doc comment, permissions: {} at the top with job-level grants, pinned action SHAs, github.repository == 'WordPress/wordpress-develop' guards, and hooks into slack-notifications.yml / failed-workflow.yml like the other workflows.
• The workflow passes the repository's workflow linting.
• There should be a way to run on the top 10, 50, 250 or any arbitrary number of plugins, perhaps with a text field in the manual run

Implementation plan

1. reusable-plugin-compatibility.yml: a reusable workflow taking wp-version, php-version, plugin-slugs (JSON array for one shard) - sets up PHP + MySQL service + WP-CLI, installs WordPress, loops the shard's plugins through install/activate/verify/cleanup, appends results to $GITHUB_STEP_SUMMARY, exits non-zero if any plugin fataled.
2. plugin-compatibility.yml: the caller - a small job that queries the .org API, builds the shard matrix, then fans out to the reusable workflow; plus the Slack notification and failed-workflow retry jobs.
3. The per-plugin verify step as a small shell function: activation exit code, a wp eval 'echo "OK";' load check (WP-CLI boots all of core plus active plugins), curl of / and /wp-admin/ via php -S, grep of wp-content/debug.log for PHP Fatal.
4. Follow-up (out of scope for the first PR): a .version-support-*.json-style config for PHP versions, multisite runs, and testing plugins in combination rather than isolation.

Open questions

• Should failures block anything, or is this signal-only to start? Signal-only (schedule + manual runs, Slack notification on failure) seems like the right first step - a broken third-party plugin shouldn't turn core CI red on every commit.
• Is 100 the right number? The API makes the count an input, so release leads could run 250 for an RC.
• Premium plugins (like WP Rocket itself) can't be fetched from the .org API. Down the road, hosts or vendors could contribute zips via a separate input, but that has licensing questions.
• Does some of this overlap with what the plugin directory's Tide/Plugin Check infrastructure already does? As far as I can tell nothing runs plugins against unreleased core versions, but confirmation from someone closer to that infrastructure would help.

Related: the Installation Tests workflow (.github/workflows/install-testing.yml) and Upgrade Tests (.github/workflows/upgrade-testing.yml), which this follows closely in structure.

Trac ticket: https://core.trac.wordpress.org/ticket/65920

AI Use

Description written with 🤖 Claude Code; the PR that follows will contain Claude-written code and carries its own AI Use section.

[adamsilverstein]

PR is up: WordPress#13198

[adamsilverstein]

Arbitrary plugin counts and manual pre-release runs are covered in WordPress#13198 - the API caps per_page at 250, so the fetch now pages through it, and shards scale with the count.