Skip to content

Build/Test Tools: Reset shared state in the base test case instead of per test file - #95

Open
sirreal wants to merge 4 commits into
trunkfrom
agent/test-framework-shared-resets
Open

Build/Test Tools: Reset shared state in the base test case instead of per test file#95
sirreal wants to merge 4 commits into
trunkfrom
agent/test-framework-shared-resets

Conversation

@sirreal

@sirreal sirreal commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Summary

A proposal, split out from the review of WordPress#13182WordPress#13196 so it can be judged on its own.

Across those fifteen PRs the same shape kept recurring: the leak was fixed in the consumer, not the producer. Four of the local patches are symptoms of gaps in the shared fixture itself, so they are proposed here as one framework change to WP_UnitTestCase_Base instead of fifteen local ones.

Each change is a separate commit so any subset can be dropped.

# Change Gap it closes
1 Reset the metadata lazyload queue in set_up() tear_down() already resets it, but wpSetUpBeforeClass() runs outside any test's tear_down(), so fixture-queued items leak into the class's first test
2 Add comment to the reset comment globals comment_alt/comment_depth/comment_thread_alt were reset; comment was not. get_comment() falls back to that global
3 Reset $_SERVER in tear_down() set_up() resets it, but class fixtures run before a class's first set_up(), so HTTPS etc. from a previous class are live while fixtures are built
4 Capture $ignore_files in set_up_before_class() It was captured on the first set_up(), i.e. after the first class's fixtures had already uploaded — permanently exempting those files from remove_added_uploads()

On (4): the check is against null, not falsiness. An empty uploads directory is a legitimate captured state; with a falsy check the scan repeats at every class boundary and pins files left behind by earlier classes. Getting this wrong made residue worse (34 leftover files vs. 2) before it was corrected — see Verification.

Verification

Full local suite, single site, PHP 8.4 / MySQL. Each arm started from an identical empty uploads directory, since residue otherwise carries between runs and corrupts the comparison.

Ordering trunk with this PR
Default 1 failure, 4,559,413 assertions, 48 leftover upload files 1 failure, 4,559,413 assertions, 0 leftover files
--order-by=reverse 129 failures, 15 errors 128 failures, 15 errors
--order-by=random --random-order-seed=20260820 82 failures, 12 errors, 2 leftover files 82 failures, 12 errors, 0 leftover files
  • Fixed: Tests_Comment_GetCommentAuthorUrlLink::test_no_comment (reverse order). Reproduced in isolation first: --order-by=reverse on that file alone fails on trunk, passes with this change.
  • Newly broken: none, in any of the three orderings.
  • The single default-order failure (Tests_Theme_ThemeDir::test_broken_themes) is a pre-existing local environment artifact — a stray html directory in the themes folder — and is present identically before and after.
  • phpcs --standard=phpcs.xml.dist clean; php -l clean; git diff --check clean.

Honest limitations

  • Multisite was not run locally. Two of the sibling PRs went red on multisite only, and this touches the shared base class, so multisite CI is the real gate here.
  • Changes 1 and 3 produced no measurable difference in any of the three orderings tested. They close the wpSetUpBeforeClass() window by symmetry with resets that already exist, but they are defensive rather than evidenced. Reviewers may reasonably want them dropped or held until a failing ordering is demonstrated.
  • Change 4 is what accounts for the leftover-file improvement (48 → 0 default, 2 → 0 random). Note that no current test class both uploads in wpSetUpBeforeClass() and calls remove_added_uploads(); the effect is cross-class, via the shared static.
  • Change 4 makes scan_user_uploads() and files_in_dir() static so they can be called from set_up_before_class(). Both are used only inside abstract-testcase.php in core, and $this->files_in_dir() calls keep working — but a downstream suite that overrides either as a non-static method would fatal. That is the main back-compat question worth a decision.

Relationship to the sibling PRs

Change 2 makes the local tear_down() in WordPress#13186 redundant. The others overlap in spirit with the resets proposed in WordPress#13188, WordPress#13190, WordPress#13193 and WordPress#13194 but do not conflict textually with any of them.

Use of AI Tools

This change was written with Claude Code. The diagnosis, the patch, and every measurement in the Verification table were produced and checked by the agent; the numbers above come from actual local suite runs, not estimates.

Trac: https://core.trac.wordpress.org/ticket/65893

🤖 Generated with Claude Code

`tear_down()` already resets the queue after every test, but fixtures created
in `wpSetUpBeforeClass()` run outside of any test's `tear_down()`. Anything
queued while building class fixtures therefore leaks into the first test of
that class.

See #65893.
`tear_down()` resets `comment_alt`, `comment_depth` and `comment_thread_alt`,
but not `comment` itself, so a test that assigns `$GLOBALS['comment']` leaks it
into every later test. `get_comment()` falls back to that global, so consumers
silently observe the previous test's comment.

See #65893.
`set_up()` already resets `$_SERVER` before each test, but class fixtures
created in `wpSetUpBeforeClass()` run before the first `set_up()` of a class.
Values such as `HTTPS` left behind by a previous class are therefore still in
place while those fixtures are built.

See #65893.
`$ignore_files` records the uploads directory contents so that
`remove_added_uploads()` can tell pre-existing files from files added by tests.
It was captured on the first `set_up()`, which runs after the first class's
`wpSetUpBeforeClass()`. Files uploaded by those fixtures were treated as
pre-existing and permanently exempted from cleanup.

Capture it once in `set_up_before_class()` instead, before any fixture runs.
The check is against `null` rather than a falsy value so that an empty uploads
directory still counts as captured; otherwise the scan repeats at every class
boundary and pins files left behind by earlier classes.

See #65893.
@github-actions

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props jonsurrell.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant