test(eval): use an attainable exact Mann-Whitney significance case - #40
Merged
aurascoper merged 1 commit intoJul 27, 2026
Merged
Conversation
No production change. The test asserted a p-value the test's own sample size
makes unreachable: with n1 = n2 = 3, the smallest attainable two-sided p is
2/20 = 0.1, so completely disjoint groups still return exactly 0.1 and no
correct implementation can clear 0.05.
Four per group makes the claim attainable — the exact minimum is 2/70 = 0.029 —
and the assertions now pin U and both sample sizes rather than only the
p-value.
n=3 per group U=0.0 p=0.1000
n=4 per group U=0.0 p=0.0286
Removes the second of three deselections in the python-contracts job (#38).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aurascoper
changed the base branch from
fix/eval-stats-zero-variance-cohens-d
to
docs/eeg-methods-scope
July 27, 2026 17:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of three PRs resolving #38. Stacked on #39 — merge that first, then this retargets to
docs/eeg-methods-scope.No production change
The test asserted a p-value its own sample size makes unreachable:
With n₁ = n₂ = 3 there are only C(6,3) = 20 arrangements, so the smallest attainable two-sided p is 2/20 = 0.1. Completely disjoint groups return exactly that. No correct implementation can pass this assertion at that sample size. The implementation was right; the test was impossible.
Four per group makes the claim attainable
Both verified against the real implementation. The assertions now also pin
u_statisticand both sample sizes, so a future change to the return shape can't slip through on the p-value alone.Result
test_eval_stats10 passed locally. Removes the second of three deselections.