Skip to content

test(eval): use an attainable exact Mann-Whitney significance case - #40

Merged
aurascoper merged 1 commit into
docs/eeg-methods-scopefrom
test/mann-whitney-attainable-significance
Jul 27, 2026
Merged

test(eval): use an attainable exact Mann-Whitney significance case#40
aurascoper merged 1 commit into
docs/eeg-methods-scopefrom
test/mann-whitney-attainable-significance

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Second of three PRs resolving #38. Stacked on #39merge that first, then this retargets to docs/eeg-methods-scope.

No production change

The test asserted a p-value its own sample size makes unreachable:

mann_whitney_u([1, 2, 3], [10, 11, 12])   # p = 0.1, asserted < 0.05

With n₁ = n₂ = 3 there are only C(6,3) = 20 arrangements, so the smallest attainable two-sided p is 2/20 = 0.1. Completely disjoint groups return exactly that. No correct implementation can pass this assertion at that sample size. The implementation was right; the test was impossible.

Four per group makes the claim attainable

n=3 per group   U=0.0  p=0.1000     (exact minimum 2/20)
n=4 per group   U=0.0  p=0.0286     (exact minimum 2/70)

Both verified against the real implementation. The assertions now also pin u_statistic and both sample sizes, so a future change to the return shape can't slip through on the p-value alone.

Result

test_eval_stats 10 passed locally. Removes the second of three deselections.

No production change. The test asserted a p-value the test's own sample size
makes unreachable: with n1 = n2 = 3, the smallest attainable two-sided p is
2/20 = 0.1, so completely disjoint groups still return exactly 0.1 and no
correct implementation can clear 0.05.

Four per group makes the claim attainable — the exact minimum is 2/70 = 0.029 —
and the assertions now pin U and both sample sizes rather than only the
p-value.

    n=3 per group   U=0.0  p=0.1000
    n=4 per group   U=0.0  p=0.0286

Removes the second of three deselections in the python-contracts job (#38).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aurascoper
aurascoper changed the base branch from fix/eval-stats-zero-variance-cohens-d to docs/eeg-methods-scope July 27, 2026 17:39
@aurascoper
aurascoper merged commit c064bcb into docs/eeg-methods-scope Jul 27, 2026
2 checks passed
@aurascoper
aurascoper deleted the test/mann-whitney-attainable-significance branch July 27, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant