Skip to content

fix(eval): stop zero-variance Cohen's d from reporting no effect - #39

Merged
aurascoper merged 1 commit into
docs/eeg-methods-scopefrom
fix/eval-stats-zero-variance-cohens-d
Jul 27, 2026
Merged

fix(eval): stop zero-variance Cohen's d from reporting no effect#39
aurascoper merged 1 commit into
docs/eeg-methods-scopefrom
fix/eval-stats-zero-variance-cohens-d

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

First of three PRs resolving #38. Removes exactly one CI deselection.

The defect

cohens_d returned 0.0 whenever pooled SD was zero, regardless of whether the means were identical or completely separated:

cohens_d([1, 1, 1, 1], [5, 5, 5, 5])  # 0.0 — "no effect" for total separation

Zero pooled variance now distinguishes the two cases:

input before after
equal constant groups 0.0 0.0 (a true zero)
separated constant groups 0.0 NaN (undefined)
under-sized samples NaN NaN (unchanged)

NaN over infinity, per the issue: d is undefined without within-group variation, NaN doesn't pretend the effect is absent, it matches the function's existing under-sized-sample behaviour, and it forces callers to treat the estimate as degenerate rather than ranking it as an ordinary finite effect.

Consolidation, because there were four copies

eval_stats.py's docstring already claims it consolidates cohens_d "previously duplicated in embedding_analyze.py and statistical_analysis.py" — but the duplicates were never removed, and analyze_results.py held a fourth. All four were byte-identical in logic, so fixing one would have left three wrong ones.

analyze_results.py, embedding_analyze.py, and statistical_analysis.py now import the shared implementation, following the sys.path.insert(0, str(Path(__file__).parent)) idiom already used in this directory for bootstrap_ci. Verified at runtime that all three resolve to the same function object.

Scope note: mann_whitney_u is also duplicated across these scripts. Left alone deliberately — that is 38-B's subject and does not belong in this diff.

Tests

test_cohens_d_large_effect conflated "large" with "zero-variance degenerate". Replaced by three tests that separate the cases:

test_cohens_d_identical_constant_groups_are_zero
test_cohens_d_separated_constant_groups_are_undefined
test_cohens_d_large_nondegenerate_effect

Expected CI counts

The issue projected 84 passed, 2 deselected. Actual will be 88 collected / 2 deselected / 86 passed: this PR replaces one test with three, so the collection grows by two. Nothing was skipped — the progression is 83 → 86 passed because the suite gained tests alongside the fix.

Local: test_eval_stats 9 passed, 1 failed (the still-deselected Mann-Whitney case, 38-B's subject). py_compile, YAML parse, and git diff --check all clean.

cohens_d returned 0.0 whenever pooled standard deviation was zero, regardless
of whether the means were identical or completely separated. Two perfectly
separated constant groups therefore reported "no effect" — the least correct
available answer, and one that ranks alongside genuine null results.

Zero pooled variance now distinguishes the two cases:

    equal constant groups       0.0   a true zero
    separated constant groups   NaN   undefined, not absent

NaN rather than infinity: d is undefined without within-group variation, NaN
does not pretend the effect is absent, it matches the function's existing
under-sized-sample behaviour, and it forces callers to treat the estimate as
degenerate instead of ranking it as an ordinary finite effect.

Finishes a consolidation the module already claimed. eval_stats.py's docstring
says it consolidates cohens_d "previously duplicated in embedding_analyze.py
and statistical_analysis.py", but the duplicates were never removed — and
analyze_results.py had a fourth copy. All three now import the shared
implementation, following the established `sys.path.insert` + `from eval_stats
import ...` idiom already used for bootstrap_ci. Fixing one copy would
otherwise have left three wrong ones.

test_cohens_d_large_effect conflated "large" with "zero-variance degenerate";
it is replaced by three tests that separate identical-constant, separated-
constant, and large-non-degenerate cases.

Removes the first of three deselections in the python-contracts job (issue #38).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aurascoper
aurascoper merged commit b8894ee into docs/eeg-methods-scope Jul 27, 2026
2 checks passed
@aurascoper
aurascoper deleted the fix/eval-stats-zero-variance-cohens-d branch July 27, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant