Skip to content

test(eval): pin biased linear CKA behavior across sample regimes - #41

Merged
aurascoper merged 1 commit into
docs/eeg-methods-scopefrom
test/biased-cka-null-regimes
Jul 27, 2026
Merged

test(eval): pin biased linear CKA behavior across sample regimes#41
aurascoper merged 1 commit into
docs/eeg-methods-scopefrom
test/biased-cka-null-regimes

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Third of three PRs resolving #38. Stacked on #40 — merge #39, then #40, then this. Closes #38.

The estimator is deliberately unchanged

cka() implements biased linear CKA over centered Gram matrices, and it stays that way. Switching to unbiased CKA would alter every prior representation-comparison result and requires a registered analysis migration, not a test cleanup.

The expectation was wrong for that estimator

X = randn(50, 128); Y = randn(50, 256)
cka(X, Y)   # 0.785, asserted < 0.3

The biased estimator's null is inflated when feature dimension approaches or exceeds sample count. At n=50 with d=128 and 256, ~0.78 on independent data is the estimator behaving as designed. Asserting < 0.3 there assumed either an unbiased estimator or n ≫ d.

Two tests, both regimes pinned

regime shapes score
n ≫ d 500 × 16 vs 500 × 32 0.042
d ≫ n 50 × 128 vs 50 × 256 0.785
test_cka_independent_is_small_when_samples_exceed_dimensions
test_biased_cka_null_is_inflated_when_dimensions_exceed_samples

The second is explicitly labelled as characterizing the estimator, not as evidence the representations are similar — so a future reader can't mistake a high null score for a finding. Neither the estimator nor the inflation can now change unnoticed.

cka() also gains a docstring recording the inflation and warning against comparing scores across n/d regimes without a matched null or permutation baseline.

The eval gate reaches zero deselections

before #39   83 passed,  3 deselected
after  #39   86 passed,  2 deselected   (suite grew by 2)
after  #40   87 passed,  1 deselected
after  this  89 passed,  0 deselected   (suite grew by 1)

Final counts pending CI. The last --deselect line is removed from the python-contracts job, and its stale "8 of 62" comment is corrected.

No estimator change. cka() implements biased linear CKA over centered Gram
matrices, and that estimator is deliberately kept: switching to unbiased CKA
would alter every prior representation-comparison result and needs a registered
analysis migration, not a test cleanup.

The old expectation was wrong for it. Independent Gaussians at n=50 with
d=128/256 score ~0.78, because the biased estimator's null is inflated when
feature dimension approaches or exceeds sample count. Asserting < 0.3 there
assumed an unbiased estimator or n >> d.

Replaced with two tests pinning both regimes, so neither the estimator nor the
inflation can change unnoticed:

    n=500, d=16/32    0.042   independent scores near zero
    n=50,  d=128/256  0.785   characterizes the inflated null

The second is explicitly labelled as characterizing the estimator, not as
evidence the representations are similar.

cka() gains a docstring stating the inflation and warning against comparing
scores across n/d regimes without a matched null or permutation baseline.

Removes the last of three deselections in the python-contracts job. The eval
gate now runs with zero deselections; closes #38.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aurascoper
aurascoper changed the base branch from test/mann-whitney-attainable-significance to docs/eeg-methods-scope July 27, 2026 17:42
@aurascoper
aurascoper merged commit 41f559a into docs/eeg-methods-scope Jul 27, 2026
2 checks passed
@aurascoper
aurascoper deleted the test/biased-cka-null-regimes branch July 27, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant