test(eval): pin biased linear CKA behavior across sample regimes - #41
Merged
Merged
Conversation
No estimator change. cka() implements biased linear CKA over centered Gram
matrices, and that estimator is deliberately kept: switching to unbiased CKA
would alter every prior representation-comparison result and needs a registered
analysis migration, not a test cleanup.
The old expectation was wrong for it. Independent Gaussians at n=50 with
d=128/256 score ~0.78, because the biased estimator's null is inflated when
feature dimension approaches or exceeds sample count. Asserting < 0.3 there
assumed an unbiased estimator or n >> d.
Replaced with two tests pinning both regimes, so neither the estimator nor the
inflation can change unnoticed:
n=500, d=16/32 0.042 independent scores near zero
n=50, d=128/256 0.785 characterizes the inflated null
The second is explicitly labelled as characterizing the estimator, not as
evidence the representations are similar.
cka() gains a docstring stating the inflation and warning against comparing
scores across n/d regimes without a matched null or permutation baseline.
Removes the last of three deselections in the python-contracts job. The eval
gate now runs with zero deselections; closes #38.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aurascoper
changed the base branch from
test/mann-whitney-attainable-significance
to
docs/eeg-methods-scope
July 27, 2026 17:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of three PRs resolving #38. Stacked on #40 — merge #39, then #40, then this. Closes #38.
The estimator is deliberately unchanged
cka()implements biased linear CKA over centered Gram matrices, and it stays that way. Switching to unbiased CKA would alter every prior representation-comparison result and requires a registered analysis migration, not a test cleanup.The expectation was wrong for that estimator
The biased estimator's null is inflated when feature dimension approaches or exceeds sample count. At n=50 with d=128 and 256, ~0.78 on independent data is the estimator behaving as designed. Asserting < 0.3 there assumed either an unbiased estimator or n ≫ d.
Two tests, both regimes pinned
The second is explicitly labelled as characterizing the estimator, not as evidence the representations are similar — so a future reader can't mistake a high null score for a finding. Neither the estimator nor the inflation can now change unnoticed.
cka()also gains a docstring recording the inflation and warning against comparing scores across n/d regimes without a matched null or permutation baseline.The eval gate reaches zero deselections
Final counts pending CI. The last
--deselectline is removed from thepython-contractsjob, and its stale "8 of 62" comment is corrected.