ci: run the Python contract suites - #37
Merged
Merged
Conversation
Neither NeuralComposeEEG nor Tests/eval has ever run in CI — build-and-test is
Swift-only. That is how a test importing an uncommitted module reached a pull
request unnoticed, and it left 62 Python tests ungated.
Adds a `python-contracts` job on ubuntu-latest (nothing here is
platform-specific, and the runner is cheaper than macOS).
pytest is a correctness requirement, not a preference. Tests/eval mixes four
harness styles, and most modules are bare `def test_*` functions with no
TestCase and no __main__ runner. `python -m unittest Tests.eval.<module>`
collects ZERO tests from those and prints OK — 8 of 62 tests would have run
behind a green check. pytest collects all 62.
Excluded, each for a stated reason rather than to force green:
test_joint_embedding.py imports mlx.core; Apple Silicon only
3 deselected cases pre-existing failures on this base, reproducing on
both NumPy 1.26.4 and 2.4.6; tracked separately.
The rest of their files still run.
Requires the lazy trapezoid resolution in the parent commit: test_session_consume
imports eeg_spectral, which could not import on numpy>=2 before it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Tracking issue for the three deselected cases: #38. It records which are wrong tests and which is a real defect — briefly: |
This was referenced Jul 27, 2026
aurascoper
changed the base branch from
fix/numpy-trapezoid-compat
to
docs/eeg-methods-scope
July 27, 2026 16:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #36 — merge that first.
test_session_consumeimportseeg_spectral, which cannot import on numpy>=2 without it, so this job would be red on its own.Prerequisite for #33 going ready, per the agreed gate
github_ci_runs_tests_eval: true.Why nothing caught the missing module
build-and-testis Swift-only. NeitherNeuralComposeEEG/testsnorTests/evalhas ever run in CI, leaving 62 Python tests ungated — which is exactly how #33's test could import an uncommitted module and still show a green check.pytest is required here, not preferred
Tests/eval mixes four harness styles. Most modules are bare
def test_*functions with noTestCaseand no__main__runner:python -m unittestcollectsunittest prints
OKfor every one of those zero-collection modules. My first draft of this job used unittest and would have shipped a green check that ran 8 of 62 tests. pytest collects all 62.Exclusions, each with a reason
test_joint_embedding.pyimportsmlx.core(Apple Silicon only) — run locally.Three cases are deselected because they already fail on this base, on both NumPy 1.26.4 and 2.4.6. They are not caused by this PR and are not silently dropped — see the tracking issue. The remaining tests in those files still run.
Result
torch installs from the CPU wheel index — the default Linux wheel is a ~800MB CUDA build for a runner with no GPU.