Skip to content

feat: make PyTorch an optional extra - #370

Merged
hmgomes merged 3 commits into
adaptive-machine-learning:mainfrom
hmgomes:feat/torch-optional
Aug 7, 2026
Merged

feat: make PyTorch an optional extra#370
hmgomes merged 3 commits into
adaptive-machine-learning:mainfrom
hmgomes:feat/torch-optional

Conversation

@hmgomes

@hmgomes hmgomes commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Closes #371.

What and why

pip install capymoa pulled torch and torchvision as hard dependencies. On Linux that resolves the default PyPI PyTorch wheel, which is CUDA-enabled, so every user downloaded ~2.9 GB across 16 nvidia/triton packages — including someone who just wants a Hoeffding tree on a CSV. Nothing on the core stream-learning path uses it.

PyTorch is now the capymoa[torch] extra.

before after
packages from a bare install 34 (16 nvidia/triton) 27 (0 nvidia/triton)
import capymoa loads 769 torch submodules loads none
import capymoa time 1.64 s 1.00 s

This is not just a pyproject.toml change

The core import path had to stop importing torch. Type annotations were the easy part; class bodies were the blocker, since they execute at import:

x_dtype: torch.dtype = torch.float32          # base/_classifier.py
device: torch.device = torch.device("cpu")    # base/_batch.py

No TYPE_CHECKING guard helps there, so the torch-backed classes move behind a lazy boundary:

  • BatchClassifier / BatchRegressor split out of _classifier.py / _regressor.py into their own modules, leaving Classifier, MOAClassifier, SKClassifier, Regressor and friends torch-free.
  • TensorDatasetWithTransform moved to datasets/_torch_utils.py. Only capymoa.ocl uses it, and OCL requires torch anyway.
  • Torch-backed public names — Batch*, Finetune, Autoencoder, OSNN, TorchStream, ABCD — are re-exported lazily via :pep:562 module __getattr__ (capymoa/_optional.py).
  • evaluation.py detects batch learners through sys.modules instead of importing them: a Batch instance cannot exist unless its module is already loaded.
  • capymoa.ocl and capymoa.ann need torch wholesale and raise OptionalDependencyError naming the install command.

Not a breaking API change

from capymoa.classifier import Finetune still works verbatim when torch is installed — the lazy re-export is transparent. Without torch you get:

OptionalDependencyError: PyTorch is required for capymoa.ocl.
Install it with: pip install capymoa[torch]

instead of a bare ModuleNotFoundError. Existing environments are unaffected: pip install -U capymoa does not uninstall a torch that is already there.

Two things found along the way

typing_extensions was an undeclared dependency. Four core modules import override from it, but it was only arriving transitively via torch's own typing-extensions>=4.10.0. Removing torch exposed it. Now declared explicitly — a latent packaging bug independent of this change.

A lazy-export pitfall, and the reason for the conf.py hunk. Overriding a module's __dir__ to return only a curated list hides eagerly-imported names from Sphinx autosummary, which then documents classes under private module paths (capymoa.drift.detectors.abcd.ABCD instead of capymoa.drift.detectors.ABCD) and breaks every cross-reference to them — 249 nitpick failures, and since docs build with -W -n, a failed Documentation job. __dir__ now unions the module's real namespace with the lazy names. The one remaining reference, the bare Optimizer typehint, is ignored alongside the existing Tensor and nn.Module entries, because autodoc_typehints_format = "short" strips the torch.optim.optimizer prefix.

Wildcard imports

Raised in review: __all__ drives from package import *, so listing the lazy names there made a wildcard import resolve every torch-backed name and fail in a torch-free install, even for someone who only wanted the core ones. It affected capymoa.base, .classifier, .stream, .anomaly, .ssl and .drift.detectors.

When PyTorch is missing the lazy names are now dropped from __all__, so import * yields exactly what is usable. An explicit from capymoa.classifier import Finetune still raises the actionable error, and with PyTorch installed __all__ is untouched — so the public API and the generated docs are unchanged. Covered by 8 new tests and a CI step.

Validation

New "Install without PyTorch" CI job installs with no extras and asserts torch is absent, import capymoa does not load it, a prequential ARF run works, and capymoa.ocl / capymoa.ann raise an actionable error. This is what makes the fix stick — without it the first stray top-level import torch silently restores the 2.9 GB install.

tests/test_no_torch.py blocks torch on sys.meta_path so the same guarantees are checked inside the normal test matrix.

Verified locally:

  • pytest 190 passed, doctest 125 passed, all 18 notebooks pass, ruff clean
  • invoke docs.build exits 0, with the same stub attribution as main
  • a real venv with no torch installed: ARF on ElectricityTiny gives 87.9, identical to the torch-installed run
  • a built wheel installed with no extras: 27 packages, zero nvidia/triton, ARF 87.9

Caveat that survives this change

pip install capymoa[torch] on Linux still pulls the CUDA stack, because that is what the default PyPI wheel depends on, and package metadata cannot pin an index (PEP 508 has no such field). The CPU route is now documented in the setup guide:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
pip install capymoa[torch]

Note for reviewers

This touches evaluation/evaluation.py, which #356 also restructures. Merging this first means #356 needs a rebase — deliberate, since this is user-facing friction.

hmgomes added a commit to hmgomes/CapyMOA that referenced this pull request Aug 4, 2026
`__all__` drives `from package import *`. The lazy torch-backed names were
still listed there, so a wildcard import resolved every one of them and
raised OptionalDependencyError in a torch-free install -- even for a user
who only wanted the core names:

    from capymoa.base import *          -> OptionalDependencyError
    from capymoa.classifier import *    -> OptionalDependencyError

Affected capymoa.base, .classifier, .stream, .anomaly, .ssl and
.drift.detectors.

When PyTorch is missing, the lazy names are now removed from `__all__`, so
`import *` yields exactly what is usable. An explicit
`from capymoa.classifier import Finetune` still raises the actionable
error. With PyTorch installed `__all__` is untouched, so the public API and
the Sphinx docs are unchanged.

The torch probe moves into a cached `torch_available()` rather than being
re-resolved per attribute access.

Adds 8 regression tests (wildcard import per package, plus __all__
filtering with and without torch) and a CI step in the no-torch job.

Reported in review of adaptive-machine-learning#370.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@hmgomes
hmgomes force-pushed the feat/torch-optional branch from 0bfce80 to c54b436 Compare August 4, 2026 11:47
hmgomes and others added 3 commits August 7, 2026 21:33
`pip install capymoa` pulled torch and, on Linux, the whole CUDA stack --
~2.9 GB across 16 nvidia/triton packages -- for every user, including
someone running a Hoeffding tree on a CSV. torch is now behind the
`capymoa[torch]` extra.

Making it optional required the core import path to stop importing torch.
Annotation-only uses were not the hard part; class bodies were, e.g.
`x_dtype: torch.dtype = torch.float32`, which executes at import. So the
torch-backed classes move behind a lazy boundary:

- BatchClassifier / BatchRegressor split out of _classifier.py /
  _regressor.py into their own modules, leaving Classifier, MOAClassifier,
  SKClassifier and friends torch-free.
- TensorDatasetWithTransform moved to datasets/_torch_utils.py; only
  capymoa.ocl uses it, and OCL requires torch anyway.
- Torch-backed public names (Batch*, Finetune, Autoencoder, OSNN,
  TorchStream, ABCD) are re-exported lazily via PEP 562 module __getattr__,
  so `from capymoa.classifier import Finetune` still works verbatim when
  torch is installed. This is not an API change for existing users.
- evaluation.py detects batch learners via sys.modules instead of importing
  them: a Batch instance cannot exist unless its module is already loaded.
- capymoa.ocl and capymoa.ann require torch wholesale and now raise
  OptionalDependencyError naming the install command.

Also declares typing-extensions, which four core modules import and which
was only arriving transitively through torch.

tests/test_no_torch.py blocks torch on sys.meta_path and asserts the core
packages import, a classification loop runs, and torch-only names raise a
helpful error. Verified additionally in a real venv with no torch
installed: ARF on ElectricityTiny gives 87.9 accuracy, identical to the
torch-installed run. `import capymoa` no longer loads torch's 769
submodules (1.64s -> 1.00s).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the previous commit, completing the optional-PyTorch work.

- New "Install without PyTorch" job in pr.yml installs CapyMOA with no
  extras and asserts torch is absent, `import capymoa` does not load it, a
  prequential ARF run works, and capymoa.ocl / capymoa.ann raise an
  actionable OptionalDependencyError. This is the regression guard: without
  it the first stray top-level `import torch` silently restores the ~2.9 GB
  install.
- README and the setup guide document `pip install capymoa[torch]`, list
  which features need it, and explain the CPU-only route on Linux where the
  default PyPI wheel is CUDA-enabled.
- Fix a regression in the lazy-export helper: overriding a module's
  __dir__ to return only the curated name list hid eagerly-imported names
  from Sphinx autosummary, which then documented classes under their
  private module paths (capymoa.drift.detectors.abcd.ABCD instead of
  capymoa.drift.detectors.ABCD). Every cross-reference to those classes
  broke -- 249 nitpick failures, and since docs build with `-W -n` the
  Documentation job failed. __dir__ now unions the module's real namespace
  with the lazy names.
- Ignore the bare `Optimizer` typehint in nitpick_ignore_regex, alongside
  the existing `Tensor` and `nn.Module` entries, since
  autodoc_typehints_format="short" strips the torch.optim.optimizer prefix.

Verified: pytest 190 passed, doctest 125 passed, all 18 notebooks pass,
ruff clean, and `invoke docs.build` exits 0 with the same stub attribution
as main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`__all__` drives `from package import *`. The lazy torch-backed names were
still listed there, so a wildcard import resolved every one of them and
raised OptionalDependencyError in a torch-free install -- even for a user
who only wanted the core names:

    from capymoa.base import *          -> OptionalDependencyError
    from capymoa.classifier import *    -> OptionalDependencyError

Affected capymoa.base, .classifier, .stream, .anomaly, .ssl and
.drift.detectors.

When PyTorch is missing, the lazy names are now removed from `__all__`, so
`import *` yields exactly what is usable. An explicit
`from capymoa.classifier import Finetune` still raises the actionable
error. With PyTorch installed `__all__` is untouched, so the public API and
the Sphinx docs are unchanged.

The torch probe moves into a cached `torch_available()` rather than being
re-resolved per attribute access.

Adds 8 regression tests (wildcard import per package, plus __all__
filtering with and without torch) and a CI step in the no-torch job.

Reported in review of adaptive-machine-learning#370.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@hmgomes
hmgomes force-pushed the feat/torch-optional branch from c54b436 to 17c0116 Compare August 7, 2026 10:12
@hmgomes
hmgomes merged commit 7e307f0 into adaptive-machine-learning:main Aug 7, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pip install capymoa pulls ~2.9 GB of CUDA; PyTorch should be an optional extra

1 participant