Skip to content

Make torch and the sklearn stack optional extras - #23

Merged
cboulay merged 1 commit into
devfrom
feat/optional-backend-extras
Aug 19, 2026
Merged

Make torch and the sklearn stack optional extras#23
cboulay merged 1 commit into
devfrom
feat/optional-backend-extras

Conversation

@cboulay

@cboulay cboulay commented Aug 19, 2026

Copy link
Copy Markdown
Member

Closes #22.

The pure-numpy processors need none of torch, scikit-learn, river or pandas, but installing ezmsg-learn to use e.g. the LRR re-referencer paid for a full PyTorch install. The backends now live behind extras:

pip install ezmsg-learn              # numpy-only processors
pip install "ezmsg-learn[sklearn]"   # + pandas, river, scikit-learn
pip install "ezmsg-learn[torch]"     # + torch
pip install "ezmsg-learn[all]"       # everything

This is a breaking change for anyone relying on pip install ezmsg-learn to supply a backend, with no deprecation cycle — as agreed on the issue, since the main downstream consumer is updated in lockstep.

Two adjustments to the shape proposed in the issue

  • scipy had to join the base dependencies, not an extra. model/refit_kalman.py imports scipy.linalg.solve_discrete_are at module level and reaches it transitively via scikit-learn today; it would have broken on a base install. numpy and array-api-compat are declared as the issue suggested.
  • dim_reduce belongs to the sklearn group. dim_reduce/adaptive_decomp.py imports sklearn.decomposition at module level, and incremental_decomp.py imports from it. The issue's module table doesn't list either.

Error messages

New src/ezmsg/learn/_optional.py builds the message; each module behind an extra wraps its backend import:

ezmsg.learn.process.base requires the optional 'torch' dependencies of ezmsg-learn,
which are not installed. Install them with: pip install "ezmsg-learn[torch]"

Two modules do better than a straight guard

  • util.py needs no extra at all now. Only the ADAPTIVE_REGRESSORS/STATIC_REGRESSORS registries touch river/sklearn, so they're built on demand; the enums and ClassifierMessage are pure ezmsg. The public registry names are preserved lazily via PEP 562 __getattr__.
  • collection/sample_adapt_regressor.py imports its backend lazily, so it needs only the extra for the model_type in use — and none at all for model_type="kalman". It would otherwise have been the one module requiring [all].

Tests and CI

tests/unit/test_optional_deps.py asserts two properties:

  1. In a subprocess, that no base-install module transitively loads a backend — this is what catches someone later adding a top-level import torch to ssr.py.
  2. That each guarded module names its extra. Only observable without the extra installed, so these skip in the dev environment and run for real in a new minimal-install CI job that installs with no extras.

The test and docs dependency-groups pull ezmsg-learn[all], so uv sync and the docs build still get every backend. (tool.uv.default-extras isn't supported by the current uv, so the self-referential extra is the working route.)

The docs workflow also gains --no-sync: uv run was re-syncing to the default groups and silently undoing the preceding uv sync --only-group docs, so the declared docs environment and the built one were different. The docs group now declares ezmsg-learn[all] outright instead of inheriting backends from the test group.

Verification

  • Full suite: 356 passed, 24 skipped (unit + dim_reduce + integration); ruff check and ruff format clean.
  • Fresh venv with pip install . installs no torch/sklearn/river/pandas. All 33 optional-dep tests pass there; from ezmsg.learn.process.ssr import LRRSettings, LRRUnit works with torch absent from sys.modules; the kalman collection builds with zero backends loaded.
  • Docs, running the new CI sequence from a deleted .venv: build succeeds with the same 4 pre-existing warnings, all 20 module source pages generated, and API pages carry real content across both extras (process.torch 9 classes, process.rnn 4, dim_reduce.adaptive_decomp 10, process.ssr 7).

🤖 Generated with Claude Code

The pure-numpy processors needed none of torch, scikit-learn, river or
pandas, but installing ezmsg-learn to use e.g. the LRR re-referencer paid
for a full PyTorch install. Move the backends behind extras:

    pip install ezmsg-learn              # numpy-only processors
    pip install "ezmsg-learn[sklearn]"   # + pandas, river, scikit-learn
    pip install "ezmsg-learn[torch]"     # + torch
    pip install "ezmsg-learn[all]"       # everything

numpy, scipy and array-api-compat become declared base dependencies; they
are imported directly today and only arrived transitively. scipy has to
be a base dependency rather than an extra because model/refit_kalman.py
imports scipy.linalg at module level and reaches it via scikit-learn
today. dim_reduce belongs to the sklearn group: adaptive_decomp.py
imports sklearn.decomposition at module level.

Modules behind an extra wrap their backend import so a missing install
names the extra to install instead of raising a bare
ModuleNotFoundError. Two modules do better than a straight guard:

- util.py needs no extra at all now. Only the regressor registries touch
  river/sklearn, so they are built on demand; the enums and
  ClassifierMessage are pure ezmsg. Public registry names are preserved
  lazily via PEP 562 __getattr__.
- collection/sample_adapt_regressor.py imports its backend lazily, so it
  needs only the extra for the model_type in use -- and none at all for
  model_type="kalman". It would otherwise be the one module requiring
  [all].

This is a breaking change for anyone relying on `pip install ezmsg-learn`
to supply a backend; no deprecation cycle.

tests/unit/test_optional_deps.py asserts in a subprocess that no
base-install module transitively loads a backend, and that each guarded
module names its extra. The latter is only observable without the extra
installed, so it skips in the dev environment and runs for real in a new
minimal-install CI job.

The test and docs dependency-groups pull ezmsg-learn[all] so `uv sync`
and the docs build still get every backend. The docs workflow gains
--no-sync: `uv run` was re-syncing to the default groups and silently
undoing the preceding `uv sync --only-group docs`, so the environment
that was declared and the one that got built were different.

Closes #22

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cboulay
cboulay merged commit d5ba18f into dev Aug 19, 2026
9 checks passed
@cboulay
cboulay deleted the feat/optional-backend-extras branch August 19, 2026 04:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make torch (and the sklearn stack) optional extras

1 participant