Skip to content

[ENH] Remove handling of GPU device ordinals #271

Open

Description

@chyunsu3

My recent work on serialization (#258, #269) ended up revealing subtle bugs with activating multiple GPU devices. #270 fixes some, but there are bound to be more issues.

The difficulty of managing multiple GPU devices in the same Python process is evident, given that it involves setting and managing implicit hidden states via cudaSetDevice and cudaGetDevice. Attempting to read or copy a memory region without a correct device context leads to cryptic errors or outright crashes. Our experience with managing device ordinal in the XGBoost project is not positive, as @trivialfis and @RAMitchell can attest.

Proposal. Remove all device management code from nvForest. nvForest should use the currently active GPU device and should not attempt to activate another GPU device.

To my best knowledge, NVIDIA cuML uses this approach. Users are expected to use cuda.core.Device explicitly before invoking cuML, if they wish to use a non-default GPU device.

TODO.

  • Do scoping work to estimate the engineering effort required.
  • Make a deprecation plan.

Activity

  1. self-assigned this
    on Nov 5, 2026
  2. changed the title [-][ENH] Refactor to remove handling of GPU device ordinals[/-] [+][ENH] Remove handling of GPU device ordinals[/+] on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions