You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
My recent work on serialization (#258, #269) ended up revealing subtle bugs with activating multiple GPU devices. #270 fixes some, but there are bound to be more issues.
The difficulty of managing multiple GPU devices in the same Python process is evident, given that it involves setting and managing implicit hidden states via cudaSetDevice and cudaGetDevice. Attempting to read or copy a memory region without a correct device context leads to cryptic errors or outright crashes. Our experience with managing device ordinal in the XGBoost project is not positive, as @trivialfis and @RAMitchell can attest.
Proposal. Remove all device management code from nvForest. nvForest should use the currently active GPU device and should not attempt to activate another GPU device.
To my best knowledge, NVIDIA cuML uses this approach. Users are expected to use cuda.core.Device explicitly before invoking cuML, if they wish to use a non-default GPU device.
TODO.
Do scoping work to estimate the engineering effort required.
Activity
[-][ENH] Refactor to remove handling of GPU device ordinals[/-][+][ENH] Remove handling of GPU device ordinals[/+]