cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

[carlos-havier]

Hi,

Very cool project!

I installed in clean Anaconda environments with the following configs:

• CUDA 12.1, Python 3.11
• CUDA 11.3, Python 3.9

And then running:

pip install -r requirements.txt

And getting from python -c "import torch; print(torch.cuda.is_available()); print(torch.version)":

True
2.2.2+cu121

In both, I run the test code at Huggingface (https://huggingface.co/yahoo-inc/photo-background-generation).

The step of the code to get the foreground mask:

fg_mask = remover.process(img, type='map') # default setting - transparent background

Gives the following error:

File ~/anaconda3/envs/pbg/lib/python3.9/site-packages/torch/nn/functional.py:2482, in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
2479 if training:
2480 _verify_batch_size(input.size())
-> 2482 return torch.batch_norm(
2483 input, weight, bias, running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled
2484 )

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

I discard a HW or library problem, given that I have many other environments running with CUDA 11.3, 11.8 and 12.1 for AI without issue.

Help figuring this out would be much appreciated

Thanks! :)

[WinXsp]

Is the appropriate version of CUDNN not installed.

Please refer to my installation command:

conda install pytorch torchvision torchaudio pytorch-cuda=12.4 cudnn=8.9.2.26=cuda12_0 -c pytorch-nightly -c nvidia -y --force-reinstall

[carlos-havier]

Indeed, your solution works. Thank you! :)