Error installing Nvidia Drivers during spark-rapids.sh initialization on Dataproc 2.1-rocky8 cluster

[SurajAralihalli]

ERROR:
modprobe: FATAL: Module nvidia not found in directory /lib/modules/4.18.0-553.63.1.el8_10.x86_64

ENV: Dataproc 2.1-rocky8

Steps:

export REGION=us-central1
export GCS_BUCKET=<bucket>
export CLUSTER_NAME=<cluster>
export NUM_GPUS=1
export NUM_WORKERS=2

gcloud dataproc clusters create $CLUSTER_NAME \
  --region=$REGION \
  --zone="" \
  --image-version=2.1-rocky8 \
  --master-machine-type=n1-standard-16 \
  --num-workers=$NUM_WORKERS \
  --worker-accelerator=type=nvidia-l4,count=$NUM_GPUS \
  --initialization-actions=gs://goog-dataproc-initialization-actions-${REGION}/spark-rapids/spark-rapids.sh \
  --worker-machine-type=g2-standard-12 \
  --temp-bucket=$GCS_BUCKET \
  --bucket=$GCS_BUCKET \
  --enable-component-gateway \
  --no-shielded-secure-boot \
  --subnet=default

[cjac]

Thanks for the report, Suraj!

Can you please attach the init action log to case 63085895 ? I will run the build as soon as I can, but that will not likely be before you are able to attach the log.

C.J.

[SurajAralihalli]

++ mktemp -u -d -p /run/tmp -t ca_dir-XXXX
+ CA_TMPDIR=/run/tmp/ca_dir-LkBV
++ get_metadata_attribute private_secret_name
++ local -r attribute_name=private_secret_name
++ local -r default_value=
++ /usr/share/google/get_metadata_value attributes/private_secret_name
++ echo -n ''
+ PSN=
+ readonly PSN
+ is_rocky
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \r\o\c\k\y ]]
++ os_id
++ sed -e s/rocky/rhel/
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
++ os_vercat
++ is_ubuntu
+++ os_id
+++ grep '^ID=' /etc/os-release
+++ cut -d= -f2
+++ xargs
++ [[ rocky == \u\b\u\n\t\u ]]
++ is_rocky
+++ os_id
+++ grep '^ID=' /etc/os-release
+++ cut -d= -f2
+++ xargs
++ [[ rocky == \r\o\c\k\y ]]
++ os_version
++ sed -e 's/[^0-9].*$//g'
++ grep '^VERSION_ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ shortname=rhel8
+ readonly shortname
+ test -v DATAPROC_IMAGE_VERSION
++ lsb_release -is
++ tr '[:upper:]' '[:lower:]'
+ readonly OS_NAME=rocky
+ OS_NAME=rocky
++ spark-submit --version
++ sed -n 's/.*version[[:blank:]]\+\([0-9]\+\.[0-9]\).*/\1/p'
++ head -n1
+ readonly SPARK_VERSION_ENV=3.3
+ SPARK_VERSION_ENV=3.3
+ [[ 3.3 == \3* ]]
+ readonly DEFAULT_XGBOOST_VERSION=1.7.6
+ DEFAULT_XGBOOST_VERSION=1.7.6
+ readonly SPARK_VERSION=3.0
+ SPARK_VERSION=3.0
+ readonly DEFAULT_SPARK_RAPIDS_VERSION=25.06.0
+ DEFAULT_SPARK_RAPIDS_VERSION=25.06.0
++ get_metadata_attribute spark-rapids-version 25.06.0
++ local -r attribute_name=spark-rapids-version
++ local -r default_value=25.06.0
++ /usr/share/google/get_metadata_value attributes/spark-rapids-version
+ readonly SPARK_RAPIDS_VERSION=25.08.0
+ SPARK_RAPIDS_VERSION=25.08.0
++ get_metadata_attribute xgboost-version 1.7.6
++ local -r attribute_name=xgboost-version
++ local -r default_value=1.7.6
++ /usr/share/google/get_metadata_value attributes/xgboost-version
++ echo -n 1.7.6
+ readonly XGBOOST_VERSION=1.7.6
+ XGBOOST_VERSION=1.7.6
++ /usr/share/google/get_metadata_value attributes/dataproc-role
+ readonly ROLE=Worker
+ ROLE=Worker
++ /usr/share/google/get_metadata_value attributes/dataproc-master
+ readonly MASTER=spark-qa-ci-test202508180714-21-rocky8-t4-m
+ MASTER=spark-qa-ci-test202508180714-21-rocky8-t4-m
++ get_metadata_attribute rapids-runtime SPARK
++ local -r attribute_name=rapids-runtime
++ local -r default_value=SPARK
++ /usr/share/google/get_metadata_value attributes/rapids-runtime
+ readonly RUNTIME=SPARK
+ RUNTIME=SPARK
++ get_metadata_attribute cuda-version 12.4.1
++ local -r attribute_name=cuda-version
++ local -r default_value=12.4.1
++ /usr/share/google/get_metadata_value attributes/cuda-version
++ echo -n 12.4.1
+ CUDA_VERSION=12.4.1
++ get_metadata_attribute driver-version 550.54.15
++ local -r attribute_name=driver-version
++ local -r default_value=550.54.15
++ /usr/share/google/get_metadata_value attributes/driver-version
++ echo -n 550.54.15
+ NVIDIA_DRIVER_VERSION=550.54.15
+ CUDA_VERSION_MAJOR=12.4
+ [[ rocky == \u\b\u\n\t\u ]]
+ SECURE_BOOT=disabled
++ mokutil --sb-state
++ awk '{print $2}'
+ SECURE_BOOT=disabled
++ get_metadata_attribute install-gpu-agent false
++ local -r attribute_name=install-gpu-agent
++ local -r default_value=false
++ /usr/share/google/get_metadata_value attributes/install-gpu-agent
++ echo -n false
+ INSTALL_GPU_AGENT=false
+ readonly INSTALL_GPU_AGENT
+ readonly HADOOP_CONF_DIR=/etc/hadoop/conf
+ HADOOP_CONF_DIR=/etc/hadoop/conf
+ readonly HIVE_CONF_DIR=/etc/hive/conf
+ HIVE_CONF_DIR=/etc/hive/conf
+ readonly SPARK_CONF_DIR=/etc/spark/conf
+ SPARK_CONF_DIR=/etc/spark/conf
+ NVIDIA_SMI_PATH=/usr/bin
+ MIG_MAJOR_CAPS=0
+ IS_MIG_ENABLED=0
+ readonly NVIDIA_BASE_DL_URL=https://developer.download.nvidia.com/compute
+ NVIDIA_BASE_DL_URL=https://developer.download.nvidia.com/compute
+ readonly NVIDIA_REPO_URL=https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64
+ NVIDIA_REPO_URL=https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64
+ main
+ is_debian
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \d\e\b\i\a\n ]]
+ check_os_and_secure_boot
+ is_debian
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \d\e\b\i\a\n ]]
+ is_ubuntu
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \u\b\u\n\t\u ]]
+ is_rocky
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \r\o\c\k\y ]]
+ is_rocky8
+ is_rocky
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \r\o\c\k\y ]]
++ os_version
++ grep '^VERSION_ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ 8.10 == \8* ]]
+ [[ disabled == \e\n\a\b\l\e\d ]]
+ [[ disabled == \e\n\a\b\l\e\d ]]
+ setup_gpu_yarn
+ [[ rocky == debian ]]
+ [[ rocky == ubuntu ]]
+ [[ rocky == rocky ]]
+ execute_with_retries 'dnf -y -q install pciutils'
+ local -r 'cmd=dnf -y -q install pciutils'
+ (( i = 0 ))
+ (( i < 10 ))
+ eval 'dnf -y -q install pciutils'
++ dnf -y -q install pciutils

Installed:
  pciutils-3.7.0-3.el8.x86_64                                                   


real	0m3.342s
user	0m2.583s
sys	0m0.365s
+ return 0
+ configure_yarn
+ [[ ! -f /etc/hadoop/conf/resource-types.xml ]]
+ printf '<?xml version="1.0" ?>\n<configuration/>'
+ set_hadoop_property resource-types.xml yarn.resource-types yarn.io/gpu
+ local -r config_file=resource-types.xml
+ local -r property=yarn.resource-types
+ local -r value=yarn.io/gpu
+ /usr/local/bin/bdconfig set_property --configuration_file /etc/hadoop/conf/resource-types.xml --name yarn.resource-types --value yarn.io/gpu --clobber
+ set_hadoop_property capacity-scheduler.xml yarn.scheduler.capacity.resource-calculator org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
+ local -r config_file=capacity-scheduler.xml
+ local -r property=yarn.scheduler.capacity.resource-calculator
+ local -r value=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
+ /usr/local/bin/bdconfig set_property --configuration_file /etc/hadoop/conf/capacity-scheduler.xml --name yarn.scheduler.capacity.resource-calculator --value org.apache.hadoop.yarn.util.resource.DominantResourceCalculator --clobber
+ set_hadoop_property yarn-site.xml yarn.resource-types yarn.io/gpu
+ local -r config_file=yarn-site.xml
+ local -r property=yarn.resource-types
+ local -r value=yarn.io/gpu
+ /usr/local/bin/bdconfig set_property --configuration_file /etc/hadoop/conf/yarn-site.xml --name yarn.resource-types --value yarn.io/gpu --clobber
+ lspci
+ grep -q NVIDIA
+ nv_smi=/usr/bin/nvidia-smi
+ test -f /usr/bin/nvidia-smi
+ is_debian
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \d\e\b\i\a\n ]]
+ is_ubuntu
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \u\b\u\n\t\u ]]
+ is_rocky
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \r\o\c\k\y ]]
+ echo 'kernel devel and headers not required on rocky.  installing from binary'
kernel devel and headers not required on rocky.  installing from binary
+ [[ 0 -eq 0 ]]
+ install_nvidia_gpu_driver
+ readonly NVIDIA_DRIVER_VERSION_PREFIX=550
+ NVIDIA_DRIVER_VERSION_PREFIX=550
+ readonly LOCAL_INSTALLER_DEB=cuda-repo-rhel8-12-4-local_12.4.1-550.54.15-1_amd64.deb
+ LOCAL_INSTALLER_DEB=cuda-repo-rhel8-12-4-local_12.4.1-550.54.15-1_amd64.deb
+ readonly LOCAL_DEB_URL=https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda-repo-rhel8-12-4-local_12.4.1-550.54.15-1_amd64.deb
+ LOCAL_DEB_URL=https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda-repo-rhel8-12-4-local_12.4.1-550.54.15-1_amd64.deb
+ readonly DIST_KEYRING_DIR=/var/cuda-repo-rhel8-12-4-local
+ DIST_KEYRING_DIR=/var/cuda-repo-rhel8-12-4-local
+ is_debian
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \d\e\b\i\a\n ]]
+ is_ubuntu
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \u\b\u\n\t\u ]]
+ is_rocky
++ os_id
++ grep '^ID=' /etc/os-release
++ cut -d= -f2
++ xargs
+ [[ rocky == \r\o\c\k\y ]]
+ execute_with_retries 'dnf -y -q update --exclude=systemd*,kernel*'
+ local -r 'cmd=dnf -y -q update --exclude=systemd*,kernel*'
+ (( i = 0 ))
+ (( i < 10 ))
+ eval 'dnf -y -q update --exclude=systemd*,kernel*'
++ dnf -y -q update '--exclude=systemd*,kernel*'

Upgraded:
  epel-release-8-22.el8.noarch                                                  
  glibc-2.28-251.el8_10.25.x86_64                                               
  glibc-common-2.28-251.el8_10.25.x86_64                                        
  glibc-devel-2.28-251.el8_10.25.x86_64                                         
  glibc-gconv-extra-2.28-251.el8_10.25.x86_64                                   
  glibc-headers-2.28-251.el8_10.25.x86_64                                       
  glibc-langpack-en-2.28-251.el8_10.25.x86_64                                   
  google-cloud-cli-534.0.0-1.x86_64                                             
  google-cloud-cli-anthoscli-534.0.0-1.x86_64                                   
  google-osconfig-agent-1:20250815.00-g1.el8.x86_64                             
  libxml2-2.9.7-21.el8_10.3.x86_64                                              
  python3-perf-4.18.0-553.66.1.el8_10.x86_64                                    
  python3-requests-2.20.0-6.el8_10.noarch                                       


real	1m9.978s
user	0m52.606s
sys	0m7.880s
+ return 0
+ execute_with_retries 'dnf -y -q install pciutils kernel-devel gcc'
+ local -r 'cmd=dnf -y -q install pciutils kernel-devel gcc'
+ (( i = 0 ))
+ (( i < 10 ))
+ eval 'dnf -y -q install pciutils kernel-devel gcc'
++ dnf -y -q install pciutils kernel-devel gcc

Installed:
  bison-3.0.4-10.el8.x86_64      elfutils-libelf-devel-0.190-2.el8.x86_64       
  flex-2.6.1-9.el8.x86_64        kernel-devel-4.18.0-553.66.1.el8_10.x86_64     


real	0m28.304s
user	0m18.037s
sys	0m8.863s
+ return 0
+ readonly NVIDIA_ROCKY_REPO_URL=https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
+ NVIDIA_ROCKY_REPO_URL=https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
+ execute_with_retries 'dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo'
+ local -r 'cmd=dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo'
+ (( i = 0 ))
+ (( i < 10 ))
+ eval 'dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo'
++ dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
Adding repo from: https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

real	0m0.361s
user	0m0.241s
sys	0m0.049s
+ return 0
+ execute_with_retries 'dnf clean all'
+ local -r 'cmd=dnf clean all'
+ (( i = 0 ))
+ (( i < 10 ))
+ eval 'dnf clean all'
++ dnf clean all
99 files removed

real	0m0.309s
user	0m0.231s
sys	0m0.075s
+ return 0
+ configure_dkms_certs
+ [[ -z '' ]]
+ echo 'No signing secret provided.  skipping'
No signing secret provided.  skipping
+ return 0
+ execute_with_retries 'dnf -y -q module install nvidia-driver:latest-dkms'
+ local -r 'cmd=dnf -y -q module install nvidia-driver:latest-dkms'
+ (( i = 0 ))
+ (( i < 10 ))
+ eval 'dnf -y -q module install nvidia-driver:latest-dkms'
++ dnf -y -q module install nvidia-driver:latest-dkms
Importing GPG key 0xD42D0685:
 Userid     : "cudatools <cudatools@nvidia.com>"
 Fingerprint: 610C 7B14 E068 A878 070D A4E9 9CD0 A493 D42D 0685
 From       : https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/D42D0685.pub

Installed:
  abattis-cantarell-fonts-0.0.25-6.el8.noarch                                   
  adwaita-cursor-theme-3.28.0-3.el8.noarch                                      
  adwaita-icon-theme-3.28.0-3.el8.noarch                                        
  at-spi2-atk-2.26.2-1.el8.x86_64                                               
  at-spi2-core-2.28.0-1.el8.x86_64                                              
  atk-2.28.1-1.el8.x86_64                                                       
  cairo-gobject-1.15.12-6.el8.x86_64                                            
  colord-libs-1.4.2-1.el8.x86_64                                                
  dconf-0.28.0-4.el8.x86_64                                                     
  dejavu-sans-mono-fonts-2.35-7.el8.noarch                                      
  dkms-3.2.1-1.el8.noarch                                                       
  egl-gbm-2:1.1.2.1-1.el8.x86_64                                                
  egl-wayland-1.1.19-2.el8.x86_64                                               
  egl-x11-1.0.2-1.el8.x86_64                                                    
  fribidi-1.0.4-9.el8.x86_64                                                    
  gdk-pixbuf2-2.36.12-7.el8_10.x86_64                                           
  gdk-pixbuf2-modules-2.36.12-7.el8_10.x86_64                                   
  glib-networking-2.56.1-1.1.el8.x86_64                                         
  gsettings-desktop-schemas-3.32.0-6.el8.x86_64                                 
  gtk-update-icon-cache-3.22.30-12.el8_10.x86_64                                
  gtk3-3.22.30-12.el8_10.x86_64                                                 
  hicolor-icon-theme-0.17-2.el8.noarch                                          
  jasper-libs-2.0.14-6.el8_10.x86_64                                            
  kmod-nvidia-latest-dkms-3:580.65.06-1.el8.x86_64                              
  libX11-devel-1.6.8-9.el8_10.x86_64                                            
  libX11-xcb-1.6.8-9.el8_10.x86_64                                              
  libXau-devel-1.0.9-3.el8.x86_64                                               
  libXcomposite-0.4.4-14.el8.x86_64                                             
  libXdamage-1.1.4-14.el8.x86_64                                                
  libXdmcp-1.1.3-1.el8.x86_64                                                   
  libXfont2-2.0.3-2.el8.x86_64                                                  
  libdatrie-0.2.9-7.el8.x86_64                                                  
  libdrm-2.4.115-2.el8.x86_64                                                   
  libepoxy-1.5.8-1.el8.x86_64                                                   
  libevdev-1.10.0-1.el8.x86_64                                                  
  libglvnd-1:1.3.4-2.el8.x86_64                                                 
  libglvnd-egl-1:1.3.4-2.el8.x86_64                                             
  libglvnd-gles-1:1.3.4-2.el8.x86_64                                            
  libglvnd-glx-1:1.3.4-2.el8.x86_64                                             
  libglvnd-opengl-1:1.3.4-2.el8.x86_64                                          
  libinput-1.16.3-3.el8_6.x86_64                                                
  libmodman-2.0.1-17.el8.x86_64                                                 
  libnvidia-cfg-3:580.65.06-1.el8.x86_64                                        
  libnvidia-fbc-3:580.65.06-1.el8.x86_64                                        
  libnvidia-gpucomp-3:580.65.06-1.el8.x86_64                                    
  libnvidia-ml-3:580.65.06-1.el8.x86_64                                         
  libpciaccess-0.14-1.el8.x86_64                                                
  libproxy-0.4.15-5.5.el8_10.x86_64                                             
  libsoup-2.62.3-9.el8_10.x86_64                                                
  libthai-0.1.27-2.el8.x86_64                                                   
  libvdpau-1.5-11.el8.x86_64                                                    
  libwacom-1.6-3.el8.x86_64                                                     
  libwacom-data-1.6-3.el8.noarch                                                
  libwayland-client-1.21.0-1.el8.x86_64                                         
  libwayland-cursor-1.21.0-1.el8.x86_64                                         
  libwayland-egl-1.21.0-1.el8.x86_64                                            
  libwayland-server-1.21.0-1.el8.x86_64                                         
  libxcb-devel-1.13.1-1.el8.x86_64                                              
  libxkbfile-1.1.0-1.el8.x86_64                                                 
  libxshmfence-1.3-2.el8.x86_64                                                 
  llvm-compat-libs-17.0.6-3.module+el8.10.0+1875+4f0b06db.x86_64                
  mesa-libEGL-23.1.4-4.el8_10.x86_64                                            
  mesa-libGL-23.1.4-4.el8_10.x86_64                                             
  mesa-libgbm-23.1.4-4.el8_10.x86_64                                            
  mesa-libglapi-23.1.4-4.el8_10.x86_64                                          
  mesa-vulkan-drivers-23.1.4-4.el8_10.x86_64                                    
  mtdev-1.1.5-12.el8.x86_64                                                     
  nvidia-driver-3:580.65.06-1.el8.x86_64                                        
  nvidia-driver-cuda-3:580.65.06-1.el8.x86_64                                   
  nvidia-driver-cuda-libs-3:580.65.06-1.el8.x86_64                              
  nvidia-driver-libs-3:580.65.06-1.el8.x86_64                                   
  nvidia-kmod-common-3:580.65.06-1.el8.noarch                                   
  nvidia-libXNVCtrl-3:580.65.06-1.el8.x86_64                                    
  nvidia-libXNVCtrl-devel-3:580.65.06-1.el8.x86_64                              
  nvidia-modprobe-3:580.65.06-1.el8.x86_64                                      
  nvidia-persistenced-3:580.65.06-1.el8.x86_64                                  
  nvidia-settings-3:580.65.06-1.el8.x86_64                                      
  nvidia-xconfig-3:580.65.06-1.el8.x86_64                                       
  ocl-icd-2.2.12-1.el8.x86_64                                                   
  opencl-filesystem-1.0-6.el8.noarch                                            
  pango-1.42.4-8.el8.x86_64                                                     
  rest-0.8.1-2.el8.x86_64                                                       
  vulkan-loader-1.3.283.0-1.el8_10.x86_64                                       
  xorg-x11-drv-fbdev-0.5.0-2.el8.x86_64                                         
  xorg-x11-drv-libinput-0.29.0-1.el8.x86_64                                     
  xorg-x11-drv-vesa-2.4.0-3.el8.x86_64                                          
  xorg-x11-nvidia-3:580.65.06-1.el8.x86_64                                      
  xorg-x11-proto-devel-2020.1-3.el8.noarch                                      
  xorg-x11-server-Xorg-1.20.11-26.el8_10.x86_64                                 
  xorg-x11-server-common-1.20.11-26.el8_10.x86_64                               
  xorg-x11-xkb-utils-7.7-28.el8.x86_64                                          


real	2m31.817s
user	2m2.292s
sys	0m6.596s
+ return 0
+ clear_dkms_key
+ [[ -z '' ]]
+ echo 'No signing secret provided.  skipping'
+ return 0
+ execute_with_retries 'dnf -y -q install cuda-toolkit'
+ local -r 'cmd=dnf -y -q install cuda-toolkit'
+ (( i = 0 ))
+ (( i < 10 ))
+ eval 'dnf -y -q install cuda-toolkit'
++ dnf -y -q install cuda-toolkit

Installed:
  cuda-cccl-13-0-13.0.50-1.x86_64                                               
  cuda-command-line-tools-13-0-13.0.0-1.x86_64                                  
  cuda-compiler-13-0-13.0.0-1.x86_64                                            
  cuda-crt-13-0-13.0.48-1.x86_64                                                
  cuda-cudart-13-0-13.0.48-1.x86_64                                             
  cuda-cudart-devel-13-0-13.0.48-1.x86_64                                       
  cuda-culibos-devel-13-0-13.0.39-1.x86_64                                      
  cuda-cuobjdump-13-0-13.0.39-1.x86_64                                          
  cuda-cupti-13-0-13.0.48-1.x86_64                                              
  cuda-cuxxfilt-13-0-13.0.39-1.x86_64                                           
  cuda-documentation-13-0-13.0.39-1.x86_64                                      
  cuda-driver-devel-13-0-13.0.48-1.x86_64                                       
  cuda-gdb-13-0-13.0.39-1.x86_64                                                
  cuda-libraries-13-0-13.0.0-1.x86_64                                           
  cuda-libraries-devel-13-0-13.0.0-1.x86_64                                     
  cuda-nsight-13-0-13.0.39-1.x86_64                                             
  cuda-nsight-compute-13-0-13.0.0-1.x86_64                                      
  cuda-nsight-systems-13-0-13.0.0-1.x86_64                                      
  cuda-nvcc-13-0-13.0.48-1.x86_64                                               
  cuda-nvdisasm-13-0-13.0.39-1.x86_64                                           
  cuda-nvml-devel-13-0-13.0.39-1.x86_64                                         
  cuda-nvprune-13-0-13.0.39-1.x86_64                                            
  cuda-nvrtc-13-0-13.0.48-1.x86_64                                              
  cuda-nvrtc-devel-13-0-13.0.48-1.x86_64                                        
  cuda-nvtx-13-0-13.0.39-1.x86_64                                               
  cuda-opencl-13-0-13.0.39-1.x86_64                                             
  cuda-opencl-devel-13-0-13.0.39-1.x86_64                                       
  cuda-profiler-api-13-0-13.0.39-1.x86_64                                       
  cuda-sandbox-devel-13-0-13.0.39-1.x86_64                                      
  cuda-sanitizer-13-0-13.0.48-1.x86_64                                          
  cuda-toolkit-13.0.0-1.x86_64                                                  
  cuda-toolkit-13-0-13.0.0-1.x86_64                                             
  cuda-toolkit-13-0-config-common-13.0.48-1.noarch                              
  cuda-toolkit-13-config-common-13.0.48-1.noarch                                
  cuda-toolkit-config-common-13.0.48-1.noarch                                   
  cuda-tools-13-0-13.0.0-1.x86_64                                               
  cuda-visual-tools-13-0-13.0.0-1.x86_64                                        
  gds-tools-13-0-1.15.0.42-1.x86_64                                             
  libcublas-13-0-13.0.0.19-1.x86_64                                             
  libcublas-devel-13-0-13.0.0.19-1.x86_64                                       
  libcufft-13-0-12.0.0.15-1.x86_64                                              
  libcufft-devel-13-0-12.0.0.15-1.x86_64                                        
  libcufile-13-0-1.15.0.42-1.x86_64                                             
  libcufile-devel-13-0-1.15.0.42-1.x86_64                                       
  libcurand-13-0-10.4.0.35-1.x86_64                                             
  libcurand-devel-13-0-10.4.0.35-1.x86_64                                       
  libcusolver-13-0-12.0.3.29-1.x86_64                                           
  libcusolver-devel-13-0-12.0.3.29-1.x86_64                                     
  libcusparse-13-0-12.6.2.49-1.x86_64                                           
  libcusparse-devel-13-0-12.6.2.49-1.x86_64                                     
  libnpp-13-0-13.0.0.50-1.x86_64                                                
  libnpp-devel-13-0-13.0.0.50-1.x86_64                                          
  libnvfatbin-13-0-13.0.39-1.x86_64                                             
  libnvfatbin-devel-13-0-13.0.39-1.x86_64                                       
  libnvjitlink-13-0-13.0.39-1.x86_64                                            
  libnvjitlink-devel-13-0-13.0.39-1.x86_64                                      
  libnvjpeg-13-0-13.0.0.40-1.x86_64                                             
  libnvjpeg-devel-13-0-13.0.0.40-1.x86_64                                       
  libnvptxcompiler-13-0-13.0.48-1.x86_64                                        
  libnvvm-13-0-13.0.48-1.x86_64                                                 
  libxkbcommon-x11-0.9.1-1.el8.x86_64                                           
  nsight-compute-2025.3.0-2025.3.0.19-1.x86_64                                  
  nsight-systems-2025.3.2-2025.3.2.367_253236224375v0-0.x86_64                  
  xcb-util-0.4.0-10.el8.x86_64                                                  
  xcb-util-image-0.4.0-9.el8.x86_64                                             
  xcb-util-keysyms-0.4.0-7.el8.x86_64                                           
  xcb-util-renderutil-0.3.9-10.el8.x86_64                                       
  xcb-util-wm-0.4.1-12.el8.x86_64                                               


real	3m22.079s
user	2m16.701s
sys	0m13.148s
+ return 0
+ modprobe nvidia
modprobe: FATAL: Module nvidia not found in directory /lib/modules/4.18.0-553.63.1.el8_10.x86_64

[cjac]

That seems strange a couple of times. Rocky 8 should install from the .run file, and I am pretty sure I removed pciutils from the dependency list. Has someone published an update to the script, maybe? I will try to look tomorrow.

[SurajAralihalli]

Thanks CJ, the init script used is spark-rapids.sh

[cjac]

ah, isn't spark-rapids.sh maintained by NV? I maintain gpu/install-gpu-driver.sh and they have diverged ; I have offered to merge them so that we don't each need to maintain our own copy. Have you tried gpu/install-gpu-driver.sh instead? It is my intention that it should be usable as a drop-in replacement.

@nvliyuan - can you help Suraj? I am quite busy with other projects and don't have the cycles to come up to speed on the new changes to your code just now.

[SurajAralihalli]

@cjac I've created PR1359 to address this issue.