[gpu] strict driver and cuda version assignment - #1275

Merged

cjac merged 112 commits into

GoogleCloudDataproc:masterfrom

LLC-Technologies-Collier:gpu-20241212

Feb 6, 2025

Commits on Jan 22, 2025

[gpu] toward a more consistent driver and CUDA install
cjac
committed
c0ea631
correcting driver for cuda 12.4
cjac
committed
f210adf
correcting cuda subversion. 12.4.0 instead of 12.4.1 so that driver and cuda match up
cjac
committed
f6ff5a3
corrected cannonical 11.8 driver version ; removed extra code and comment ; added better description of what is in the runfile
cjac
committed
e36b25b
skipping most tests ; using 11.7 from the cuda 11 line instead of the less well supported 11.8
cjac
committed
a2400a7
verified that the cuda and driver versions match up
cjac
committed
a137719
reducing log capture
cjac
committed
693bc7f
temporarily increasing machine shape for build caching
cjac
committed
4ce1efc
64 is too many for a single T4
cjac
committed
05b3e2b
added a subversion for 11.7
cjac
committed
e2ab509
add more tests to the install function
cjac
committed
1a39be6
only including architectures supported by this version of CUDA
cjac
committed
41ae069
pinning down versions better ; more caching ; more ram disks ; new pytorch and tensorflow test functions
cjac
committed
39ac281
using maximum from 8.9 series on rocky for 11.7
cjac
committed
f116717
skip full build
cjac
committed
976f869
pinning to bazel-7.4.0
cjac
committed
6ef2fdb
NCCL requires gcc-11 for cuda11
cjac
committed
1539cdb
rocky8 is now building from the source in the .run file
cjac
committed
9a54f4c
reverting to previous state of only selecting a compiler version on latest releases
cjac
committed
3316518
replaced literal path names with variable values ; indexing builds by the signing key used
cjac
committed
722e436
moved variable definition to prepare function ; moved driver signing to build phase
cjac
committed
f42fee6
test whether variable is defined before checking its value
cjac
committed
a13122e
cache only the bins and logs
cjac
committed
3b72048
build index of kernel modules after unpacking ; remove call to non-existent function
cjac
committed
2cc19ce
only build module dependency index once
cjac
committed
5a2d783
skipping CUDA 11 NCCL build on debian12
cjac
committed
1cf12ab
skip cuda11 on debian12, rocky9
cjac
committed
77a95ff
renamed verify_pyspark to verify_instance_pyspark
cjac
committed
0b2da14
failing somewhat gracefully ; skipping tests that would fail
cjac
committed
0c1df7f
skipping single node tests for rocky8
cjac
committed
ce60b03
re-enable other tests
cjac
committed
d16e625
Specifying bazel version with variable
cjac
committed
7284ad7
fixing up some skip logic
cjac
committed
35e4ba2
replaced OS_NAME with _shortname
cjac
committed
be37569
skip more single instance tests for rocky8
cjac
committed
c9d1d95
fixing indentation ; skipping redundant test
cjac
committed
b63ae17
remove retries of flakey tests
cjac
committed
94c1f13
oops ; need to define the cuda version to test for
cjac
committed
ac477b3
passing -q to gcloud to generate empty passphrase if no ssh key exists ; selecting a more modern version of the 550 driver
cjac
committed
db7aacf
including instructions on how to create a secure-boot key pair
cjac
committed
e152fd8
-e for expert, not -p for pro
cjac
committed
f113ef8
updated 11.8 and 12.0 driver versions
cjac
committed
dfc433d
added a signature check test which allows granular selection of platform to test, but does not yet verify signatures
cjac
committed
77fc42a
tuning the layout of arguments to userspace.run
cjac
committed
8ed498e
scoping DEFAULT_CUDA_VERSION correctly ; exercising rocky including kerberos on 12.6
cjac
committed
842d7e5
add a connect timeout to the ssh call instead of trying to patch around a longer than expected connection delay
cjac
committed
bb35d11
add some entropy to the process
cjac
committed
2541a6f
perhaps a re-run would have fixed 2.0-rocky8 on that last run
cjac
committed
ab668ff
increasing init action timeout to account for uncached builds
cjac
committed
934289a
cache non-open kernel build results
cjac
committed
e5920f8
per-kernel sub-directory for kmod tarballs
cjac
committed
386177d
using upstream repo and branch
cjac
committed
b9668e0
corrected grammar error
cjac
committed
2f0148a
testing Kerberos some more
cjac
committed
19b9ddb
better implementation of numa node selection
cjac
committed
1e5fc0f
this time with a test which is exercised
cjac
committed
4023031
skip debian11 on Kerberos
cjac
committed
03f59a6
also skipping 2.1-ubuntu20 on kerberos clusters
cjac
committed
f2146e3
re-adjusting tests to be performed ; adjusting rather than skipping known failure cases
cjac
committed
1cb99f8
more temporal variance
cjac
committed
3a238d1
skipping CUDA=12.0 for ubuntu22
cjac
committed
cc16aa8
kerberos not known to succeed on 2.0-rocky8
cjac
committed
3ac04bc
2.2 dataproc images do not support CUDA <= 12.0
cjac
committed
c6bf91a
skipping SINGLE configuration for rocky8 again
cjac
committed
d1b3d48
not testing 2.0
cjac
committed
751e7a0
trying without test retries ; retries should happen within the test, not by re-running the test
cjac
committed
e5e3a9e
kerberos only works on 2.2
cjac
committed
c1cd1d9
using expectedFailure instead of skipTest for tests which are known to fail
cjac
committed
eac2d46
document one of the failure states
cjac
committed
bf1f0c6
skipping expected failures
cjac
committed
12e6de9
updated manual-test-runner.sh instructions
cjac
committed
f7bf9ab
this one generated from template after refactor
cjac
committed
47a6e3b
do not point to local rpm pgp key
cjac
committed
26719af
re-ordering to reduce delta from master
cjac
committed
74c09f4
custom image usage can come later
cjac
committed
53c1ef1
see #1283
cjac
committed
97046b1
replaced incorrectly removed presubmit.sh and removed custom image key creation script intended to be removed in 70f37b638e8309a669625844034946fc1b51037a
cjac
committed
484308b
revert nearly to master
cjac
committed
61b94da
can include extended test suite later
cjac
committed
8b4f4f8
order commands correctly
cjac
committed
3bc45ff
placing all completion files in a common directory
cjac
committed
6a76b4e
extend supported version list to include latest release of each minor version and their associated driver
cjac
committed
e592146
tested with CUDA 11.6.2/510.108.03
cjac
committed
4559ecc
exercised with cuda 11.1
cjac
committed
16c8485
reverting cloudbuild/Dockerfile to master
cjac
committed
afd5f2f
nvidia is 404ing for download.nvidia.com ; using us.download.nvidia.com
cjac
committed
2272f97
skipping rocky9
cjac
committed
3b2dc66
* adding version 12.6 to the support matrix
cjac
committed
0c420b7
incorrect version check removed
cjac
committed
f69d071
only install pytorch if include-pytorch metadata set to true
cjac
committed
73ffce5
since call to install_pytorch is protected by metadata check, skip metadata check within the function ; create new function harden_sshd_config and call it
cjac
committed
521df62
increasing timeout and machine shape to reduce no-cache build time
cjac
committed
c0b60b2
skip full test run due to edits to integration_tests directory
cjac
committed
30c97c4
ubuntu18 does not know about kex-gss ; use correct driver version number for cuda 11.1.1 url generation
cjac
committed
84b1fb9
on rocky9 sshd service is called sshd instead of ssh as the rest of the platforms call it
cjac
committed
11cbe95
kex-gss is new in debian11
cjac
committed
56fe50c
all rocky call it sshd it seems
cjac
committed
b1cd1d0
cudnn no longer available on debian10
cjac
committed
ca94393

Commits on Jan 23, 2025

compared with #1282 ; this change matches parity more closely
cjac
committed
1d2166c
slightly better variable declaration ordering ; it is better still in the templates/ directory from #1282
cjac
committed
50142f6
install spark rapids
cjac
committed
6363203
cache the results of nvidia-smi --query-gpu
cjac
committed
dba00df
reduce development time
cjac
committed
96a8d6d
exercising more CUDA variants ; testing whether tests fail on long runs
cjac
committed
11f099c
try to reduce concurrent builds ; extend build time further ; only enable spark rapids on images >= 2.1
cjac
committed
8ae2c0a

Commits on Jan 24, 2025

Commits on Jan 28, 2025

configure for use with JupyterLab
cjac
committed
8936442

Commits on Jan 29, 2025

2.2 should use 12.6.3 (latest)
cjac
committed
0bc3c1f

Commits on Feb 4, 2025

Addressing review from cnauroth
cjac
committed
e56ddd0

Commits on Feb 6, 2025

reverting changes to presubmit.sh
cjac
committed
3384a4d