Skip to content

Support Data Center precompiled driver container for Arm (Ubuntu 24.04)#533

Open
shivakunv wants to merge 1 commit into
mainfrom
precompiled-arm-support
Open

Support Data Center precompiled driver container for Arm (Ubuntu 24.04)#533
shivakunv wants to merge 1 commit into
mainfrom
precompiled-arm-support

Conversation

@shivakunv

@shivakunv shivakunv commented Jan 6, 2026

Copy link
Copy Markdown
Contributor

Code Changes Summary:

  • Platform Support
    Added support for the ARM64 platform.
    AMD64 remains the default architecture.

  • Artifacts Update
    ARM64 build artifacts are now uploaded with the -arm64 suffix.

  • Instance Type and Region Mapping
    g4dn.xlarge:
    Architecture: AMD64
    Supported Region: us-west-1
    Used for AMD64 builds.

    g5g.xlarge:
    Architecture: ARM64
    Supported Region: us-west-2
    Used for ARM64 builds.

passed pipeline: https://github.com/NVIDIA/gpu-driver-container/actions/runs/22180871853

passed pipeline: https://github.com/NVIDIA/gpu-driver-container/actions/runs/22337833186

@shivakunv shivakunv changed the title Precompiled arm support Support Data Center precompiled driver container for Arm (Ubuntu 24.04) Jan 6, 2026
@shivakunv shivakunv force-pushed the precompiled-arm-support branch 2 times, most recently from 6405d48 to 574ce43 Compare January 14, 2026 17:22
@shivakunv shivakunv force-pushed the precompiled-arm-support branch 4 times, most recently from 20726a8 to 46aa0d1 Compare February 12, 2026 12:07
@shivakunv shivakunv force-pushed the precompiled-arm-support branch 3 times, most recently from c008150 to b684015 Compare February 19, 2026 13:11
@shivakunv shivakunv marked this pull request as ready for review February 19, 2026 13:12
Comment thread .github/workflows/precompiled.yaml Outdated
@shivakunv shivakunv self-assigned this Feb 19, 2026
Comment thread ubuntu24.04/precompiled/nvidia-driver Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds ARM64 (aarch64) platform support to the Ubuntu 24.04 precompiled driver container builds, while maintaining AMD64 as the default architecture. The changes enable multi-platform Docker builds and update the CI/CD pipeline to handle both architectures.

Changes:

  • Added ARM64 platform support for Ubuntu 24.04 precompiled driver containers with architecture-specific package handling
  • Updated CI workflow to build, test, and publish both AMD64 and ARM64 artifacts with platform-specific suffixes
  • Modified Holodeck test infrastructure to support ARM64 instances (g5g.xlarge in us-west-2) and Ubuntu 24.04 OS specification

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
ubuntu24.04/precompiled/nvidia-driver Added conditional installation of libnvidia-fbc1 package (AMD64 only)
ubuntu24.04/precompiled/local-repo.sh Added conditional downloads for ARM64-incompatible packages (linux-signatures-nvidia, libnvidia-fbc1)
ubuntu24.04/precompiled/Dockerfile Made i386 architecture and CUDA repository URLs conditional based on target architecture
tests/scripts/findkernelversion.sh Added optional PLATFORM_SUFFIX parameter for artifact matching and platform-specific manifest inspection
tests/scripts/ci-precompiled-helpers.sh Added PLATFORM_SUFFIX parameter support for kernel version testing
tests/holodeck_ubuntu24.04.yaml Removed file (merged into holodeck_ubuntu.yaml)
tests/holodeck_ubuntu.yaml Removed hardcoded ingressIpRanges and AMI, added OS specification support
multi-arch.mk Removed AMD64-only platform restriction for ubuntu24.04 builds
Makefile Added DOCKER_BUILD_PLATFORM_OPTIONS to base image build targets
.github/workflows/precompiled.yaml Added platform matrix dimension, platform-aware artifact naming, ARM64 e2e testing with appropriate instance types, and Holodeck version update

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/precompiled.yaml Outdated
Comment thread .github/workflows/precompiled.yaml Outdated
@shivakunv shivakunv force-pushed the precompiled-arm-support branch 3 times, most recently from ee1265d to 49429dd Compare February 21, 2026 08:03
@shivakunv shivakunv marked this pull request as draft February 23, 2026 15:33
@shivakunv shivakunv force-pushed the precompiled-arm-support branch from 32e68a1 to cdbfe9a Compare February 24, 2026 05:21
@shivakunv shivakunv marked this pull request as ready for review February 24, 2026 06:48
@shivakunv shivakunv force-pushed the precompiled-arm-support branch from cdbfe9a to e224399 Compare February 25, 2026 04:14
Comment thread .github/workflows/precompiled.yaml Outdated
Comment thread .github/workflows/precompiled.yaml Outdated
Comment thread .github/workflows/precompiled.yaml Outdated
@shivakunv shivakunv force-pushed the precompiled-arm-support branch from 2f00f8b to 4a75c51 Compare March 11, 2026 03:51
@shivakunv shivakunv force-pushed the precompiled-arm-support branch from 4a75c51 to c7ce51a Compare April 1, 2026 06:08
@shivakunv shivakunv force-pushed the precompiled-arm-support branch 2 times, most recently from 8b1afd4 to dd078b8 Compare April 9, 2026 06:50
Comment thread ubuntu24.04/precompiled/Dockerfile
Comment thread ubuntu24.04/precompiled/nvidia-driver Outdated
@shivakunv shivakunv force-pushed the precompiled-arm-support branch from dd078b8 to 783e783 Compare April 17, 2026 13:04
@shivakunv shivakunv force-pushed the precompiled-arm-support branch from 783e783 to d44324e Compare May 6, 2026 07:44
@shivakunv shivakunv requested review from rahulait and tariq1890 May 6, 2026 09:24
@shivakunv shivakunv force-pushed the precompiled-arm-support branch from d44324e to 7d8aff1 Compare May 8, 2026 05:08
@shivakunv shivakunv force-pushed the precompiled-arm-support branch from 7d8aff1 to c035d23 Compare May 20, 2026 03:47
@shivakunv shivakunv force-pushed the precompiled-arm-support branch 7 times, most recently from 34d0170 to 0627494 Compare June 1, 2026 13:25
Comment thread .github/workflows/precompiled.yaml Outdated
Comment thread base/Dockerfile Outdated
Comment thread .github/workflows/precompiled.yaml Outdated
Comment thread .github/workflows/precompiled.yaml Outdated
Comment thread ubuntu24.04/precompiled/nvidia-driver
@shivakunv shivakunv force-pushed the precompiled-arm-support branch from 2d7ac2e to 4ecbadb Compare June 1, 2026 16:09
Comment thread Makefile Outdated
Comment thread ubuntu24.04/precompiled/nvidia-driver Outdated
Comment thread ubuntu24.04/precompiled/Dockerfile
pattern: driver-images-*-${{ env.KERNEL_VERSION }}-${{ env.DIST }}*
path: ./tests/
merge-multiple: true
- name: Install skopeo

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need skopeo?

@shivakunv shivakunv Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 478 (Line: 356): Pushing the multi-arch oci-archive to the registry as a manifest list.
The github actions runner's docker load --platform exists in the cli but the daemon can not hold multi-arch images.
docker load + docker push would mean loop per platform, push each, then docker manifest create to stitch the manifest list.
skopeo copy streams oci-archive: registry preserving multi-arch in one step.

Line 379: Build outputs one multi-arch oci-archive (amd64 + arm64). The e2e test needs a single platform as docker-archive (for docker load).
On the github actions runner's docker save --platform does not help either, so without skopeo we would have to build amd64 and arm64 as separate single arch images. separate artifacts need to upload on github.
skopeo copy --override-arch extracts the platform we need.

consistency: same tool in both places.

From what I recall, tried regctl earlier and hit an oci-archive error. github pipeline logs have been cleared since three months have passed. can reinvestigate if preferred.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we would have to build amd64 and arm64 as separate single arch images. separate artifacts need to upload on github.

This is an approach we already follow in other repositories so I prefer this for the time being. If we want to use skopeo let's evaluate it first and then do it in a separate commit if we decide to move forward with skopeo

@shivakunv shivakunv Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback.

we already agreed to move away from building separate per platform images and stitching them together.

please check:
#533 (comment): On putting more thought into this, can we do a multiarch build of the precompiled image instead of building the arm64 and amd64 images separate and then stitching them together?
#533 (comment): As mentioned in my earlier comment, let's do a single multiarch build instead of building individual platform-specific images and then merging them together.

for other repo or non-precompiled driver container:
Single buildx run pushes directly to registry with --output=type=image,push=true
no artifacts upload/download for e2e.

for precompiled repo (before change) :

-> build single arch image ( docker save image ) -> upload image as artifacts -> e2e: download version specific image artifact tar -> e2e: rsync tar image to target -> e2e: run gpu-operator on target with this tar image -> on Success publish image ( docker load + docker push platform specific amd64)

After change:

-> build multi arch image -> upload multi arch image as artifacts -> e2e: download version specific multi arch image artifact tar -> e2e: extract target specific (platform specific amd64 or arm64 ) image with skopeo -> e2e: rsync tar image to target -> e2e: run gpu-operatoron target with this tar image -> on Success publish multi arch image with skopeo

skopeo evaluation status:
skopeo was evaluated against the nv_only docker registry. Also, e2e tests are passing.

@shivakunv shivakunv force-pushed the precompiled-arm-support branch 2 times, most recently from 6b58bd4 to b9bd20d Compare June 3, 2026 03:31
@shivakunv shivakunv removed their assignment Jun 9, 2026
@rajathagasthya

Copy link
Copy Markdown
Contributor

Let's get another approval and resolve all open threads before merging.

Comment on lines 429 to 435
apt-get install --no-install-recommends -y \
linux-signatures-nvidia-${KERNEL_VERSION} \
linux-modules-nvidia-${DRIVER_BRANCH}-server-open-${KERNEL_VERSION}
else
echo "Installing Closed NVIDIA driver kernel modules..."
apt-get install --no-install-recommends -y \
linux-objects-nvidia-${DRIVER_BRANCH}-server-${KERNEL_VERSION} \
linux-signatures-nvidia-${KERNEL_VERSION} \
linux-modules-nvidia-${DRIVER_BRANCH}-server-${KERNEL_VERSION}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can reduce the duplication here by conditionally installing only linux-objects-nvidia-${DRIVER_BRANCH}-server-${KERNEL_VERSION}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you clarify ?
what I understood : ( only for linux-objects-nvidia-${DRIVER_BRANCH}-server-${KERNEL_VERSION})

if [ "$KERNEL_TYPE" != "kernel-open" ]; then
    apt-get install --no-install-recommends -y \
        linux-objects-nvidia-${DRIVER_BRANCH}-server-${KERNEL_VERSION}
fi
if [ "$KERNEL_TYPE" = "kernel-open" ]; then
    apt-get install --no-install-recommends -y \
        linux-modules-nvidia-${DRIVER_BRANCH}-server-open-${KERNEL_VERSION}
else
    apt-get install --no-install-recommends -y \
        linux-modules-nvidia-${DRIVER_BRANCH}-server-${KERNEL_VERSION}
fi

is more complex .

Signed-off-by: Shiva Kumar (SW-CLOUD) <shivaku@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants