pcluster imagebuilder fails on Rocky 9.7 amis

[ezrizhu]

pcluster imagebuilder pulls the latest kmod for lustre client, which is only built for RHEL/Rocky9.8, which does not work on Rocky as Rocky 9.8 has yet to be released as of now.

https://github.com/aws/aws-parallelcluster-cookbook/blob/release-3.15/cookbooks/aws-parallelcluster-environment/resources/lustre/partial/_install_lustre_centos_redhat.rb

https://github.com/aws/aws-parallelcluster-cookbook/blob/release-3.15/cookbooks/aws-parallelcluster-environment/resources/lustre/lustre_rocky8.rb

$ dnf --disablerepo='*' --enablerepo=aws-fsx info kmod-lustre-client
Last metadata expiration check: 1 day, 7:24:13 ago on Sun May 24 21:47:19 2026.
Installed Packages
Name         : kmod-lustre-client
Version      : 2.15.6
Release      : 1.fsx30.el9
Architecture : x86_64
Size         : 16 M
Source       : lustre-client-2.15.6-1.fsx30.el9.src.rpm
Repository   : @System
From repo    : aws-fsx
Summary      : lustre-client kernel module(s)
URL          : https://wiki.whamcloud.com/
License      : GPL
Description  : This package provides the lustre-client kernel modules built for
             : the Linux kernel 5.14.0-611.49.1.el9_7.x86_64 for the x86_64
             : family of processors.

Available Packages
Name         : kmod-lustre-client
Version      : 2.15.6
Release      : 1.fsx31.el9
Architecture : x86_64
Size         : 2.7 M
Source       : lustre-client-2.15.6-1.fsx31.el9.src.rpm
Repository   : aws-fsx
Summary      : lustre-client kernel module(s)
URL          : https://wiki.whamcloud.com/
License      : GPL
Description  : This package provides the lustre-client kernel modules built for
             : the Linux kernel 5.14.0-687.5.3.el9_8.x86_64 for the x86_64
             : family of processors.

Resulting in below error

Stdout: ---- Begin output of ["dnf", "-y", "install", "kmod-lustre-client-0:2.15.6-1.fsx31.el9.x86_64", "lustre-client-0:2.15.6-1.fsx31.el9.x86_64"] ----
2026-05-25T10:20:08.289Z
Stdout: STDOUT: Last metadata expiration check: 0:00:20 ago on Mon 25 May 2026 10:19:11 AM UTC.
2026-05-25T10:20:08.289Z
Stdout: (try to add '--skip-broken' to skip uninstallable packages)
2026-05-25T10:20:08.289Z
Stdout: STDERR: Error:
2026-05-25T10:20:08.289Z
Stdout: Problem: conflicting requests
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(kmalloc_caches) = 0x4344f508 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(kmalloc_trace) = 0x544be8df needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(kthread_create_on_node) = 0x73ea27e3 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(module_layout) = 0x3075a628 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(pcpu_hot) = 0xd294630d needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(wake_up_process) = 0xa06cd0d4 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__alloc_pages) = 0x3f53e658 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__cpuhp_remove_state) = 0xe544a11f needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__cpuhp_setup_state) = 0xdaff2ec8 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__d_drop) = 0x46dd881a needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__ethtool_get_link_ksettings) = 0x6881b6c5 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__filemap_set_wb_err) = 0xf7de2cbd needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__folio_batch_release) = 0x2a6f7d57 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__folio_lock) = 0x8b781cb4 needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__folio_put) = 0x07ae575c needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
Stdout: - nothing provides kernel(__free_pages) = 0xa763ceef needed by kmod-lustre-client-2.15.6-1.fsx31.el9.x86_64 from aws-fsx
2026-05-25T10:20:08.289Z
<snip>

[nyetsche]

Rocky 9.8 is released: https://docs.rockylinux.org/10/release_notes/9_8/

ImageBuilder fails but for a slightly different reason now; the Chef cookbook creates a yum repo (aws-fsx) with the major release embedded in the URL

https://github.com/aws/aws-parallelcluster-cookbook/blob/develop/cookbooks/aws-parallelcluster-environment/resources/lustre/partial/_install_lustre_centos_redhat.rb#L19

sets base_url defined at

https://github.com/aws/aws-parallelcluster-cookbook/blob/develop/cookbooks/aws-parallelcluster-environment/resources/lustre/partial/_install_lustre_centos_redhat.rb#L21

defined as [...]/el/#{node['platform_version']}/$basearch

which ends up as on disk as:

baseurl=https://fsx-lustre-client-repo.s3.amazonaws.com/el/9.7/$basearch

But there is nothing (I've found) in https://fsx-lustre-client-repo.s3.amazonaws.com/el/9.8/

[ezrizhu]

I was also able to reproduce a failure, @nyetsche is this also what you saw?

2026-06-01T22:02:58.183Z
Stdout: == Installing EFA packages ==
2026-06-01T22:02:58.183Z
Stdout: dkms is already installed
2026-06-01T22:02:58.183Z
Stdout: infiniband-diags-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: libibumad-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: libibverbs-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: libibverbs-utils-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: librdmacm-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: librdmacm-utils-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: rdma-core-devel-61.0-2.el9 already installed
2026-06-01T22:02:58.183Z
Stdout: Alternative rdma-core is available
2026-06-01T22:02:58.183Z
Stdout: Last metadata expiration check: 0:19:49 ago on Mon Jun 1 21:42:30 2026.
2026-06-01T22:02:58.183Z
Stdout: No match for argument: infiniband-diags-compat
2026-06-01T22:02:58.183Z
Stdout: STDERR: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Error: Unable to find a match: kernel-devel-5.14.0-611.5.1.el9_7.x86_64
2026-06-01T22:02:58.183Z
Stdout: Error: Failed to install packages.
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Main config did not have a skip_missing_names_on_install attr. before setopt
2026-06-01T22:02:58.183Z
Stdout: Error: Unable to find a match: infiniband-diags-compat
2026-06-01T22:02:58.183Z
Stdout: Error: Failed to install packages.
2026-06-01T22:02:58.183Z
Stdout: Error: failed to install EFA packages, exiting
2026-06-01T22:02:58.183Z
Stdout: ---- End output of "bash" ----
2026-06-01T22:02:58.183Z
Stdout: Ran "bash" returned 1
2026-06-01T22:02:58.226Z
CmdExecution: Command execution has been completed
2026-06-01T22:02:58.226Z
CmdExecution: [ ERROR ] Command execution has resulted in an error
2026-06-01T22:02:58.226Z
CmdExecution: Stderr: + . /opt/parallelcluster/system_info ++ OS=rocky9 ++ PLATFORM=RHEL ++ RELEASE=rocky.9.8 ++ VERSION_ID=9.8 ++ COOKBOOK_URL=https://us-gov-west-1-aws-parallelcluster.s3.us-gov-west-1.amazonaws.com/parallelcluster/3.15.0/cookbooks/aws-parallelcluster-cookbook-3.15.0.tgz ++ CINC_URL=https://us-gov-west-1-aws-parallelcluster.s3.us-gov-west-1.amazonaws.com/archives/cinc/cinc-install-1.4.0.sh ++ COOKBOOK_NAME=aws-parallelcluster-cookbook-3.15.0 ++ KERNEL_PACKAGES_PREFIX=kernel + echo 'Calling chef-client with /etc/parallelcluster/image_dna.json' + cat /etc/parallelcluster/image_dna.json + cinc-client --local-mode --config /etc/chef/client.rb --log_level info --force-formatter --no-color --chef-zero-port 8889 --json-attributes /etc/parallelcluster/image_dna.json --override-runlist aws-parallelcluster-entrypoints::install
2026-06-01T22:02:58.226Z
CmdExecution: ExitCode 1
2026-06-01T22:02:58.226Z

[nyetsche]

Not quite, your error is a combination of failing to install (via dnf):

infiniband-diags-compat

and

kernel-devel-5.14.0-611.5.1.el9_7.x86_64

The infiniband-diags-compat issue I had to work around in a forked version of aws-parallelcluster (since we can't update to a newer version); during the build process something tries to do a dnf install of rdma-core, libibverbs, and so on, and infiniband-diags-compat is included in the list.

The Rocky repos provide most of those packages, but with a later revision - rdma-core-61.0-2, not rdma-core-61.0-1 (as in your output). Ex: https://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/Packages/r/rdma-core-61.0-2.el9.x86_64.rpm

There's no infiniband-diags-compat in the Rocky repos though, it's instead part of the aws-efa-installer tarball ( ex: https://efa-installer.amazonaws.com/aws-efa-installer-1.34.0.tar.gz but version varies across the pcluster version ).

In my branch, I hacked around by downloading https://efa-installer.amazonaws.com/aws-efa-installer-1.34.0.tar.gz and manually installing the RPM dnf install /tmp/aws-efa-installer/RPMS/ROCKYLINUX9/x86_64/rdma-core/infiniband-diags-compat-52.0-1.el9.x86_64.rpm since I'm not relying on these advanced RDMA/networking features anyway.

[nyetsche]

The second error (if fixing the infiniband-diags-compat doesn't end up skipping over it) is that Rocky doesn't keep older package versions in the default repos. So at https://dl.rockylinux.org/pub/rocky/9/AppStream/x86_64/os/Packages/k/ you'll see there's just the two latest kernel-devel:

kernel-devel-5.14.0-687.10.1.el9_8.0.1.x86_64.rpm
kernel-devel-5.14.0-687.12.1.el9_8.x86_64.rpm

I'm guessing that the base AMI you're booting from is 5.14.0-611.5.1.el9_7.x86_64 and it doesn't install the latest 5.14.0-687.12.1.el9_8.x86_64.rpm (mirror with missing package?). I'm less sure how to fix that for you, except maybe try another official Rocky9 AMI.

[hanwen-cluster]

Hi all,

We are aware that the current ParallelCluster versions don't support RHEL/Rocky 9.8 because EFA and Lustre incompatibility. The infiniband-diags-compat error is from EFA, while the kmod issue is from Lustre.

The only solution is to use RHEL/Rocky 9.7. When building the image, please disable UpdateOsPackages, which is disabled by default if not specified.

The next ParallelCluster release will support RHEL/Rocky 9.8

Thank you for reporting the issue!
Hanwen

[maestro1315]

any updates on when this will be released ?

[hanwen-cluster]

We do not have a plan to do a quick release for this. So the support will come with pcluster 3.16.0 in several months.

If you would like us to fast-track it, can you describe your use case?

[ebagrenrut]

@hanwen-cluster It's a matter of maintaining patched cluster nodes. Given the increased use of LLMs to find vulnerabilities in the Linux kernel and other distribution-provided packages, and the fact that RHEL/Rocky 9.7 are no longer receiving security patches and now have known vulnerabilities, it's crucial for AWS' customers who use these distributions to ensure that they can upgrade to 9.8 as soon as possible.

[maestro1315]

@hanwen-cluster It's a matter of maintaining patched cluster nodes. Given the increased use of LLMs to find vulnerabilities in the Linux kernel and other distribution-provided packages, and the fact that RHEL/Rocky 9.7 are no longer receiving security patches and now have known vulnerabilities, it's crucial for AWS' customers who use these distributions to ensure that they can upgrade to 9.8 as soon as possible.

This is 100% correct.

[ezrizhu]

We do not have a plan to do a quick release for this. So the support will come with pcluster 3.16.0 in several months.

If you would like us to fast-track it, can you describe your use case?

Similar to the other comments here, I'd like to install the latest Rocky patches & dependency updates to the various linux LPEs instead of having to maintain all of those security patches. As HPC clusters in their nature enables (albeit trusted) users to execute code, it is trivial for users code to exploit a linux LPE on the HPC cluster.

[hanwen-cluster]

Hi @maestro1315 and @ezrizhu

This absolutely makes sense!

As soon as EFA releases 1.49.0, you can use the following DevSettings to build AMIs on Rocky 9.8:

Build:
  InstanceType: ...
  ParentImage: ...
DevSettings:
  Cookbook:
    ExtraChefAttributes: |
      {"cluster": {"efa": {"version": "1.49.0", "sha256": "cf2e9281a2328a243c76f911a490faed43ca0fecfe4733c25e34b2e92a32c309"}}}

In the meantime, RHEL 9.6 is still supported and receiving security updates. However, Rocky 9.6 is EOL.
In conclusion, staying on Rocky 9.7 or RHEL 9.6 is the best for now.

Thank you,
Hanwen

[ebagrenrut]

I can confirm that with the EFA 1.49.0 work-around, I am able to build a Rocky 9.8 image with pcluster 3.15.1.

[hanwen-cluster]

ParallelCluster started to support RHEL 9.8 starting from v3.16.0. Please use 3.16.0 and 3.16.1 to use RHEL 9.8