Cluster update times out while fetching compute node LaunchTemplates for queues

[inoue-hideki-a-opc]

Required Info

• AWS ParallelCluster version: 3.15.0
• OS: Ubuntu 24.04 LTS
• Region: us-east-1

Bug description and how to reproduce

When queue settings are changed and the cluster is updated with pcluster update-cluster, the process times out while the Chef run on the head node retrieves the Compute Node LaunchTemplates associated with the queues.

We believe this occurs because the current implementation assumes that processing all queues will complete within 30 seconds.

As a temporary workaround, we changed the timeout setting from 30 to 300 directly in the following file:

• /etc/chef/cookbooks/aws-parallelcluster-platform/resources/fetch_dna_files.rb

Expected behavior

As the number of queues increases, simply extending the timeout does not provide a fundamental solution. The timeout should apply to the processing of each queue rather than to the processing of all queues together.

Related Issues

• Issue #7203: HeadNode checks all compute nodes when adding queue, causing timeout with running jobs

[hgreebe]

Hello,
Could you provide a cluster configuration and what update you are making?
How many queues are in your cluster?

[inoue-hideki-a-opc]

@hgreebe Thank you for your reply. Our configuration is as follows. Although there are 23 queues, since we have an internal proxy set up, it’s possible that communication with the AWS API from the Head Node is taking some time. I’m not entirely sure about the specific changes made to the cluster, but I believe we added about one or two queues.

Region: us-east-1

Image:
  Os: ubuntu2404

SharedStorage:
  - Name: apps
    StorageType: Ebs
    MountDir: /shared
    EbsSettings:
      VolumeType: gp2
      Size: 200

HeadNode:
  InstanceType: t3a.large
  Networking:
    SubnetId: subnet-00000000000000000
    ElasticIp: false
    Proxy:
      HttpProxyAddress: http://proxy.example.com:8080
  Ssh:
    KeyName: example-pcluster-key
    AllowedIps: 172.16.0.0/12
  LocalStorage:
    RootVolume:
      Size: 100
  Dcv:
    Enabled: false

Monitoring:
  Logs:
    CloudWatch:
      Enabled: true
      RetentionInDays: 180

Scheduling:
  Scheduler: slurm
  ScalingStrategy: all-or-nothing
  SlurmSettings:
    ScaledownIdletime: 10
    QueueUpdateStrategy: DRAIN
    Database:
      Uri: example-slurmdb-instance.abcdefghijkl.us-east-1.rds.amazonaws.com
      UserName: admin
      PasswordSecretArn: arn:aws:secretsmanager:us-east-1:000000000000:secret:/example-pcluster/slurmdbdPassword-xxxxxx
  SlurmQueues:
    - Name: q-example-gpu-single
      CapacityType: ONDEMAND
      ComputeResources:
        - Name: g4dnxlarge
          Instances:
            - InstanceType: g4dn.xlarge
          MinCount: 0
          MaxCount: 50
      Networking:
        SubnetIds:
          - subnet-00000000000000000
          - subnet-00000000000000001
        Proxy:
          HttpProxyAddress: http://proxy.example.internal:8080
  [... and 22 queues]