API doesn't allow deletion

[sean-smith]

Required Info:

• AWS ParallelCluster version [e.g. 3.1.1]: 3.1.2

Bug description and how to reproduce:
When deleting a cluster with pcluster API, I get:

API: iam:DetachRolePolicy User: arn:aws:sts::905784713722:assumed-role/WX-parallelclusterapiNeste-ParallelClusterUserRole-OWFSQ9VL7CL1/WX-parallelclusterapiNeste-ParallelClusterFunction-NQNwoIOyf6tT is not authorized to perform: iam:DetachRolePolicy on resource: role api-cluster-2022-05-09-17-46-53-RoleHeadNode-EXNM6B7GER1S because no identity-based policy allows the iam:DetachRolePolicy action

Wondering if we can add iam:DetachRolePolicy to the ParallelClusterUserRole created by the API?

In the cluster's config I've set the following AdditionalIamPolicies:

    AdditionalIamPolicies:
      - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
      - Policy: arn:aws:iam::aws:policy/AmazonS3FullAccess
      - Policy: arn:aws:iam::aws:policy/AmazonFSxFullAccess
      - Policy: arn:aws:iam::aws:policy/SecretsManagerReadWrite

[hanwen-pcluste]

Hi Sean!

Sorry for the late reply!

When deploying ParallelCluster API, you can turn on EnableIamAdminAccess to allow IAM related operation. By default the AWS Lambda function executing AWS ParallelCluster API operations is configured with a IAM role that prevents any privileged IAM access (EnableIamAdminAccess=false). This makes the API unable to execute operations that require the creation of IAM roles or policies.

Thank you,
Hanwen

[t-brown]

Hi @hanwen-pcluste , I am getting the same error even with EnableIamAdminAccess=true, for example this is my configuration in CDK:

region = Aws.REGION
version = "3.1.2"
template_url = f"https://{region}-aws-parallelcluster.s3.{region}.amazonaws.com/parallelcluster/{version}/api/parallelcluster-api.yaml"
params = {
    "ApiDefinitionS3Uri": f"s3://{region}-aws-parallelcluster/parallelcluster/{version}/api/ParallelCluster.openapi.yaml",
    "EnableIamAdminAccess": "true",
     "CreateApiUserRole": "false",
}

self.api = CfnStack(self, "APITemplate", template_url=template_url, parameters=params)

Do you have any suggestions on how to possibly start debugging this?

[zbarr]

I am seeing the same issue. EnableIamAdminAccess=true but getting this in the Cloudformation event log:

API: iam:DetachRolePolicy User: arn:aws:sts:::assumed-role// is not authorized to perform: iam:DetachRolePolicy on resource: role because no identity-based policy allows the iam:DetachRolePolicy action

[zbarr]

More clarification now that I'm less lost in the sauce. Duplicate status reason for failure from cloudformation:
API: iam:DetachRolePolicy User: arn:aws:sts::<account id>:assumed-role/<stack name>-ParallelClusterUserRole-SMKBU57WKIRH/<stack name>-ParallelClusterFunction-G0Hp4ybkw4K6 is not authorized to perform: iam:DetachRolePolicy on resource: role <cluster name>-Role15b342af42246b70-1LKMNB77RIUS3 because no identity-based policy allows the iam:DetachRolePolicy action

Followed by:
The following resource(s) failed to delete: [Role15b342af42246b70, RoleHeadNode].

I dug into -ParallelClusterUserRole-SMKBU57WKIRH and the it has the policy, -DefaultParallelClusterIamAdminPolicy-PX5O0ZY2OF78. Snippet from that policy:
{ "Condition": { "ArnLike": { "iam:PolicyARN": [ "arn:aws:iam::<account id>:policy/parallelcluster*", "arn:aws:iam::<account id>:policy/parallelcluster/*", "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy", "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore", "arn:aws:iam::aws:policy/AWSBatchFullAccess", "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess", "arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole", "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role", "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy", "arn:aws:iam::aws:policy/service-role/AmazonEC2SpotFleetTaggingRole", "arn:aws:iam::aws:policy/EC2InstanceProfileForImageBuilder", "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" ] } }, "Action": [ "iam:AttachRolePolicy", "iam:DetachRolePolicy" ], "Resource": "arn:aws:iam::<account id>:role/parallelcluster/*", "Effect": "Allow", "Sid": "IamPolicy" }
... which makes me believe that this should be working. Any help would be great, this is a big blocker for me.

[sean-smith]

@zbarr You can resolve the error by adding permissions - see this guide: https://isc22.hpcworkshops.com/04-container-parallelcluster/01-update-lambda.html I put together for one of our conferences.

We'll keep this issue open to track until we have a real solution.

[zbarr]

Shouldn't this be marked as a bug, rather than enhancement? It's broken functionality.

[stephenmsachs]

... which makes me believe that this should be working. Any help would be great, this is a big blocker for me.

I ran into the same issue with pcluster-manager here: aws-samples/pcluster-manager#279

As the conditions for AttachRolePolicy snd DetachRolePolicy are the same, I would expect this to work as well. Is this a bug in CFN? Further proof for somehting wrong is that the remaining role (the one that cannot be deleted in CFN) does not have any policies attached to it.

[chenwany]

Hi @zbarr and @stephenmsachs
Thanks for reporting the issue. We are able to reproduce and tracking it internally. We are working on the fix of it.

[YikaiHu]

Any update? I still can not delete a cluster through the UI.

I encountered the following CDK issue:

The resource RoleHeadNode is in a DELETE_FAILED state                                                                                       
  This AWS::IAM::Role resource is in a DELETE_FAILED state.                                                                                                                        
                                                                                                                                                                                   
  Resource handler returned message: "Cannot delete entity, must detach all policies first. (Service: Iam, Status Code: 409, Request ID: 71a915d7-fffc-467f-a0ae-ab83aa1cc896)     
  (SDK Attempt Count: 1)" (RequestToken: 35c46a28-715c-1702-28f5-331756b46c41, HandlerErrorCode: GeneralServiceException)