Skip to content

HDDS-17374. Handle old clients not passing StoragePolicy in allocateBlock - #11716

Merged
devmadhuu merged 6 commits into
apache:HDDS-11233from
anuragp010:HDDS-17374
Nov 19, 2026
Merged

devmadhuu merged 6 commits into
apache:HDDS-11233from
anuragp010:HDDS-17374

Conversation

@anuragp010

@anuragp010 anuragp010 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

With an upgraded cluster containing all OMs supporting storage policy, an older client without the Storage Policy changes will not send the storagePolicy field in any request made to the OM. This is of consequence to allocateBlock as it allocates block based on the storage policy sent by the client. If the client doesn't send a storage policy, it uses the bucket's policy and falls back to the default. But in the scenario where the bucket policy changes from the time key is created to the subsequent allocateBlock call, then allocateBlock will not follow the key's storage policy. This PR fixes that by getting the key's storage policy from the openKeyTable if not passed. Should the openKey not have a policy (Key is created pre upgrade on the OM), then allocateBlock will use the bucket's storage policy, falling back to the default if bucket's policy is null.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17374

How was this patch tested?

  1. Unit Tests
  2. Integration Test: testOldClientAllocateBlockKeepsKeyPolicyWhenBucketPolicyChanges

@github-actions github-actions Bot added the om label Oct 23, 2026
@anuragp010

Copy link
Copy Markdown
Contributor Author

The behaviour of allocateBlock is to allocate a block based on the args passed to it. We would be modifying the core behaviour of allocateBlock here, which is not intended. Hence, the following scenarios must be documented for the feature - If the bucket policy changes before a subsequent allocateBlock made from an older client, then some blocks of the key will not respect the StoragePolicy of the key.

@anuragp010 anuragp010 closed this Oct 26, 2026
@anuragp010
anuragp010 deleted the HDDS-17374 branch October 26, 2026 10:24
@devmadhuu
devmadhuu self-requested a review November 12, 2026 06:59
@anuragp010
anuragp010 restored the HDDS-17374 branch November 12, 2026 06:59
@anuragp010 anuragp010 reopened this Nov 12, 2026
@anuragp010

Copy link
Copy Markdown
Contributor Author

This change is need to handle the scenario mentioned in the pr-comment of the (PR).

@anuragp010

Copy link
Copy Markdown
Contributor Author

Please review @devmadhuu , @ashishkumar50 , @amaliujia !

@devmadhuu devmadhuu left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anuragp010 for the patch. The code changes look good. Could we also add an acceptance test using a real older client binary to cover the bucket-policy change during block allocation?

The existing cross-version compatibility tests provide these references:

Reference What it does
xcompat/test-new.sh Runs older clients against the current cluster.
xcompat/clients.yaml Defines an actual 2.2.1 client image, alongside other released versions.
compatibility/write.robot Uses the selected client binary to write keys, including RATIS/EC cases.
compatibility/read.robot Reads keys and compares their contents with the original file.

For this PR, the test could cover the following scenario:

Step Action Expected result
1 Start the updated, finalized cluster. Use a new client to create a WARM bucket. Bucket policy is WARM.
2 A real older client starts writing a key and allocates its first block. First block uses DISK.
3 Pause that writer. A new client changes the bucket policy to COLD. The open key retains WARM.
4 Resume the same older writer and force another block allocation. The next block still uses DISK.
5 Complete the write and read the key back. Content matches, and every block follows WARM.

Please use an explicit pause/resume signal between the writer and test controller to make the policy change deterministic.

@anuragp010

Copy link
Copy Markdown
Contributor Author

Thanks for the review and detailed pointers, @devmadhuu . I agree this scenario needs an acceptance test with a real older client, but can we take it up as part of a follow-up JIRA (HDDS-17828) rather than in this PR?

The existing xcompat tests run one command to completion in one container. This scenario needs an old-client write that stops after the first block is allocated. While it is stopped, a new client changes the bucket policy, and then the old client resumes. That needs:

  • an old-client writer that can pause and resume, for example reading from a FIFO (named pipe), with a smaller ozone.scm.block.size so the second block triggers allocateBlock.
  • a signal between the old-client and new-client containers, so the policy change happens at a fixed point.
  • changes to lib.sh and possibly a separate cluster configuration which has multiple volumes with different storage types.

That is a larger change than this fix.

@devmadhuu

Copy link
Copy Markdown
Contributor

Thanks for the review and detailed pointers, @devmadhuu . I agree this scenario needs an acceptance test with a real older client, but can we take it up as part of a follow-up JIRA (HDDS-17828) rather than in this PR?

The existing xcompat tests run one command to completion in one container. This scenario needs an old-client write that stops after the first block is allocated. While it is stopped, a new client changes the bucket policy, and then the old client resumes. That needs:

* an old-client writer that can pause and resume, for example reading from a FIFO (named pipe), with a smaller ozone.scm.block.size so the second block triggers allocateBlock.

* a signal between the old-client and new-client containers, so the policy change happens at a fixed point.

* changes to lib.sh and possibly a separate cluster configuration which has multiple volumes with different storage types.

That is a larger change than this fix.

Agree. Thanks, lets handle in follow up PR.

@devmadhuu devmadhuu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anuragp010 for the patch. Changes LGTM +1

@devmadhuu
devmadhuu requested a review from amaliujia November 16, 2026 09:00

@amaliujia amaliujia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core idea is simple and easy to understand.

Just have a minor comment.

ACLType.WRITE, allocateBlockRequest.getClientID());

// An older client does not send the storage policy in allocateBlock requests. Using the bucket policy for them
// is wrong when the bucket policy changes while the key is being written.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the comment in this two lines read odd.

I am wondering if the following is better

Suggested change
// is wrong when the bucket policy changes while the key is being written.
// Older clients, and keys opened without a policy, do not send the policy in allocateBlock. In such cases,
// use the policy stored on the open key so that a bucket policy change during the write does not move later blocks to another tier.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review, @amaliujia !

I have reworded the comment with one minor tweak to your suggestion: I omitted "keys opened without a policy".

Once the storage policy feature is finalized, if a key is created without the client explicitly passing a policy, it is resolved during the createKey/createFile request's preExecute phase on the OM. As a result, the OMKeyInfo object sent back to the client already contains a storage policy. This same OMKeyInfo object is then used for subsequent allocateBlock calls. Therefore, the scenario addressed by this PR will only occur with older clients.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM

@anuragp010
anuragp010 requested a review from amaliujia November 18, 2026 06:41

@amaliujia amaliujia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Probably can just merge this PR.

@devmadhuu
devmadhuu marked this pull request as ready for review November 19, 2026 05:50
@devmadhuu
devmadhuu merged commit e9f780f into apache:HDDS-11233 Nov 19, 2026
127 of 128 checks passed
@devmadhuu

Copy link
Copy Markdown
Contributor

Thanks @anuragp010 for the patch. Thanks @amaliujia for the review.

@anuragp010

Copy link
Copy Markdown
Contributor Author

Thanks @devmadhuu and @amaliujia for the review !

@anuragp010
anuragp010 deleted the HDDS-17374 branch November 19, 2026 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants