Repository navigation
HDDS-17724. Limit command-status report size in datanode heartbeats. - #11962
sadanand48 wants to merge 23 commits into
Conversation
9e27fef to
0526bcb
Compare
…/java/org/apache/hadoop/ozone/container/common/states/endpoint/TestHeartbeatEndpointTask.java
smengcl
left a comment
There was a problem hiding this comment.
Thanks @sadanand48 for the patch
sumitagrawl
left a comment
There was a problem hiding this comment.
@sadanand48 given few comment
smengcl
left a comment
There was a problem hiding this comment.
lgtm.
@sumitagrawl Do you want to take another look?
|
Thanks @sadanand48 for the patch.
Can you please re-enable workflows in your fork? Afterwards you may want to manually disable some: |
| public static final String HDDS_IPC_MAXIMUM_DATA_LENGTH = | ||
| "ipc.maximum.data.length"; |
There was a problem hiding this comment.
@sadanand48 TestOzoneConfigurationFields failed. ipc.maximum.data.length should not be declared in HddsConfigKeys (since ozone-default.xml does not have it, nor should it), but in a non-config class (which is not checked by TestOzoneConfigurationFields). For now it can be moved to e.g. HeartbeatEndpointTask.
|
@sadanand48 Please don't re-run failed workflow of the outdated commit (853c5f8), it cancels the workflow for the newer commit (d35de55). |
…/java/org/apache/hadoop/ozone/container/common/states/endpoint/TestHeartbeatEndpointTask.java
|
I had a discussion with @sumitagrawl where he proposed approach 2 - the publisher-side alternative and attached the patch so we can compare both approaches before deciding. Approach 1: split while constructing the heartbeat (the current PR) CommandStatusReportPublisher remains unchanged and publishes one aggregate CommandStatusReportsProto.
Trade-off: Approach 2: split in CommandStatusReportPublisher
Advantages:
Trade-offs:
@sumitagrawl @smengcl @aryangupta1998 could you please share your thoughts to conclude on this one. Thanks |
What changes were proposed in this pull request?
HDDS-16723 added a count-based limit for reports in datanode heartbeats, but it did not limit their serialized size. A small number of command-status reports containing accumulated delete-block ACKs could still exceed ipc.maximum.data.length.
This PR:
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17724
How was this patch tested?
Unit tests
Developed with help of Cursor (GPT-5.6 Sol)