Skip to content

fix(arrow-array): validate FixedSizeBinaryBuilder value length without truncation - #12293

Open
Surajsuthar wants to merge 3 commits into
apache:mainfrom
Surajsuthar:fix/arrow-array-fixed-size-bin-builder
Open

Surajsuthar wants to merge 3 commits into
apache:mainfrom
Surajsuthar:fix/arrow-array-fixed-size-bin-builder

Conversation

@Surajsuthar

Copy link
Copy Markdown
Contributor

Important

Before opening this PR, enable CI in your fork.

Delete this notice after enabling CI.

Which issue does this PR close?

Closes #12285

Rationale for this change

FixedSizeBinaryBuilder::append_value previously cast the input length to i32, allowing values whose length differed from value_length by a multiple of 2^32.

What changes are included in this PR?

Compare the input length as usize to avoid truncation.

…iffers from value_length by a multiple of 2^32
@boring-cyborg boring-cyborg Bot added the arrow-core Core crates related to representing arrow data label Nov 17, 2026

@Subodh-17 Subodh-17 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Surajsuthar,

The change from value.as_ref().len() as i32 to value.as_ref().len() != self.value_length as usize cleanly avoids 32-bit length truncation in FixedSizeBinaryBuilder::append_value!

Please see the line comment regarding the added unit test: (1usize << 32) + 1 overflows on 32-bit targets (i686, wasm32) and allocates 4.3 GiB on 64-bit systems, which can cause test suite failures on memory-constrained CI environments.

@Rich-T-kid Rich-T-kid left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FixedSizeBinaryDictionaryBuilder::{append, append_n} still use the truncating value.as_ref().len() as i32 check. With a length that wraps to byte_width, these checks pass. get_or_insert_key then calls the stricter FixedSizeBinaryBuilder::append_value, discards its error, and appends a key for a value that was never inserted. finish can therefore build a dictionary with an out-of-bounds key. Please use the same usize comparison in both prechecks

The regression test also allocates 4,294,967,297 bytes in the default suite and overflows when run on 32-bit.

@Rich-T-kid
Rich-T-kid self-requested a review November 17, 2026 16:50
@Surajsuthar

Copy link
Copy Markdown
Contributor Author

FixedSizeBinaryDictionaryBuilder::{append, append_n} still use the truncating value.as_ref().len() as i32 check. With a length that wraps to byte_width, these checks pass. get_or_insert_key then calls the stricter FixedSizeBinaryBuilder::append_value, discards its error, and appends a key for a value that was never inserted. finish can therefore build a dictionary with an out-of-bounds key. Please use the same usize comparison in both prechecks

The regression test also allocates 4,294,967,297 bytes in the default suite and overflows when run on 32-bit.

@Rich-T-kid i added prechecks on FixedSizeBinaryDictionaryBuilder::{append, append_n} and make regression test to run on 64-bit

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow-core Core crates related to representing arrow data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FixedSizeBinaryBuilder::append_value accepts values whose length differs from value_length by a multiple of 2^32

3 participants