Repository navigation
fix(arrow-array): validate FixedSizeBinaryBuilder value length without truncation - #12293
Surajsuthar wants to merge 3 commits into
Conversation
…iffers from value_length by a multiple of 2^32
Subodh-17
left a comment
There was a problem hiding this comment.
Hi @Surajsuthar,
The change from value.as_ref().len() as i32 to value.as_ref().len() != self.value_length as usize cleanly avoids 32-bit length truncation in FixedSizeBinaryBuilder::append_value!
Please see the line comment regarding the added unit test: (1usize << 32) + 1 overflows on 32-bit targets (i686, wasm32) and allocates 4.3 GiB on 64-bit systems, which can cause test suite failures on memory-constrained CI environments.
There was a problem hiding this comment.
FixedSizeBinaryDictionaryBuilder::{append, append_n} still use the truncating value.as_ref().len() as i32 check. With a length that wraps to byte_width, these checks pass. get_or_insert_key then calls the stricter FixedSizeBinaryBuilder::append_value, discards its error, and appends a key for a value that was never inserted. finish can therefore build a dictionary with an out-of-bounds key. Please use the same usize comparison in both prechecks
The regression test also allocates 4,294,967,297 bytes in the default suite and overflows when run on 32-bit.
@Rich-T-kid i added prechecks on |
Important
Before opening this PR, enable CI in your fork.
Delete this notice after enabling CI.
Which issue does this PR close?
Closes #12285
Rationale for this change
FixedSizeBinaryBuilder::append_valuepreviously cast the input length toi32, allowing values whose length differed fromvalue_lengthby a multiple of2^32.What changes are included in this PR?
Compare the input length as
usizeto avoid truncation.