Repository navigation
fix(arrow-ipc): do not write a validity bitmap for RunEndEncoded in V4 - #12267
Merged
Merged
Conversation
The reader never consumes a validity buffer for RunEndEncoded arrays, in any metadata version, so the extra buffer written for V4 shifted every later buffer by one and the run-ends buffer was read as empty. Align the V4 branch of has_validity_bitmap with V5 and with the reader. Closes apache#12248
Rich-T-kid
approved these changes
Nov 16, 2026
Rich-T-kid
left a comment
Contributor
There was a problem hiding this comment.
nice and simple, thanks @Mola-maker
Contributor
Author
|
Appreciate! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
A
RunEndEncodedarray written withMetadataVersion::V4does not read back.has_validity_bitmapinarrow-ipc/src/writer.rsexcluded onlyNullin the V4 branch, so the writer emitted a validity buffer for RunEndEncoded arrays (a 0-length one whennull_countis 0). The reader never consumes a validity buffer for RunEndEncoded in any metadata version, so it takes that buffer as the run-ends buffer and every later buffer is off by one. The reader fails withInvalidArgumentError("Need at least 8 bytes in buffers[0] in array of type Int32, but got 0").What changes are included in this PR?
has_validity_bitmapnow also excludesDataType::RunEndEncoded(_, _)in the V4 branch, matching the V5 branch, the reader, and the function's own doc comment (which already stated RunEndEncoded has no validity bitmap). The flatbuffers spec lists only Union's validity bitmap as a V4 to V5 difference, so no V4 reader expects one.test_roundtrip_stream_run_array_v4inarrow-ipc/src/reader.rs, following the issue reproducer exactly (Int32 run-ends[2, 5], values[10, 20], written withMetadataVersion::V4, read back and asserted equal).Are these changes tested?
Yes. The new test fails on current main with the exact error from the issue and passes with the fix. The full
arrow-ipcsuite (167 tests) and the downstreamarrow-arrayandarrow-datasuites (1068 tests) pass, andcargo fmtandcargo clippyare clean.Are there any significant user-facing changes?
Behavior fix only: RunEndEncoded arrays written with MetadataVersion::V4 now read back correctly. No public API change.
AI usage disclosure
AI assistance was used to implement the fix and tests, following the approach described in #12248; the changes were verified with the crate's test suite, clippy and fmt.