Skip to content

fix(arrow-ipc): do not write a validity bitmap for RunEndEncoded in V4 - #12267

Merged
Rich-T-kid merged 1 commit into
apache:mainfrom
Mola-maker:fix/12248-ree-v4-validity
Nov 16, 2026
Merged

Rich-T-kid merged 1 commit into
apache:mainfrom
Mola-maker:fix/12248-ree-v4-validity

Conversation

@Mola-maker

@Mola-maker Mola-maker commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

A RunEndEncoded array written with MetadataVersion::V4 does not read back. has_validity_bitmap in arrow-ipc/src/writer.rs excluded only Null in the V4 branch, so the writer emitted a validity buffer for RunEndEncoded arrays (a 0-length one when null_count is 0). The reader never consumes a validity buffer for RunEndEncoded in any metadata version, so it takes that buffer as the run-ends buffer and every later buffer is off by one. The reader fails with InvalidArgumentError("Need at least 8 bytes in buffers[0] in array of type Int32, but got 0").

What changes are included in this PR?

  • has_validity_bitmap now also excludes DataType::RunEndEncoded(_, _) in the V4 branch, matching the V5 branch, the reader, and the function's own doc comment (which already stated RunEndEncoded has no validity bitmap). The flatbuffers spec lists only Union's validity bitmap as a V4 to V5 difference, so no V4 reader expects one.
  • One regression test test_roundtrip_stream_run_array_v4 in arrow-ipc/src/reader.rs, following the issue reproducer exactly (Int32 run-ends [2, 5], values [10, 20], written with MetadataVersion::V4, read back and asserted equal).

Are these changes tested?

Yes. The new test fails on current main with the exact error from the issue and passes with the fix. The full arrow-ipc suite (167 tests) and the downstream arrow-array and arrow-data suites (1068 tests) pass, and cargo fmt and cargo clippy are clean.

arrow-rs-12248-verify-before-fix arrow-rs-12248-verify-after-fix

Are there any significant user-facing changes?

Behavior fix only: RunEndEncoded arrays written with MetadataVersion::V4 now read back correctly. No public API change.

AI usage disclosure

AI assistance was used to implement the fix and tests, following the approach described in #12248; the changes were verified with the crate's test suite, clippy and fmt.

The reader never consumes a validity buffer for RunEndEncoded arrays,
in any metadata version, so the extra buffer written for V4 shifted
every later buffer by one and the run-ends buffer was read as empty.
Align the V4 branch of has_validity_bitmap with V5 and with the
reader.

Closes apache#12248
@boring-cyborg boring-cyborg Bot added the arrow-ipc arrow-ipc crate label Nov 16, 2026
@Rich-T-kid Rich-T-kid self-assigned this Nov 16, 2026
@Rich-T-kid Rich-T-kid added the bug label Nov 16, 2026

@Rich-T-kid Rich-T-kid left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice and simple, thanks @Mola-maker

@Rich-T-kid
Rich-T-kid added this pull request to the merge queue Nov 16, 2026
@Mola-maker

Copy link
Copy Markdown
Contributor Author

Appreciate!

Merged via the queue into apache:main with commit 7dd9f12 Nov 16, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow-ipc arrow-ipc crate bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IPC: RunEndEncoded written with MetadataVersion::V4 does not read back

2 participants