Skip to content

fix(arrow-ipc): compute variadic buffer counts from the unsliced run array - #12268

Open
Mola-maker wants to merge 1 commit into
apache:mainfrom
Mola-maker:fix/12249-ree-zero-row-variadic
Open

Mola-maker wants to merge 1 commit into
apache:mainfrom
Mola-maker:fix/12249-ree-zero-row-variadic

Conversation

@Mola-maker

@Mola-maker Mola-maker commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

An IPC stream containing a RunEndEncoded array with Utf8View (or BinaryView) values sliced to zero rows does not read back. In encode_record_batch_data, the buffers and the variadic buffer counts came from two different arrays: write_array_data writes the children of unslice_run_array(array_data), but append_variadic_buffer_counts walked the original array_data. For a zero-row slice, into_zero_offset_run_array returns an empty array whose Utf8View values own no data buffers (0 written), while the original values still own one (count 1). The reader then fails with IpcError("Invalid variadic count 1 for Utf8View column, with 2 buffer(s) left in the message"). The same happens when the view array is nested inside the run values, e.g. RunEndEncoded<Int32, ListView<Utf8View>>.

What changes are included in this PR?

  • In encode_record_batch_data, RunEndEncoded columns are unsliced before the variadic counts are computed, so the counts come from the same unsliced array the writer wrote. append_variadic_buffer_counts itself is unchanged.
  • Three regression tests in arrow-ipc/src/reader.rs: the issue reproducer exactly (Utf8View values with a non-inlined string, batch sliced to 0 rows), plus the nested ListView<Utf8View> and List<Utf8View> variants from the issue.

Are these changes tested?

Yes. All three new tests fail on current main with the exact error from the issue and pass with the fix. The full arrow-ipc suite (169 tests) and the downstream arrow-array and arrow-data suites (1068 tests) pass, and cargo fmt and cargo clippy are clean.
arrow-rs-12249-verify-before-fix
arrow-rs-12249-verify-after-fix

Are there any significant user-facing changes?

Behavior fix only: zero-row slices of RunEndEncoded arrays with view values now round-trip correctly. No public API change.

AI usage disclosure

AI assistance was used to implement the fix and tests, following the approach described in #12249; the changes were verified with the crate's test suite, clippy and fmt.

@boring-cyborg boring-cyborg Bot added the arrow-ipc arrow-ipc crate label Nov 16, 2026
@Rich-T-kid
Rich-T-kid self-requested a review November 16, 2026 02:17
@Rich-T-kid Rich-T-kid added the bug label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow-ipc arrow-ipc crate bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IPC: zero-row slice of RunEndEncoded with Utf8View values fails to read: Invalid variadic count

2 participants