Skip to content

perf(parquet): copy bytes once in DeltaLengthByteArrayEncoder and DeltaByteArrayEncoder - #12273

Closed
dtenedor wants to merge 2 commits into
apache:mainfrom
dtenedor:perf-delta-byte-array-encoders
Closed

dtenedor wants to merge 2 commits into
apache:mainfrom
dtenedor:perf-delta-byte-array-encoders

Conversation

@dtenedor

Copy link
Copy Markdown

Which issue does this PR close?

None.

Rationale for this change

The column-level delta byte array encoders, which the non-Arrow column writer API uses, do extra
work for every value:

  • DeltaLengthByteArrayEncoder keeps a ByteArray clone of every value, an atomic increment and
    decrement each, and copies the bytes at flush.
  • Both encoders allocate temporary Vecs on every put.
  • DeltaByteArrayEncoder copies every value into its previous-value buffer and slices a new
    ByteArray for every suffix.

The Arrow writer has its own byte array encoders and is not affected.

What changes are included in this PR?

  • Values are copied once, as they are put, into a single byte buffer that is reserved from the
    input's total length.
  • Lengths are encoded in stack batches of 128.
  • DeltaByteArrayEncoder compares each value with its predecessor in place and keeps only the last
    value of each put.

Output is byte-identical.

The first commit adds the benchmark cases, so it can serve as the benchmark baseline. #12272 starts
with the same commit.

Are these changes tested?

Covered by the existing encoder tests. Output was also checked to be byte-identical to main's for
DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY over several datasets, value lengths and put
chunkings.

The first commit adds delta byte array cases to the encoding bench:

  • 16K sorted keys with shared prefixes
  • 16K random strings of 8-31 bytes
  • 1K values of 1 KiB

The encoder is reused across iterations. Measured on an Intel Xeon Platinum 8559C, default x86-64
target, median of 5 alternating rounds:

encoding bench Speedup
DELTA_BYTE_ARRAY encode: random / sorted_keys / large_1k 3.26x / 2.27x / 1.70x
DELTA_LENGTH_BYTE_ARRAY encode: random / sorted_keys 2.90x / 1.90x
DELTA_LENGTH_BYTE_ARRAY encode: large_1k 0.78x
decode, all cases 0.99-1.01x

DELTA_LENGTH_BYTE_ARRAY gets slower on large values because their bytes are now copied twice: once
at put and again at flush. Before, put only took a clone. The Arrow writer's fallback encoder
already works this way.

Are there any significant user-facing changes?

estimated_memory_size of DeltaLengthByteArrayEncoder now counts the byte buffer's capacity.
Before, it counted the number of buffered values.

AI usage disclosure

An AI coding agent (Claude Code) generated the code, the benchmark cases and this description, and
ran the benchmarks and the output checks. This PR is a draft until the submitter has reviewed it.

This pull request and its description were written by Isaac.

dtenedor and others added 2 commits November 16, 2026 07:17
… encoding bench

Co-authored-by: Isaac <no-reply@databricks.com>
…taByteArrayEncoder

DeltaLengthByteArrayEncoder kept a clone of every ByteArray (a reference count
increment and decrement per value) and copied the bytes at flush, and both
encoders allocated temporary Vecs on every put. DeltaByteArrayEncoder also
copied each value into its previous-value buffer and sliced a new ByteArray for
every suffix.

Values are now copied into one byte buffer as they are put, lengths are encoded
in stack batches of 128, and DeltaByteArrayEncoder compares each value with its
predecessor in place, keeping only the last value of a put. Output is
byte-identical.

Co-authored-by: Isaac <no-reply@databricks.com>
@boring-cyborg boring-cyborg Bot added the parquet Changes to the parquet crate label Nov 16, 2026
@dtenedor dtenedor closed this Nov 16, 2026
@dtenedor
dtenedor deleted the perf-delta-byte-array-encoders branch November 16, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant