Repository navigation
Conversation
… encoding bench Co-authored-by: Isaac <no-reply@databricks.com>
…taByteArrayEncoder DeltaLengthByteArrayEncoder kept a clone of every ByteArray (a reference count increment and decrement per value) and copied the bytes at flush, and both encoders allocated temporary Vecs on every put. DeltaByteArrayEncoder also copied each value into its previous-value buffer and sliced a new ByteArray for every suffix. Values are now copied into one byte buffer as they are put, lengths are encoded in stack batches of 128, and DeltaByteArrayEncoder compares each value with its predecessor in place, keeping only the last value of a put. Output is byte-identical. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
None.
Rationale for this change
The column-level delta byte array encoders, which the non-Arrow column writer API uses, do extra
work for every value:
DeltaLengthByteArrayEncoderkeeps aByteArrayclone of every value, an atomic increment anddecrement each, and copies the bytes at flush.
Vecs on everyput.DeltaByteArrayEncodercopies every value into its previous-value buffer and slices a newByteArrayfor every suffix.The Arrow writer has its own byte array encoders and is not affected.
What changes are included in this PR?
input's total length.
DeltaByteArrayEncodercompares each value with its predecessor in place and keeps only the lastvalue of each
put.Output is byte-identical.
The first commit adds the benchmark cases, so it can serve as the benchmark baseline. #12272 starts
with the same commit.
Are these changes tested?
Covered by the existing encoder tests. Output was also checked to be byte-identical to main's for
DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY over several datasets, value lengths and
putchunkings.
The first commit adds delta byte array cases to the
encodingbench:The encoder is reused across iterations. Measured on an Intel Xeon Platinum 8559C, default x86-64
target, median of 5 alternating rounds:
encodingbenchDELTA_LENGTH_BYTE_ARRAY gets slower on large values because their bytes are now copied twice: once
at
putand again at flush. Before,putonly took a clone. The Arrow writer's fallback encoderalready works this way.
Are there any significant user-facing changes?
estimated_memory_sizeofDeltaLengthByteArrayEncodernow counts the byte buffer's capacity.Before, it counted the number of buffered values.
AI usage disclosure
An AI coding agent (Claude Code) generated the code, the benchmark cases and this description, and
ran the benchmarks and the output checks. This PR is a draft until the submitter has reviewed it.
This pull request and its description were written by Isaac.