Repository navigation
Conversation
LevelEncoder::put_with_observer fed every level through RleEncoder::put, whose run state lives in the encoder and was stored and reloaded per level, and called the observer once per level outside runs, where the column writer's histogram update made each level wait for the previous one: 5-7 ns per level for random or nullable definition levels. It now encodes through RleEncoder's group-at-a-time put_batch path, generalized over the value width so i16 levels are not widened first, with an observer told about each RLE run or single-value group as one run. Values of bit-packed groups are counted without calling the observer, per group for one-bit levels and in a small histogram up to four bits, and reported once per call. Runs continue to be extended a 32-value block at a time with vector compares. put_with_observer's contract now promises totals per level rather than one call per run; its callers only accumulate counts. Output is byte-identical. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
LevelEncoder::put_with_observercosts 5-7 ns per level for random or nullable definition levels,more than encoding most values. There are two causes:
RleEncoder::put, whose run state lives in the encoder and is storedand reloaded per level.
makes each level wait on the previous one.
What changes are included in this PR?
RleEncoder's group-at-a-timeput_batchpath is generalized over the value width asput_batch_observed<T: RleValue, O: RunObserver<T>>, soi16levels use it without beingwidened.
put_batchbehaves as before.and in a small histogram for up to 4 bits. The counts are reported once per call.
put_with_observer's documented contract changes from "called once per run" to "totals perlevel". Its callers only accumulate counts.
Output is byte-identical.
Are these changes tested?
inputs, and ragged batch sizes. It also compares the encoded bytes against per-value
put.columns.
level_encodingbench, Intel Xeon Platinum 8559C, default x86-64 target, median of 5 alternatingrounds:
level_encodingbencharrow_writer, 7 rounds:primitive1.25x,list_primitive1.27x,list_nested1.27x,primitive_sparse_99pct_null1.13x,struct_sparse_99pct_null1.09x,string1.06x. Non-nullcontrols, which encode no levels, were 0.97-1.01x over 9 rounds.
A first version counted levels in separate passes after encoding. It slowed run-heavy inputs to
0.36-0.55x, because the encoder already reports runs for free. Taking the counts from the encoder's
own runs and groups is what keeps those cases fast.
None of the measured inputs got slower.
Are there any significant user-facing changes?
The documented contract of
LevelEncoder::put_with_observer, which is experimental and hidden fromthe docs.
AI usage disclosure
An AI coding agent (Claude Code) generated the code, the tests and this description, and ran the
benchmarks and the output checks. This PR is a draft until the submitter has reviewed it.
This pull request and its description were written by Isaac.