Repository navigation
Conversation
…able The dictionary interner's hash table held only each value's index into the dictionary, so every lookup read the value back from the dictionary to compare it, a second dependent cache miss after the table entry, and every time the table grew, hashbrown re-hashed each value read back the same way. Lookups were 35% of low-cardinality dictionary writes, and growth 7-15% of high-cardinality ones, which build a dictionary until it outgrows its page size limit. For values of one fixed width of at most 8 bytes (BOOLEAN, INT32, INT64, FLOAT, DOUBLE), table entries now keep the value's bytes next to its index, so lookups compare in the table and growth hashes the entry itself. For other values, entries keep the value's hash, so growth no longer re-hashes, and probes compare full hashes before reading the dictionary. Dictionaries are built in the same order, so output is byte-identical. Entries double in size. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
None.
Rationale for this change
The dictionary interner's hash table stores only each value's dictionary index. That has two costs:
cache miss after the table entry, and it makes
DictEncoder::put35% of low-cardinalitydictionary writes.
High-cardinality columns build a dictionary until it outgrows the dictionary page size limit, so
growth costs them 7-15% of write time.
What changes are included in this PR?
Storagegets an associated const,INLINE_VALUES, which defaults to false. The columndictionary encoder sets it for BOOLEAN, INT32, INT64, FLOAT and DOUBLE.
u64, next to its index. Lookupscompare in the table, and growth hashes the entry itself.
hash. Growth doesn't re-hash, and probes compare hashes before reading the value.
Dictionaries are built in the same order, so output is byte-identical. Table entries double in
size, from 8 to 16 bytes, and
estimated_memory_sizereflects that.Are these changes tested?
Covered by the existing dictionary and writer tests. Writer output was also checked to be
byte-identical to main's.
arrow_writerbench, Intel Xeon Platinum 8559C, default x86-64 target, median of 7 alternatingrounds:
arrow_writerbenchint64_dictionary_1pct_cardinality/defaultprimitive_dictionary_1pct_cardinality/defaultfloat64_dictionary_1pct_cardinality/defaultprimitive_non_null/defaultprimitive_non_null/parquet_2primitive/defaultprimitive_dictionary/defaultint64_dictionary/defaultStoring the full hash next to each key, for every type, helps growth but leaves the dictionary read
on every hit. Keeping fixed-width values in the entry is what speeds up the low-cardinality cases.
None of the measured inputs got slower. The cost is the table's memory: 8 more bytes per distinct
value.
Are there any significant user-facing changes?
No. The new const is on the interner's
Storagetrait, which is crate-private.AI usage disclosure
An AI coding agent (Claude Code) generated the code and this description, and ran the benchmarks and
the output checks. This PR is a draft until the submitter has reviewed it.
This pull request and its description were written by Isaac.