Skip to content

Add opt-in dictionary encoding selection from Arrow samples - #12291

Open
Weijun-H wants to merge 6 commits into
apache:mainfrom
Weijun-H:engineer/arrow-11384-dictionary-sampling
Open

Weijun-H wants to merge 6 commits into
apache:mainfrom
Weijun-H:engineer/arrow-11384-dictionary-sampling

Conversation

@Weijun-H

@Weijun-H Weijun-H commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #11384.

Rationale for this change

Approximate parquet-java's first-page dictionary choice from Arrow samples before writing, without changing writer defaults.

What changes are included in this PR?

Adds choose_dictionary_encoding and DictionaryEncodingOptions. Dictionaries are disabled when sampled dictionary and index bytes are not smaller than raw bytes, or the dictionary size limit is exceeded. Nonempty samples also disable Parquet v1 fixed-length byte-array dictionaries.

Sampling stops at the first-page or row-group row limit. The first size check is configurable. Row-group byte limits and custom Parquet schemas are not supported. Page boundaries can differ from parquet-java because size is checked every row and level bytes are excluded. Parquet-java may merge different NaN payloads; this helper distinguishes them. Small samples can reject useful dictionaries.

Uses the existing ahash::RandomState. Fixed-width keys are stored in hash-table buckets, avoiding per-key heap allocations. Small-batch total cost can still exceed writing without sampling.

Adapted from Comet's DictionaryChooser.

Are these changes tested?

25 unit tests, including parquet-java decisions recorded through Iceberg (ported from Comet) and file readback, plus a doc example.

Local measurements

Commit 05a779244; Ratio = (sampling + write) / write without sampling.

(Apple M4 Max, macOS, Rust 1.99.1, release opt-level 3, 16 codegen units, no LTO; in-memory files; median of 20 runs per input; 20,000-row sample in each case)

Input (rows written) Sampling Sampling + write Write without sampling Ratio
75 mixed columns (20k) 19.6 ms 41.1 ms 25.6 ms 1.60×
Nested List<Int32> (20k) 0.50 ms 0.98 ms 0.48 ms 2.05×
Constant Int64 (20k) 0.10 ms 0.17 ms 0.07 ms 2.37×
Random Int64 with nulls (20k) 0.31 ms 0.40 ms 0.21 ms 1.95×
200-byte strings (20k) 0.18 ms 1.87 ms 1.68 ms 1.11×
4 columns (1M) 1.93 ms 98.7 ms 103.4 ms 0.95×

For small batches, sampling plus writing still costs more than writing without sampling. These are microbenchmarks, not end-to-end results.

Are there any significant user-facing changes?

New opt-in API. Existing writer behavior is unchanged.

AI usage disclosure

AI tools adapted the reference implementation and generated the helper, tests, and documentation.

@boring-cyborg boring-cyborg Bot added the parquet Changes to the parquet crate label Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet: choose dictionary encoding per column the way parquet-java does

1 participant