Skip to content

fix(parquet): build bloom filters from dictionary entries - #1164

Draft
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/parquet-dictionary-bloom-hashing
Draft

fix(parquet): build bloom filters from dictionary entries#1164
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/parquet-dictionary-bloom-hashing

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • track dictionary entries referenced by encoded values
  • populate Bloom filters from those entries once per column chunk
  • cover direct Arrow dictionary arrays, multiple data pages, null indices, and dictionary fallback
  • keep the existing per-value path after fallback to plain encoding

Why

Bloom filter updates currently happen in the ordinary value-writing paths. Direct dictionary-array writes bypass those paths and write indices directly, which leaves their Bloom filters empty. This can produce false negatives when readers use the filter.

Ordinary dictionary encoding also hashes every logical value even though inserting the same hash repeatedly does not change the filter. Reusing the dictionary reduces hashing from the row count to the number of referenced dictionary entries while dictionary encoding remains active.

Benchmark

Apple M1 Pro, 100,000 int32 values, median of 10 runs with a 500 ms benchmark time:

Cardinality Before After Speedup B/op allocs/op
1 3.568 ms 1.745 ms 2.04x as fast -75.0% -51.8%
10 3.902 ms 2.136 ms 1.83x as fast -71.7% -50.8%
100 3.956 ms 2.118 ms 1.87x as fast -69.5% -49.0%

Speedups are calculated directly from the displayed median times. Higher-cardinality cases that trigger dictionary fallback remain close to the existing path.

Tests

  • full repository test suite
  • race tests for parquet/file, parquet/internal/encoding, and parquet/pqarrow
  • vet for the changed packages

@fallintoplace
fallintoplace marked this pull request as draft August 10, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant