Repository navigation
feat: generate benchmark datasets into parameter-encoded paths with Parquet layout options - #819
Merged
Merged
Conversation
gabotechs
force-pushed
the
gabrielmusat/benchmark-dataset-layout
branch
from
October 30, 2026 09:29
f421095 to
129cc62
Compare
gabotechs
changed the base branch from
main
to
gabrielmusat/remove-gen-scripts
October 30, 2026 09:30
…arquet layout options The dfbench prepare-* commands now own where and how datasets are generated: - Datasets are written to <root>/<suite>_parquet/<key>=<value>/..., a path that records every generation parameter (e.g. tpch_parquet/scale_factor=1/max_file_size=134217728/dictionary=pages/sorted=false). --root defaults to the repository's testdata/ and accepts an s3:// prefix. - Each TPC-H and TPC-DS table is generated as one ordered stream and split into <table>/N.parquet files by --max-file-size, replacing --partitions. TPC-DS files now cover contiguous key ranges instead of interleaved rows. - --dictionary none|pages|arrow controls Parquet dictionary pages and Arrow dictionary types; --sorted declares sorting_columns, now for TPC-DS too. - Generating a complete dataset again is a no-op (_SUCCESS); any other non-empty destination is rejected. The Iceberg conversion follows the same rule and mirrors the source path under <suite>_iceberg/.../target_file_size=N. - prepare-* and dfbench-iceberg prepare print only the dataset path on stdout, so DATASET=$(dfbench prepare-tpch) works; progress goes to stderr. Tests, the console example and docs use the new layout, and the plan snapshots are regenerated for the new file paths and groupings.
gabotechs
force-pushed
the
gabrielmusat/benchmark-dataset-layout
branch
from
October 30, 2026 09:56
129cc62 to
59e60be
Compare
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #820, which removes the
gen-*.shscripts; review that first.Makes benchmark datasets reproducible from their path and adds Parquet layout knobs:
dfbench prepare-*writes to<root>/<suite>_parquet/<key>=<value>/..., a path that records every generation parameter (e.g.tpch_parquet/scale_factor=1/max_file_size=134217728/dictionary=pages/sorted=false).--rootdefaults to the repository'stestdata/and accepts ans3://prefix.<table>/N.parquetby--max-file-size(default 128 MiB), replacing--partitions. TPC-DS files now cover contiguous key ranges instead of interleaved rows.--dictionary none|pages|arrowcontrols Parquet dictionary pages and Arrow dictionary types;--sorteddeclaressorting_columns, now for TPC-DS as well._SUCCESS); any other non-empty destination is rejected.dfbench-iceberg preparefollows the same rule and mirrors the source under<suite>_iceberg/.../target_file_size=N.prepare-*anddfbench-iceberg prepareprint only the dataset path on stdout, soDATASET=$(dfbench prepare-tpch)works; a progress bar reports per-table progress on stderr.Tests and the console example generate into the same layout, so the plan snapshots are regenerated for the new file paths and groupings (most of the diff, in
tests/*_plans_test.rs). The ClickBench CI cache moves totestdata/clickbench_parquet/.Breaking for local usage:
--partitionsand--outputare gone fromprepare-*, and existing local datasets undertestdata/<suite>/need regenerating.🤖 Generated with Claude Code