Read Iceberg v3 deletion vectors in DataFusion scans - #414
Merged
Merged
Conversation
Replace IcebergDvExec's manual per-file offset cursor with DataFusion 55's native RowNumber virtual column, which materializes each row's true absolute file position. This removes the cursor's documented limitation (misalignment under repartition_file_scans when a file's row groups split across partitions) and makes DV filtering correct regardless of scan partitioning, batch order, or row-group pruning. - dv_exec: filter via a stateless per-row `dv.is_deleted(row_number)` lookup; drop the offsets cursor and the rank/select machinery; strip both internal columns (row-number always, path column when not opted in); repartitioning is now safe. - table_scan: add the RowNumber virtual column (with an unknown ColumnStatistics entry) to the DV scan schema, thread it through the internal projection, and enable predicate pushdown on the DV data scan (safe: supports_filters_pushdown is Inexact, so a FilterExec is retained above the scan). - Tests: rewrite dv_exec unit tests for the row-number model (non-contiguous, split/out-of-order, interleaved cases the cursor could not handle) and add an in-crate real ParquetSource scan test proving true row numbers survive row-group pruning and drive the DV filter end to end. - Reformat the pre-existing deletion_vector.rs files to satisfy rustfmt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds read support for Iceberg v3 deletion vectors (DVs) in
datafusion_icebergscans.deletion-vector-v1Puffin blobs into an in-memory roaring bitmap (iceberg-rust-spec), loaded eagerly via ranged object-store reads keyed by the referenced data-file path (iceberg-rust).IcebergDvExecphysical node that filters rows by(data_file_path, absolute_row_position)— a per-file delete-index rather than a join.RowNumbervirtual column, so filtering is correct under row-group pruning, predicate pushdown, and inter-/intra-file repartitioning.This implements the read side of the deletion vectors section of the Iceberg spec. Writing DVs is a separate follow-up.
Architecture
supports_filters_pushdownisInexact, so aFilterExecis retained above the scan).Validation
cargo test -p datafusion_iceberg --lib— unit tests forIcebergDvExec(non-contiguous positions from pruned row groups, files split across out-of-order batches, interleaved files, column stripping) plus an in-crate real-ParquetSourcescan test proving true row numbers survive row-group pruning and drive the filter end to end.cargo clippy -p datafusion_iceberg --lib --tests -- -D warnings,cargo fmt --all -- --check.Relationship to #412
#412 reads v2 position-delete files (Parquet). Both features filter rows by
(data_file_path, row_position)and share theRowNumberprimitive introduced here. Once this merges, #412 can integrate by feeding v2 position-delete files into the same per-file bitmap index consumed byIcebergDvExec, unifying v2 and v3 on one operator. A detailed integration plan is posted on #412.