Skip to content

Read Iceberg v3 deletion vectors in DataFusion scans - #414

Merged
JanKaul merged 2 commits into
mainfrom
deletion-vector-v1
Sep 23, 2026
Merged

JanKaul merged 2 commits into
mainfrom
deletion-vector-v1

Conversation

@JanKaul

@JanKaul JanKaul commented Sep 23, 2026

Copy link
Copy Markdown
Owner

Summary

Adds read support for Iceberg v3 deletion vectors (DVs) in datafusion_iceberg scans.

  • Parse deletion-vector-v1 Puffin blobs into an in-memory roaring bitmap (iceberg-rust-spec), loaded eagerly via ranged object-store reads keyed by the referenced data-file path (iceberg-rust).
  • Apply DVs during the scan with a custom IcebergDvExec physical node that filters rows by (data_file_path, absolute_row_position) — a per-file delete-index rather than a join.
  • Derive the absolute row position from DataFusion's native Parquet RowNumber virtual column, so filtering is correct under row-group pruning, predicate pushdown, and inter-/intra-file repartitioning.

This implements the read side of the deletion vectors section of the Iceberg spec. Writing DVs is a separate follow-up.

Architecture

Puffin DV blob ─> load_deletion_vectors ─> HashMap<data_file_path, DeletionVector(roaring)>
                                                        │
data scan (+ __data_file_path, RowNumber virtual col) ─┴─> IcebergDvExec  ─> filtered rows
                                                             (probe bitmap by true row number)
  • Data files with no DV take the existing scan path untouched — zero overhead.
  • The DV index holds only the compact roaring bitmaps; the scan stays streaming and preserves ordering.
  • Predicate pushdown is enabled on the DV data scan (safe: supports_filters_pushdown is Inexact, so a FilterExec is retained above the scan).

Validation

  • cargo test -p datafusion_iceberg --lib — unit tests for IcebergDvExec (non-contiguous positions from pruned row groups, files split across out-of-order batches, interleaved files, column stripping) plus an in-crate real-ParquetSource scan test proving true row numbers survive row-group pruning and drive the filter end to end.
  • cargo clippy -p datafusion_iceberg --lib --tests -- -D warnings, cargo fmt --all -- --check.

Relationship to #412

#412 reads v2 position-delete files (Parquet). Both features filter rows by (data_file_path, row_position) and share the RowNumber primitive introduced here. Once this merges, #412 can integrate by feeding v2 position-delete files into the same per-file bitmap index consumed by IcebergDvExec, unifying v2 and v3 on one operator. A detailed integration plan is posted on #412.

JanKaul and others added 2 commits September 23, 2026 11:07
Replace IcebergDvExec's manual per-file offset cursor with DataFusion 55's
native RowNumber virtual column, which materializes each row's true absolute
file position. This removes the cursor's documented limitation (misalignment
under repartition_file_scans when a file's row groups split across partitions)
and makes DV filtering correct regardless of scan partitioning, batch order,
or row-group pruning.

- dv_exec: filter via a stateless per-row `dv.is_deleted(row_number)` lookup;
  drop the offsets cursor and the rank/select machinery; strip both internal
  columns (row-number always, path column when not opted in); repartitioning
  is now safe.
- table_scan: add the RowNumber virtual column (with an unknown ColumnStatistics
  entry) to the DV scan schema, thread it through the internal projection, and
  enable predicate pushdown on the DV data scan (safe: supports_filters_pushdown
  is Inexact, so a FilterExec is retained above the scan).
- Tests: rewrite dv_exec unit tests for the row-number model (non-contiguous,
  split/out-of-order, interleaved cases the cursor could not handle) and add an
  in-crate real ParquetSource scan test proving true row numbers survive
  row-group pruning and drive the DV filter end to end.
- Reformat the pre-existing deletion_vector.rs files to satisfy rustfmt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@JanKaul
JanKaul merged commit 732ca30 into main Sep 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant