Skip to content

perf(variant): read unshredded columns through views in variant_get - #12310

Draft
cetra3 wants to merge 3 commits into
apache:mainfrom
pydantic:variant-views-3-get
Draft

cetra3 wants to merge 3 commits into
apache:mainfrom
pydantic:variant-views-3-get

Conversation

@cetra3

@cetra3 cetra3 commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

This PR is stacked on #12308 and #12309. Its own change is the third commit, "perf(variant): read unshredded columns through views in variant_get". It is a draft until they merge.

Rationale for this change

For a path over unshredded data, variant_get builds a Variant for every row and then walks the path through it. Building the Variant checks the first and last offsets of the metadata and of each object and list, and each field lookup validates the UTF-8 of the names that it probes. Rows that do not contain the path do all of that and produce a null.

What changes are included in this PR?

  • VariantView::get_path follows a path of field names and list indexes through the views of feat(variant): add header-only views of encoded variant bytes #12309.
  • variant_get reads a column that has no typed_value through it, for field and index paths and for typed and Variant output. A row reads only the headers and names along the path, and decodes only the value that it finds.
  • A crate-private BinaryColumn resolves the array type of the metadata and value columns once, not on every row.

The gain comes from skipping the per-row Variant construction and the UTF-8 validation of probed names.

Are these changes tested?

The existing variant_get tests cover the new path, because their unshredded input now takes it. VariantView::get_path has a test that compares it with Variant::get_path.

Benchmarks (local run, this PR against #12309, x86_64 Linux):

benchmark #12309 this PR change
variant_get_binary_from_string 316.95 µs 207.79 µs −34%
variant_get_binary_from_unshredded_binary 264.24 µs 194.71 µs −26%
variant_get_list_index_all_oob_int64_64_rows 421.86 ns 418.12 ns −1%
variant_get_list_index_shredded 26.80 µs 25.72 µs −4%
variant_get_list_index_unshredded 576.96 µs 308.11 µs −47%
variant_get_shredded_utf8 342.91 ns 341.04 ns −1%
variant_get_unshredded_object_path_262k_rows 22.79 ms 14.98 ms −34%
variant_get_utf8_from_date 386.90 µs 306.39 µs −21%
variant_get_utf8_from_decimal 265.38 µs 186.75 µs −30%
variant_get_utf8_from_float 255.41 µs 176.07 µs −31%
variant_get_utf8_from_int 256.21 µs 179.43 µs −30%
variant_get_utf8_from_list 369.64 µs 323.23 µs −13%
variant_get_utf8_from_map_in_list 367.95 µs 328.31 µs −11%
variant_get_utf8_from_time 452.92 µs 370.30 µs −18%
variant_get_utf8_from_timestamp_with_timezone 320.17 µs 241.92 µs −24%
variant_get_utf8_from_timestamp_without_timezone 318.81 µs 243.21 µs −24%
variant_get_utf8_from_unshredded_string 321.26 µs 214.52 µs −33%

No benchmark in this set is slower. The shredded benchmarks (variant_get_shredded_utf8, variant_get_list_index_shredded, variant_get_list_index_all_oob_int64_64_rows) do not take the new path.

Are there any user-facing changes?

New public API in parquet-variant: VariantView::get_path.

On malformed variant bytes, variant_get on an unshredded column returns an error, or a null under safe casting. Before, this path could panic.

🤖 Generated with Claude Code

cetra3 and others added 3 commits November 18, 2026 12:36
Take `OffsetSizeBytes` by value in `unpack_u32_at_offset`, remove
`unpack_u32` in favor of it, and extract `str_from_bytes` from
`string_from_slice`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Add `VariantView`, `VariantMetadataView`, `VariantObjectView` and
`VariantListView`. Building a view reads one header. Each accessor reads
and bounds-checks only the bytes it needs, and compares field names as
bytes.

`VariantMetadata`, `VariantObject` and `VariantList` are now built by
promoting a view, and their accessors delegate to it.
`ObjectBuilder::finish` sorts fields with the new
`MetadataBuilder::cmp_field_names`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`variant_get` reads each row of an unshredded column through
`VariantView::get_path`. A row reads only the headers and field names
along the path, and decodes only the value that it finds.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet-variant parquet-variant* crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant