Skip to content

perf(variant): copy leftover object fields by field id when shredding - #12311

Draft
cetra3 wants to merge 4 commits into
apache:mainfrom
pydantic:variant-views-4-shred
Draft

cetra3 wants to merge 4 commits into
apache:mainfrom
pydantic:variant-views-4-shred

Conversation

@cetra3

@cetra3 cetra3 commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

This PR is stacked on #12308, #12309 and #12310. Its own change is the fourth commit, "perf(variant): copy leftover object fields by field id when shredding". It is a draft until they merge.

Rationale for this change

When shred_variant and unshred_variant copy an object field that is not shredded, they decode it into a Variant, look its name up in the metadata dictionary to get the field id back, and encode the value again. The output uses the same metadata dictionary as the input, so the field id and the encoded bytes are already correct.

What changes are included in this PR?

  • ObjectBuilder::insert_encoded adds a field by its field id and its encoded bytes.
  • VariantView::bytes returns the exact encoding of a value.
  • shred_variant and unshred_variant copy leftover object fields through them. They match field names as bytes against a sorted list of the shredded field names.

Are these changes tested?

The existing shredding tests cover the changed code. VariantView::bytes has tests for every primitive type, and for the fields of an object and the elements of a list, whose encodings must tile the value region of their parent.

Benchmarks (local run, this PR against #12310, x86_64 Linux):

benchmark #12310 this PR change
shred_variant_partial_object_8k_rows 10.90 ms 9.42 ms −14%
shred_variant_unmatched_object_8k_rows 14.43 ms 11.36 ms −21%
unshred_variant_list 5.44 ms 5.59 ms +3%
unshred_variant_list_of_objects 11.26 ms 11.46 ms +2%
unshred_variant_nested_fallback 74.62 ms 75.06 ms +1%
unshred_variant_object 2.93 ms 3.05 ms +4%
unshred_variant_partial_object 24.13 ms 21.81 ms −10%
unshred_variant_partial_object_unsorted_metadata 107.97 ms 62.79 ms −42%
unshred_variant_primitive 263.33 µs 260.74 µs −1%
variant_get_unshredded_object_path_262k_rows 14.98 ms 13.61 ms −9%
variant_get_list_index_unshredded 308.11 µs 283.84 µs −8%
variant_get_utf8_from_unshredded_string 214.52 µs 233.12 µs +9%
variant_get_binary_from_string 207.79 µs 226.29 µs +9%

Slower inputs: unshred_variant_object is 4% slower and unshred_variant_list is 3% slower. variant_get_utf8_from_unshredded_string and variant_get_binary_from_string are 9% slower in this run, although this PR does not change variant_get; these two benchmarks have moved by 10–15% with unrelated changes on this machine.

Are there any user-facing changes?

New public API in parquet-variant: ObjectBuilder::insert_encoded and VariantView::bytes.

On malformed variant bytes, shred_variant and the leftover copy in unshred_variant return an error. Before, these paths could panic.

🤖 Generated with Claude Code

cetra3 and others added 4 commits November 18, 2026 12:36
Take `OffsetSizeBytes` by value in `unpack_u32_at_offset`, remove
`unpack_u32` in favor of it, and extract `str_from_bytes` from
`string_from_slice`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Add `VariantView`, `VariantMetadataView`, `VariantObjectView` and
`VariantListView`. Building a view reads one header. Each accessor reads
and bounds-checks only the bytes it needs, and compares field names as
bytes.

`VariantMetadata`, `VariantObject` and `VariantList` are now built by
promoting a view, and their accessors delegate to it.
`ObjectBuilder::finish` sorts fields with the new
`MetadataBuilder::cmp_field_names`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`variant_get` reads each row of an unshredded column through
`VariantView::get_path`. A row reads only the headers and field names
along the path, and decodes only the value that it finds.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`shred_variant` and `unshred_variant` copy each leftover object field by
its field id and encoded bytes, through the new
`ObjectBuilder::insert_encoded` and `VariantView::bytes`. A leftover
field is no longer decoded into a `Variant` and encoded again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet-variant parquet-variant* crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant