Repository navigation
Conversation
Subodh-17
approved these changes
Nov 17, 2026
Subodh-17
left a comment
There was a problem hiding this comment.
Comprehensive Technical Review of PR #12294
I performed an independent technical review and code audit of PR #12294 (perf(parquet): keep zstd contexts per thread, not per codec) at HEAD a1fd0f57a72c1fbef3724ecd7b689f6b6d8a76b0.
1. Technical Analysis & Correctness
- Context Lifecycle & State Isolation: Moving
zstd::bulk::CompressorandDecompressorintothread_local!storage cleanly addresses the context recreation overhead identified in #12250. Because libzstd'sZSTD_compressCCtxandZSTD_decompressDCtxautomatically reset session boundaries on each bulk invocation, reusing contexts across column chunks on the same thread produces byte-for-byte identical output to ephemeral contexts. - Concurrency & Thread Safety: By keying compressors by
(thread, level)and decompressors per thread, worker threads in parallel runtimes (e.g., DataFusion / Rayon / Tokio) operate completely lock-free with zero thread contention. - Architectural Simplification: Removing
compressoranddecompressorfields fromZSTDCodecleaves onlylevel: ZstdLevel. This transformsZSTDCodecinto a lightweight,Copy,Send, andSyncstruct (previouslyCompressorpointers preventedSync), significantly improving ergonomics. - Panic-Safety on Re-entrancy: Using
cell.try_borrow_mut()with an explicit fallback to ephemeral context allocation (Err(_) => decompress_with(&mut zstd::bulk::Decompressor::new()?, ...)) ensures that any unexpected re-entrant call will gracefully allocate rather than panic onRefCell::borrow_mut().
2. Test Verification
- The new
test_codec_zstd_shared_contexts_match_fresh_contextstest provides thorough coverage: spawning 4 threads, testing multiple compression levels (1, 3, 9, 19), and asserting both bit-level equality against standalonebulk::Compressorinstances and full round-trip decoding across payloads from 0 B to 3 MiB. - Fork CI passed all checks cleanly.
This is a clean, well-benchmarked, and production-ready optimization that fully resolves #12250. LGTM!
Contributor
|
I think I considered this sometime ago. A primary concern is that So even if there is no parquet data being read / written we still have this allocation around, which is a bit weird for libraries. IMO it would be better if we can change the code to reuse the context more (across chunks / read operations) instead. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
create_codecbuilds oneZSTDCodecper column chunk, and each codec owns itszstd contexts. After #11858 they are created lazily, but a reader still creates
(and frees) one decompression context per column chunk it reads, and a writer
holds one compression context per column. For a query engine reading many small
column chunks, context creation is a measurable share of CPU: in our service
(DataFusion over zstd Parquet), a uprobe counted ~582k
ZSTD_createDCtxcalls in110 s on 59.3.2, about 11% of query CPU.
What changes are included in this PR?
As proposed in #12250: one decompressor per thread and one compressor per
(thread, level), created on first use;
ZSTDCodeckeeps only its level. A zstdbulk call resets its session, so a reused context produces the same bytes as a
fresh one. A re-entrant call (not possible today) falls back to a fresh context
rather than panicking.
Are these changes tested?
New test
test_codec_zstd_shared_contexts_match_fresh_contexts: four threads,codecs at levels 1/3/9/19 interleaved, inputs from 0 B to 3 MiB; compressed bytes
must equal a fresh
bulk::Compressor's, and round-trip. (Negative control: makingthe lookup ignore the level fails it.)
Measured on 59.3.2 + this patch, same machine, alternated rounds:
ZSTD_createDCtx: 581,990 -> 852 per 110 s run;ZSTD_createCCtx: 583,418 -> 2,603.Existing criterion benches, main vs this branch (medians, pinned cores):
compressionZSTD compress/decompress of 1 MiB: -0.8% to -3.2%;arrow_writer*/zstd*: mostly within ±2% in one run; the slower outliers (up to +17%, all*_1pct_cardinality) re-measured alternated within ±0.6%, and two cases 6-9% faster were not re-measured;dictionary_skip: -0.5% / +0.1%. These build one codec per run over few column chunks, so they show the thread-local lookup adds no cost; none reads many small column chunks, which is where the gain is (the context creation counts above). I can add such a benchmark in a separate PR first if that helps review.Are there any significant user-facing changes?
No API change. Contexts now live for the thread's lifetime (bounded by threads x
levels used, not by columns).
AI usage disclosure
The change, the test and this description were drafted with Claude Code (an AI coding assistant), following the fix proposed in #12250; the measurements above were run with its help on our own deployment.