Skip to content

perf(parquet): keep zstd contexts per thread, not per codec - #12294

Open
aptro wants to merge 1 commit into
apache:mainfrom
aptro:parquet-zstd-thread-contexts
Open

aptro wants to merge 1 commit into
apache:mainfrom
aptro:parquet-zstd-thread-contexts

Conversation

@aptro

@aptro aptro commented Nov 17, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

create_codec builds one ZSTDCodec per column chunk, and each codec owns its
zstd contexts. After #11858 they are created lazily, but a reader still creates
(and frees) one decompression context per column chunk it reads, and a writer
holds one compression context per column. For a query engine reading many small
column chunks, context creation is a measurable share of CPU: in our service
(DataFusion over zstd Parquet), a uprobe counted ~582k ZSTD_createDCtx calls in
110 s on 59.3.2, about 11% of query CPU.

What changes are included in this PR?

As proposed in #12250: one decompressor per thread and one compressor per
(thread, level), created on first use; ZSTDCodec keeps only its level. A zstd
bulk call resets its session, so a reused context produces the same bytes as a
fresh one. A re-entrant call (not possible today) falls back to a fresh context
rather than panicking.

Are these changes tested?

New test test_codec_zstd_shared_contexts_match_fresh_contexts: four threads,
codecs at levels 1/3/9/19 interleaved, inputs from 0 B to 3 MiB; compressed bytes
must equal a fresh bulk::Compressor's, and round-trip. (Negative control: making
the lookup ignore the level fails it.)

Measured on 59.3.2 + this patch, same machine, alternated rounds:

  • ZSTD_createDCtx: 581,990 -> 852 per 110 s run; ZSTD_createCCtx: 583,418 -> 2,603.
  • Query CPU per poll, small reads (8 columns): 9.63 -> 7.88 ms (-18%); large reads: 81 -> 57 ms (-30%). Latency p50 down in every row.

Existing criterion benches, main vs this branch (medians, pinned cores): compression ZSTD compress/decompress of 1 MiB: -0.8% to -3.2%; arrow_writer */zstd*: mostly within ±2% in one run; the slower outliers (up to +17%, all *_1pct_cardinality) re-measured alternated within ±0.6%, and two cases 6-9% faster were not re-measured; dictionary_skip: -0.5% / +0.1%. These build one codec per run over few column chunks, so they show the thread-local lookup adds no cost; none reads many small column chunks, which is where the gain is (the context creation counts above). I can add such a benchmark in a separate PR first if that helps review.

Are there any significant user-facing changes?

No API change. Contexts now live for the thread's lifetime (bounded by threads x
levels used, not by columns).

AI usage disclosure

The change, the test and this description were drafted with Claude Code (an AI coding assistant), following the fix proposed in #12250; the measurements above were run with its help on our own deployment.

@boring-cyborg boring-cyborg Bot added the parquet Changes to the parquet crate label Nov 17, 2026
@aptro aptro changed the title [Parquet] Keep zstd contexts per thread, not per codec perf(parquet): keep zstd contexts per thread, not per codec Nov 17, 2026

@Subodh-17 Subodh-17 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comprehensive Technical Review of PR #12294

I performed an independent technical review and code audit of PR #12294 (perf(parquet): keep zstd contexts per thread, not per codec) at HEAD a1fd0f57a72c1fbef3724ecd7b689f6b6d8a76b0.

1. Technical Analysis & Correctness

  • Context Lifecycle & State Isolation: Moving zstd::bulk::Compressor and Decompressor into thread_local! storage cleanly addresses the context recreation overhead identified in #12250. Because libzstd's ZSTD_compressCCtx and ZSTD_decompressDCtx automatically reset session boundaries on each bulk invocation, reusing contexts across column chunks on the same thread produces byte-for-byte identical output to ephemeral contexts.
  • Concurrency & Thread Safety: By keying compressors by (thread, level) and decompressors per thread, worker threads in parallel runtimes (e.g., DataFusion / Rayon / Tokio) operate completely lock-free with zero thread contention.
  • Architectural Simplification: Removing compressor and decompressor fields from ZSTDCodec leaves only level: ZstdLevel. This transforms ZSTDCodec into a lightweight, Copy, Send, and Sync struct (previously Compressor pointers prevented Sync), significantly improving ergonomics.
  • Panic-Safety on Re-entrancy: Using cell.try_borrow_mut() with an explicit fallback to ephemeral context allocation (Err(_) => decompress_with(&mut zstd::bulk::Decompressor::new()?, ...)) ensures that any unexpected re-entrant call will gracefully allocate rather than panic on RefCell::borrow_mut().

2. Test Verification

  • The new test_codec_zstd_shared_contexts_match_fresh_contexts test provides thorough coverage: spawning 4 threads, testing multiple compression levels (1, 3, 9, 19), and asserting both bit-level equality against standalone bulk::Compressor instances and full round-trip decoding across payloads from 0 B to 3 MiB.
  • Fork CI passed all checks cleanly.

This is a clean, well-benchmarked, and production-ready optimization that fully resolves #12250. LGTM!

@Dandandan

Copy link
Copy Markdown
Contributor

I think I considered this sometime ago.

A primary concern is that thread_local "leaks" this allocation / context per thread.

So even if there is no parquet data being read / written we still have this allocation around, which is a bit weird for libraries.

IMO it would be better if we can change the code to reuse the context more (across chunks / read operations) instead.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Parquet] Every column writer's ZSTDCodec owns its own zstd context: a wide schema holds thousands of them at once

4 participants