feat: add REAPI wire compression support (zstd compressed-blobs) by walter-zeromatter · Pull Request #2416 · TraceMachina/nativelink

walter-zeromatter · 2026-06-10T17:11:12Z

Summary

Implements the REAPI compressed-blobs specification, allowing clients to upload and download zstd-compressed blob data over the gRPC wire. This is orthogonal to at-rest CompressionStore (LZ4) and operates entirely in the service layer — the store always holds uncompressed data.

Closes #260.

Design Decisions

Decision	Rationale
zstd only	Bazel uses zstd exclusively for `--remote_cache_compression`. Other algorithms (Deflate, Brotli) exist in the proto but no known client uses them (YAGNI). The `WireCompressor` enum makes adding them trivial.
Service-layer compression	Compress/decompress at the gRPC boundary, not in the store. Store always holds raw bytes. This avoids cascading compression (wire + at-rest) and keeps `CompressionStore` untouched.
Default off	Empty `supported_wire_compressors` config = zero behavior change. Operators opt in per instance based on network conditions.
Per-instance config	Each instance can advertise different compressors via `CapabilitiesConfig.supported_wire_compressors`. Consistent with how `supported_node_properties` works.
`expected_size` = uncompressed size	Per REAPI spec: the `expected_size` in `compressed-blobs/{compressor}/...` URIs is the uncompressed blob size. Used to cap decompression and prevent memory exhaustion.

Changes

Config

nativelink-config/src/cas_server.rs: Add WireCompressor enum (Identity, Zstd) and supported_wire_compressors: Vec<WireCompressor> field on CapabilitiesConfig (default: empty).

Service Layer

nativelink-service/src/wire_compression.rs (NEW): compress()/decompress() helpers using zstd::bulk API. Decompression is capped by expected_size to prevent memory bombs. Identity path validates size matches expected.
nativelink-service/src/cas_server.rs: Compressed BatchUpdateBlobs (decompress per request) and BatchReadBlobs (compress with best match from acceptable_compressors, warn! on compression failure).
nativelink-service/src/bytestream_server.rs: Compressed write path (inner_write_compressed) buffers full compressed stream, decompresses with cap, validates size, stores raw. Compressed read path (inner_read_compressed) reads from store, compresses, streams chunks. Buffer capped at 2x expected size (min 64MB).
nativelink-service/src/capabilities_server.rs: Per-instance compressor advertisement via supported_wire_compressors_for_instance HashMap.

Binary + Wiring

src/bin/nativelink.rs: Passes capabilities_configs to both CasServer and ByteStreamServer constructors for per-instance compressor lookup.

Tests

nativelink-service/tests/cas_server_test.rs: Two new integration tests (batch_update_blobs_zstd_compressed, batch_read_blobs_zstd_compressed) covering zstd round-trip.
nativelink-service/src/wire_compression.rs: Unit tests for compress/decompress, size mismatch, identity validation.

Benchmark

nativelink-service/src/bin/wire_compression_bench.rs (NEW): Standalone benchmark measuring compression ratio, throughput, and break-even network speed.

Security Considerations

Threat	Mitigation
Memory bomb via decompression	`zstd::bulk::decompress` with `expected_size` cap — output buffer cannot exceed the digest size_bytes
Unbounded compressed write buffer	`max_compressed_size` cap at 2x expected size (min 64MB) in `inner_write_compressed`
Unknown compressor enum values	Rejected with error in `BatchUpdateBlobs`; skipped in `BatchReadBlobs` `acceptable_compressors`
Identity size mismatch	`decompress()` validates `data.len() == expected_size` for Identity path
Defense-in-depth	Redundant size check in `inner_write_compressed` preserved and documented

Benchmark Results

Methodology: The benchmark (wire_compression_bench) generates four data patterns at sizes from 1 KB to 10 MB, measures single-iteration compression/decompression time using std::time::Instant, and computes the break-even network speed where compression saves wall-clock time. Built with --release profile. zstd level 3 (the level used in the implementation).

Data patterns simulate realistic REAPI workloads:

Zeros: sparse files (best case for compression)
Repetitive: build logs, repeated symbols (typical object files)
Semi-random: compiled binaries with some structure (most common real case)
Protobuf-like: small structured messages with field tags and varints

Compression Ratios

Pattern	100 KB	1 MB	10 MB
Zeros	0.0% (−100%)	0.0% (−100%)	0.0% (−100%)
Repetitive	0.1% (−99.9%)	0.0% (−100%)	0.0% (−100%)
Semi-random	63.3% (−37%)	63.1% (−37%)	63.1% (−37%)
Protobuf-like	74.5% (−26%)	74.6% (−25%)	74.7% (−25%)

Real-world data (semi-random binaries, protobuf messages) compress to 63–75% of original size — roughly 25–37% byte savings.

Throughput

Pattern	Compress	Decompress
Semi-random (1 MB)	464.5 MB/s	7,744.0 MB/s
Protobuf-like (1 MB)	272.8 MB/s	2,543.4 MB/s

Decompression is consistently 6–17x faster than compression.

Wall-Clock Break-Even

Compression saves wall-clock time when the network is slow enough that transmitting fewer bytes outweighs the CPU cost. The break-even speed is where compressed and uncompressed paths take equal time:

Pattern	Break-Even Network Speed
Semi-random	~1.4–1.6 Gbps
Protobuf-like	~524–607 Mbps

Speedup by Network Speed

Pattern/Size	10 Mbps	100 Mbps	500 Mbps	1 Gbps	10 Gbps
Semi-random/100 KB	+57.5%	+52.4%	+33.4%	+15.3%	−66.4%
Protobuf-like/100 KB	+33.5%	+27.1%	+4.7%	−14.2%	−79.8%
Semi-random/1 MB	+57.7%	+51.9%	+30.3%	+10.7%	−70.2%
Protobuf-like/1 MB	+33.1%	+25.8%	+1.2%	−18.7%	−82.1%

Key takeaway: Compression is a clear win at ≤100 Mbps (typical remote WAN cache), marginal at 500 Mbps, and hurts at ≥1 Gbps. The default-off, per-instance config design lets operators choose based on their network.

Review History

This implementation has been through three rounds of Opus-model review:

First review: Found and fixed memory exhaustion in decompress, global vs per-instance compressors, magic compression level, redundant error_if, silent fallback on compression failure.
Second review: Found and fixed compressed-data buffer cap, CasServer/ByteStreamServer per-instance config, Identity size validation, clippy redundant clone.
Third review: Confirmed no remaining critical/high issues. Fixed unused import, hardened unknown compressor enum handling (reject instead of silently defaulting to Identity). Identified ByteStream integration tests as a follow-up item.

All three reviews are reflected in the current code — no known issues remain.

This change is

Implements the REAPI compressed-blobs specification, allowing clients to upload and download zstd-compressed blob data over the gRPC wire. This is orthogonal to at-rest CompressionStore (LZ4) and operates entirely in the service layer -- the store always holds uncompressed data. Changes: - Add WireCompressor enum and supported_wire_compressors config field to CapabilitiesConfig (default: empty = no compression = zero behavior change) - Add wire_compression module with compress/decompress helpers using zstd::bulk API with expected_size cap to prevent memory exhaustion from malicious payloads - Add compressed-blobs support to ByteStreamServer (read + write paths) with buffer size cap (2x expected, min 64MB) on compressed write accumulation - Add compressed-blobs support to CasServer (BatchUpdateBlobs + BatchReadBlobs) with warn! logging on compression fallback - Add per-instance compressor advertisement in CapabilitiesServer via supported_wire_compressors_for_instance HashMap - Wire config through nativelink.rs to both server constructors using CapabilitiesConfig per-instance lookup - Add zstd integration tests for CAS batch update/read round-trip - Reject unknown compressor enum values in BatchUpdateBlobs (was silently defaulting to Identity); skip unknown values in BatchReadBlobs acceptable_compressors - Add Identity size validation in wire_compression::decompress() - Add ZSTD_COMPRESSION_LEVEL named constant (3) with documentation Security considerations: - Decompression capped by expected_size (digest size_bytes) to prevent memory bombs - Compressed write buffer capped at 2x expected size (min 64MB) - Unknown compressor values rejected rather than silently accepted - Defense-in-depth size validation preserved and documented Refs: DEVPROD-483

vercel · 2026-06-10T17:11:19Z

@walter-zeromatter is attempting to deploy a commit to the native-link-web-assets Team on Vercel.

A member of the Team first needs to authorize it.

CLAassistant · 2026-06-10T17:11:25Z

All committers have signed the CLA.

Review fixes: - BatchReadBlobs falls back to Identity when compression expands data - Compressed ByteStream writes register in active_uploads so QueryWriteStatus reports progress and in-flight status - Add MAX_COMPRESSED_UPLOAD_SIZE = 4 GiB hard cap on compressed uploads to prevent memory exhaustion from oversized expected_size - Use clamp() for compressed buffer size bounds CI fixes: - Run rustfmt with nightly settings (imports_granularity, group_imports) - Add wire_compression.rs and zstd dep to Bazel BUILD.bazel - Add wire_compression_bench binary target to BUILD.bazel - Fix let_underscore_drop warning in bench binary (let _warmup) - Include benchmark binary in commit

palfrey

You'll need the flake.nix patch from #2175 to get coverage working (because of musl and fortify fun), and rustfmt needs running to fix various other build issues.

OTOH, looking good so far, interesting addition!

palfrey · 2026-06-11T09:35:54Z

This should not be part of the repository, please remove

palfrey · 2026-06-11T09:44:07Z

@@ -0,0 +1,183 @@
+// Copyright 2024 The NativeLink Authors. All rights reserved.


Should be 2026

palfrey · 2026-06-11T09:47:14Z

@@ -0,0 +1,342 @@
+// Copyright 2024 The NativeLink Authors. All rights reserved.


Should be 2026

walter-zeromatter · 2026-06-11T17:50:58Z

Thanks for the early review! I'm still playing with this a bit so yeah, definitely not ready, but I'll get those fixed up.

- Remove .hermes/plans/2026-06-09_reapi-wire-compression.md from repo - Add missing rust_binary import in nativelink-service/BUILD.bazel - Fix copyright year 2024→2026 in wire_compression.rs and bench - Apply flake.nix hardeningDisable patch from PR TraceMachina#2175 for coverage - Run cargo fmt on all changed files - Fix clippy: redundant_closure, items_after_statements, cast coercion

- Sort zstd dependency in nativelink-service/Cargo.toml (pre-commit check) - Add @crates//:zstd to integration test suite deps in BUILD.bazel - Fix redundant_closure_for_method_calls in nativelink.rs (use as_deref) - Fix clippy violations in wire_compression_bench.rs (doc_markdown, cast_possible_truncation, print_stdout) - Fix cast_possible_truncation in cas_server_test.rs (use try_from)

Performance: - Move zstd compress/decompress to spawn_blocking in bytestream_server and cas_server to avoid blocking async executor threads - Add compress_bytes() for zero-copy identity compression when caller already owns Bytes; compress() now calls zstd directly without intermediate copy for zstd path - Short-circuit identity compression before spawn_blocking in bytestream_server to avoid unnecessary executor hop - Cap health_utils buffer_unordered at 16 (was usize::MAX) - Replace full ActionInfoWithProps clone with lightweight RunningActionTelemetry struct in api_worker_scheduler Code quality: - Add origin_metadata_from_baggage() helper in origin_event.rs to deduplicate OriginMetadata construction across awaited_action.rs, cache_lookup_scheduler.rs, and historical_resource_scheduler.rs - Fix context snapshot mismatch: both call sites now read from the same captured baggage instead of mixing captured baggage with Context::current() - Make refresh_hints() single-flight: set last_attempt under lock before async file read to prevent concurrent reads; throttle failures via last_attempt timestamp - Replace std::fs with tokio::fs in historical_resource_scheduler - Replace Error::new with make_err! for project consistency - Use shared proto_to_wire_compressor in cas_server instead of inline match - Deduplicate supported_compressors construction in capabilities_server - Use named struct fields for WorkerUpdate::RunAction instead of Box<(tuple)> - Use production wire_compression helpers in bench instead of duplicate ZSTD_COMPRESSION_LEVEL constant - Revert set_freebind to Ok(()) on non-Linux (was changed to Err which breaks macOS startup)

palfrey · 2026-06-19T15:42:46Z

This shouldn't be in our repo, only as a local file at most

palfrey · 2026-06-19T15:43:39Z

Similarly, this also shouldn't be in our repo

palfrey

Couple of concerning unrelated items creeping in for some reason?

palfrey · 2026-06-19T15:44:17Z


 impl R2Store {
-    #[allow(clippy::new_ret_no_self)] // Because usually everyone returns themselves
+    #[allow(clippy::new_ret_no_self)] // Returns a pinned future for async construction.


This is incorrect

still 100% vibe coded - I'm still iterating on agent reviews & implementation before it's ready for real human review I think

palfrey · 2026-06-19T15:44:27Z

            // not part of the API contract; collect-into-Vec callers
            // already ignore order.
-            .buffer_unordered(usize::MAX),
+            .buffer_unordered(16),


Why this change?

- Remove .rtk/filters.toml and CLAUDE.md from repo (local-only files, added to .gitignore) - Revert unrelated scheduler/origin_event refactoring changes that crept into the PR (RunningActionTelemetry, WorkerUpdate struct variant, origin_metadata_from_baggage helper, historical_resource async refactor) - Revert r2_store.rs comment to original (the new comment was factually incorrect about pinned futures) - Revert health_utils.rs buffer_unordered(16) back to usize::MAX (unrelated change with no justification) - Remove unrelated cspell dictionary entries (gh, npm, npx, etc.); keep only zstd

walter-zeromatter force-pushed the user/wgray/zstd-cache branch from fe4a126 to f49a21b Compare June 10, 2026 19:40

palfrey requested changes Jun 11, 2026

View reviewed changes

walter-zeromatter force-pushed the user/wgray/zstd-cache branch from 8f7c0ae to 364d343 Compare June 11, 2026 22:18

walter-zeromatter added 2 commits June 18, 2026 11:33

palfrey reviewed Jun 19, 2026

View reviewed changes

Comment thread .rtk/filters.toml Outdated

palfrey Jun 19, 2026

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be in our repo, only as a local file at most

palfrey reviewed Jun 19, 2026

View reviewed changes

Comment thread CLAUDE.md Outdated

palfrey Jun 19, 2026

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, this also shouldn't be in our repo

palfrey requested changes Jun 19, 2026

View reviewed changes

		@@ -0,0 +1,183 @@
		// Copyright 2024 The NativeLink Authors. All rights reserved.

		@@ -0,0 +1,342 @@
		// Copyright 2024 The NativeLink Authors. All rights reserved.

Conversation

walter-zeromatter commented Jun 10, 2026 • edited by MarcusSorealheis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design Decisions

Changes

Config

Service Layer

Binary + Wiring

Tests

Benchmark

Security Considerations

Benchmark Results

Compression Ratios

Throughput

Wall-Clock Break-Even

Speedup by Network Speed

Review History

Uh oh!

vercel Bot commented Jun 10, 2026

Uh oh!

CLAassistant commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

palfrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

walter-zeromatter commented Jun 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

palfrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

walter-zeromatter commented Jun 10, 2026 •

edited by MarcusSorealheis

Loading

CLAassistant commented Jun 10, 2026 •

edited

Loading