Skip to content

fix(wal): select bounded Parquet 60 through vendored WAL3 - #82

Draft
jratliff79 wants to merge 3 commits into
cachix:mainfrom
jratliff79:codex/wal3-parquet60-bounded-20261114
Draft

jratliff79 wants to merge 3 commits into
cachix:mainfrom
jratliff79:codex/wal3-parquet60-bounded-20261114

Conversation

@jratliff79

Copy link
Copy Markdown
Contributor

Purpose

The current all-features WAL graph selects Parquet 55.2 and Thrift 0.17. This draft proposes a reproducible WAL3 → Parquet 60.1 source path with bounded decoding of malformed fragments. It is based directly on cachix/casita main 44cfd5c5ac6d635e327393b473071861b7f7c209 and contains none of #55, #69, #76, #77, or #81.

Changes

  • Select a local WAL3 source mirror directly from Casita, and select a locally repaired Parquet 60.1 source directly from WAL3. Downstream Git consumers therefore resolve the same sources; no root Cargo patch or dependency-policy exception is used.
  • Preserve the Chroma WAL3 production source from the pinned integration revision. The standalone manifest selects Arrow/Parquet 60.1; vendor/README.md records source revisions, packaging changes, licenses, and compressed patches for the bounded Parquet decoder and historical ZSTD window repair.
  • Add frozen Linux child-process controls for valid, truncated, unbacked-statistics, and unbacked-decompressed-size fragments under an unchanged 256 MiB address-space limit. Keep eight historical Snappy/ZSTD codec cases in Casita's test graph and check repeated current-fragment truncation at the Casita metadata boundary.

Verification on this exact head

  • The old graph passed two and failed two frozen 256 MiB allocation controls; this head passes all four on ARM64 Linux with Rust 1.94.1: cargo test -p casita --features s3 --locked --test wal_allocation -- --test-threads=1.
  • cargo test -p casita --all-features --locked -- --test-threads=4 passed on ARM64 macOS with Rust 1.99.0, the repository's RustFS 1.0.1 binary, and protoc: 992 library tests passed, 57 ignored, zero failed; all executed integration targets passed, including eight historical codec cases; 11 doctests passed. The Linux-only allocation target runs zero cases on macOS and is covered by the separate Linux command above.
  • A fresh clone resolved local WAL3 0.1.0, local Parquet 60.1.0, Arrow 60.1.0, and no Thrift. It generated the same ignored root lockfile hash as the source worktree. The unchanged cargo-deny --all-features --locked check all passed advisories, bans, licenses, and sources. cargo fmt --all --check, git diff --check, and frozen fixture hashes passed. A separate Git consumer pinned to this commit resolved Casita, WAL3, and Parquet from that one Git revision.

Draft review boundary

This adds 377 files and about 211,000 lines, mostly copied package source. Please review whether this source ownership and update path is acceptable before treating the PR as ready. Registry advisory tools may not match path sources by registry identity, so the copied Parquet and WAL3 source and their upstream versions need explicit advisory review.

The local Linux container's 11.66 GiB limit killed the full all-features link, including with one Cargo build job; it did not reach that suite's tests. Hosted exact-head Linux/Windows/macOS checks remain necessary. S3 short-read/retry, live mixed-version rollback, concurrent peak RSS, and workload qualification are also open.

This draft does not merge or replace the contributor PRs, change release policy, relax integrity checks or thresholds, adopt a pilot pin, or start paid workloads. The paid-run pause stays in place. Fable 5.2 reviewed the dependency route independently; Codex checked the complete source selection, red/green controls, policy, and test results. AI review, hosted CI, and maintainer approval are separate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant