Cross-implementation conformance: read the Raincloud corpus (247 Vortex files) and publish a compatibility matrix

[dfa1]

Context

@mprammer proposed using Raincloud as a cross-implementation conformance corpus: ~249 curated real-world datasets, 247 of which are built as both Parquet and Vortex (written by the Vortex Python bindings, currently vortex-data==0.69.0). Reading all of them correctly from vortex-java — validated against the Parquet sibling as oracle — would give both projects a real-world compatibility matrix, far stronger evidence than synthetic round-trip fixtures.

Approach

For each slug with a Vortex artifact:

1. Build (or pull) the parquet/<slug>.parquet + vortex/<slug>.vortex pair via the Raincloud pipeline.
2. Read the .vortex with vortex-java (vortex-cli schema / count / export).
3. Oracle check: compare exported values against the Parquet sibling (pyarrow).
4. Record per slug: schema parsed / row count / values match / encodings + layouts encountered / failure detail.

Output: a per-dataset compatibility matrix plus an encoding/layout coverage summary, published under docs/.

Plan

• Proof of concept: 4 small datasets (uci-iris, kepler-exoplanet-search-results, palmer-archipelago-antarctica-penguin-data, countries-of-the-world incl. VARIANT columns) — build, read, compare
• Conformance harness: scripted runner over any set of built slugs, emitting matrix rows (machine-readable + markdown)
• Scale out to the full buildable corpus (small/medium slugs first; large ones — ClickBench, Wikipedia, JSONBench — need multi-hour builds and disk budget)
• Publish compatibility matrix + file issues for each gap found (unknown encodings/layouts, value mismatches)
• CI story: recurring subset run (blocked on mirror access — see open questions)

Open questions for @mprammer

• Is there a mirror (RAINCLOUD_MIRROR) we could read prepared artifacts from, so CI doesn't rebuild datasets from upstream? Resolution order cache → mirror → build makes this the difference between a download and multi-hour rebuilds.
• Should the matrix pin the vortex-data version per run (files written by 0.69.0 today may use different encodings than future releases)?
• A few source URLs may have rotted since the last build — want failures reported here or on the Raincloud tracker?

Progress

Results will be posted as comments on this issue as phases complete.

[dfa1]

Proof-of-concept results (4 small datasets, vortex-data==0.69.0)

Built via the Raincloud pipeline on macOS/arm64; read with vortex-java 0.12.1-SNAPSHOT (vortex-cli); oracle = the Parquet sibling via pyarrow, compared cell-by-cell against vortex-java's CSV export.

dataset	shape	schema	count	full scan / values	detail
uci-iris	150 × 5	✅	✅	✅ 750/750 cells exact	floats + nullable, zero diffs
kepler-exoplanet-search-results	9,564 × 153	✅	✅	❌	lazy dict U8 values — #206
palmer-archipelago-antarctica-penguin-data	344 × 8	✅	✅	❌	lazy dict U16 values — #206
countries-of-the-world	262 × 3	✅	✅	❌	nested struct column — #207

Takeaways:

• Footer/postscript/schema/row-count paths handle everything the Python bindings wrote — all failures are in full-scan decode.
• Real-world data finds gaps synthetic fixtures missed on the very first swing: narrow dict values (2 of 4 datasets!) and struct-of-struct columns. This is exactly the value @mprammer predicted.
• The comparison harness (build → export → pyarrow oracle diff) works end-to-end and is ready to scale out once DictLayoutDecoder: lazy dict rejects U8/U16/I8/I16 value ptypes (hit by real Python-written files) #206 / ScanIterator: nested struct column fails chunk planning ("0 flats" vs widest column) #207 land.

Next: fix the two bugs, re-run these four, then widen to more of the buildable corpus.

[dfa1]

Update: harness scaled to the full catalog + 2 more bugs found

Widening the PoC to 10 hydrated datasets surfaced two additional reader gaps:

• CSV export renders unsigned integers as signed — silent value corruption (U8 132 → -124) #208 — unsigned integers rendered as signed (uci-wine: U8 value 132 exported as -124). Silent corruption, not a crash — every value ≤ 127 is correct, which is why synthetic fixtures never caught it.
• RleEncodingDecoder: unsupported ptype F64 (fastlanes.rle over doubles) #209 — fastlanes.rle has no F64 arm (uci-seoul-bike-sharing-demand).

Current matrix over the triaged slice: 4 ok (uci-iris, uci-seeds, uci-wine-quality, uci-estimation-of-obesity-levels — value-for-value exact against the Parquet oracle), 6 known gaps (#206 ×3, #207, #208, #209), 237 untriaged.

Harness (implemented, pending merge): the checked-in status matrix now enumerates all 247 vortex-enabled slugs; a hydrate script resolves any subset through the raincloud loader (cache → mirror → local build, pinned to v0.2.1, per-slug size budget); the integration test discovers whatever is hydrated — ok entries must match the oracle exactly, gap entries must keep failing (a fix forces the matrix flip in the same change), untriaged entries run and report without failing the build. A weekly scheduled workflow runs a size-capped sweep.

[dfa1]

Triage round 2: 10 more datasets — 13/20 now fully passing

Built and classified 10 new slugs against the fix branches (PRs #212/#213/#214 in flight):

dataset	result
uci-adult (48,842 × 15)	✅ value-for-value
uci-abalone, uci-mushroom, uci-forest-fires, uci-dry-bean-dataset, insurance	✅ value-for-value
international-football-results (49,503 rows, date32)	✅ values exact (CSV renders dates as epoch days — cosmetic)
uci-auto-mpg, uci-heart-disease	❌ #208 (unsigned U8 columns)
uci-magic-gamma-telescope	❌ new bug #215 — IOOBE in VarBinArray.DictMode on narrow (U8) dict-offset ptypes from vortex.sequence offsets

Running matrix: 13 ok, 7 gaps (#207, #208 ×4, #209, #215), 227 untriaged. Three fixes already merged-or-in-review from round 1 (#206 ✅, #210/#211 in PR #212, #208 in PR #213, #209 in PR #214). All 20 triaged slugs are pinned by RaincloudConformanceIntegrationTest — every ok must stay value-exact, every gap must keep failing until its fix flips it.

No rotted URLs in this batch — all 10 fetched cleanly on first attempt.