fix: render nested struct columns as JSON object cells in CSV export (#217)#222
Merged
Conversation
…217) A nested struct column (StructArray, decodable since #207) now renders as one JSON object cell {"field":value,...} with fields in declared order: numbers/booleans unquoted via the shared leaf rules, strings JSON-escaped and quoted, nested structs recursed, and null fields as JSON null (inside JSON the bare-CSV empty-field convention would be ambiguous). Only the JSON layer escapes; fastcsv independently quotes the cell. VortexWriter.arrayLength learns StructData so struct columns can be written in tests. Pinned by countries-of-the-world (Raincloud corpus, #205): its data struct column blocked CSV export after the #207 scan fix. Closes #217 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8 ok (uci-car-evaluation, uci-air-quality, google-cluster-trace-2011, uci-census-income, uci-bank-marketing, uci-sms-spam-collection, uci-optical-recognition, uci-human-activity-recognition) plus 2 new gaps pinned as gap:221 (misaligned per-column chunk grids: uci-beijing-multi-site-air-quality, emotions-dataset-for-nlp). countries-of-the-world flips to ok with the #217 fix (its conformance run aborts on the documented hardwood-oracle nested-parquet limitation rather than verifying, visible as a skip). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #217, the last gap split from #207: with nested struct columns now decodable, CSV export threw
unsupported array type: StructArray.Format decision: a struct cell renders as one JSON object
{"field":value,...}with fields in declared order — numbers/booleans unquoted via the shared leaf rules (incl. the unsigned gating), strings JSON-escaped and double-quoted (minimal correct escaper: quote, backslash, short escapes,\uXXXXfor control chars — no new dependency), nested structs recursed, and null fields as JSONnull(inside JSON, bare CSV's empty-field null convention would be ambiguous). Only the JSON layer escapes; fastcsv independently quotes the cell.VortexWriter.arrayLengthlearnsStructDataso tests can write struct columns.Validation: 14/14
CsvExporterTest(mixed-type struct, null struct field as JSON null, quote/comma escaping);countries-of-the-worldexports all 262 rows with well-formed JSON cells and flips tookin the matrix (its conformance run shows the documented oracle-abort skip — hardwood cannot read its nested parquet).Also carries triage round 3 (second commit): 10 more datasets against a fully-fixed build — 8 pass exactly (categoricals at 48k rows, timestamps, wide numerics, text; a float32-vs-double artifact in the scratchpad compare script was verified bitwise-identical with numpy), 2 pinned as the new gap #221 (misaligned per-column chunk grids, two severity tiers documented in the issue). Matrix: 28 ok / 2 gaps (#221) / 217 untriaged; conformance suite 31/31 over all 30 hydrated slugs.
Closes #217
🤖 Generated with Claude Code