Repository navigation
Conversation
A BYTE_ARRAY decimal is a big-endian two's complement integer of any length, and a writer may sign-extend it past the fewest bytes it needs. The Arrow reader and StatisticsConverter converted each value with sign_extend_be, which panics on more bytes than the Arrow decimal type holds, so reading a DECIMAL(38, 0) holding 9 in 19 bytes, or its statistics, panicked. An empty min or max also panicked. Both now drop leading bytes that only repeat the sign, so such a value converts unchanged. A value that does not fit the Arrow type is a read error in the reader, and a null min or max in StatisticsConverter, as for INT64 statistics that do not fit a Decimal32. An empty min or max is null too.
Subodh-17
approved these changes
Nov 17, 2026
Subodh-17
left a comment
There was a problem hiding this comment.
Comprehensive Technical Review of PR #12288
I conducted an independent code audit and edge-case verification of PR #12288 (fix(parquet): read a sign-extended BYTE_ARRAY decimal without panic) at HEAD 8156629098a217c5451030aab5e9306f9d2286eb.
1. Specification Compliance & Root-Cause Resolution
- Parquet Spec Compliance: Under the Parquet format specification (
LogicalTypes.md#L241-L242), variable-lengthBYTE_ARRAYdecimals are encoded as big-endian two's complement integers. Upstream engines (such as Trino and Apache Spark) commonly write values with redundant leading sign bytes (0x00padding for non-negative values or0xFFpadding for negative values). - Elimination of Panic Vectors: Previously,
sign_extend_be::<N>executed an unconditional assertionassert!(b.len() <= N), crashing readers with a panic whenever the serialized byte count exceeded target type width (e.g., > 16 bytes forDecimal128or > 32 bytes forDecimal256). PR #12288 safely resolves this viatry_sign_extend_be::<N>.
2. Truncation Safety & Error Propagation
- Mathematical Correctness of Slicing: Slicing at
b.len() - Nand assertingprefix.iter().all(|&b| b == sign)wheresign = if value[0] & 0x80 == 0 { 0 } else { 0xff }correctly guarantees that discarded bytes carry zero magnitude and match the retained sign bit. Any magnitude overflow or sign flip returnsNone. - Catchable Error in
ByteArrayReader: Replacingmap(sign_extend_be)withtry_from_unarygracefully returnsParquetError::General("BYTE_ARRAY decimal value of {len} bytes is out of range for {data_type}"), replacing fatal process crashes with standard catchable query engine errors. - Spec-Compliant Handling in
StatisticsConverter: ReturningOk(None)whentry_sign_extend_befails safely demotes unconvertible or out-of-range min/max column statistics to null statistics rather than failing query execution or aborting via panic.
3. Verification
- Verified all author unit tests and evaluated 8 additional adversarial edge cases via bitwise simulation (including 18 redundant sign bytes for both positive and negative values, boundary values
i128::MAXwith redundant padding, boundary overflowi128::MAX + 1returningNone, required sign-bit preservation[0x00, 0x80], and zero encodings). All simulation checks passed. - Fork CI Rust unit and integration tests passed cleanly.
The fix is robust, mathematically sound, and resolves the issue without introducing regressions. LGTM!
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related issues:
BYTE_ARRAYdecimal reader and inStatisticsConverter. It doesn't add the fuzzer that the issue proposes.BYTE_ARRAYDECIMALcolumns #10860 / fix: Avoid incorrect min/max stats for BYTE_ARRAY decimals of unequal byte lengths #10861 fixed wrong min/max statistics forBYTE_ARRAYdecimals on the write side. This is the read side, and a panic rather than a wrong value.FIXED_LEN_BYTE_ARRAYdecimals inferred asDecimal256) and Parquet reader forces Decimal32/64 columns to Decimal128 #8549 (INT32/INT64decimals widened toDecimal128) are about which Arrow type the reader picks, so they're unaffected by this change.Rationale for this change
A
BYTE_ARRAYDECIMALstores its unscaled value as a big-endian two's complement integer. The spec does not limit its length: it says "the minimum number of bytes to store the unscaled value should be used" (LogicalTypes.md#L241-L242), not must. A writer can therefore sign-extend a value with redundant leading0x00or0xFFbytes, and the file is still valid.Reading a
DECIMAL(38, 0)that holds9in 19 bytes panics with:The same happens for
Decimal256past 32 bytes.StatisticsConverterhas the same assumption.from_bytes_to_i32,from_bytes_to_i64,from_bytes_to_i128andfrom_bytes_to_i256(statistics.rs#L41-L66) sign-extend the min and max bytes to the width of the Arrow type, sorow_group_mins,row_group_maxes,data_page_minsanddata_page_maxespanic on such a file for every Arrow decimal type.What changes are included in this PR?
bit_util::try_sign_extend_besign-extends likesign_extend_be, but also accepts input longer thanNbytes when the extra leading bytes only repeat the sign. It returnsNonewhenbis empty or the value does not fit inNbytes.ByteArrayReaderconvertsDecimal128andDecimal256through one helper that usestry_sign_extend_be. Values that fit read back unchanged. A value that does not fit returnsBYTE_ARRAY decimal value of {n} bytes is out of range for {data_type}instead of panicking.StatisticsConverter'sfrom_bytes_to_*functions usetry_sign_extend_beand returnOption. A min or max that does not fit the Arrow type becomes null, as anINT64statistic that does not fit aDecimal32already does (statistics.rs#L296-L299, #L1165).The path for values of normal width is unchanged apart from one length comparison.
The
FIXED_LEN_BYTE_ARRAYreader does not hit this panic: the column width selects an Arrow type that holds it, and a narrower schema hint is rejected with an error.Are these changes tested?
Yes. New tests in
arrow::array_reader::byte_array::testswriteBYTE_ARRAYdecimal columns and read them back:Also passed the CI tests on the original PR in my fork: init-js#2
Decimal128(9 and -9 in 19 bytes,i128::MINin 17 bytes) andDecimal256(5 in 34 bytes,i256::MINin 40 bytes), with nulls;Decimal128orDecimal256, which now return an error.A new test in
tests/arrow_reader/statistics.rswrites sign-extendedBYTE_ARRAYdecimals with page statistics, and checks the row group and data page mins and maxes forDecimal32,Decimal64,Decimal128andDecimal256, including a max that only fits the wider types, and an empty min and max, which become null.All three fail on
mainand pass with this change. A unit test fortry_sign_extend_becovers the boundary cases, where the dropped bytes look like sign bytes but do not match the sign of the kept bytes.Are there any significant user-facing changes?
No API change.
AI usage disclosure
The fix started from a patch we wrote and use internally at AWS. An AI assistant (Claude) adapted it to current
main, extended it toStatisticsConverter, drafted the tests and this description, and ran the tests againstmainand this branch. I reviewed the changes and chose the behaviour for values that do not fit: a read error in the reader, and a null statistic.