Skip to content

fix(parquet): read a sign-extended BYTE_ARRAY decimal without panic - #12288

Open
init-js wants to merge 1 commit into
apache:mainfrom
init-js:fix-wide-binary-decimal
Open

init-js wants to merge 1 commit into
apache:mainfrom
init-js:fix-wide-binary-decimal

Conversation

@init-js

@init-js init-js commented Nov 17, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Related issues:

Rationale for this change

A BYTE_ARRAY DECIMAL stores its unscaled value as a big-endian two's complement integer. The spec does not limit its length: it says "the minimum number of bytes to store the unscaled value should be used" (LogicalTypes.md#L241-L242), not must. A writer can therefore sign-extend a value with redundant leading 0x00 or 0xFF bytes, and the file is still valid.

Reading a DECIMAL(38, 0) that holds 9 in 19 bytes panics with:

Array too large, expected less than 16

The same happens for Decimal256 past 32 bytes.

StatisticsConverter has the same assumption. from_bytes_to_i32, from_bytes_to_i64, from_bytes_to_i128 and from_bytes_to_i256 (statistics.rs#L41-L66) sign-extend the min and max bytes to the width of the Arrow type, so row_group_mins, row_group_maxes, data_page_mins and data_page_maxes panic on such a file for every Arrow decimal type.

What changes are included in this PR?

  • bit_util::try_sign_extend_be sign-extends like sign_extend_be, but also accepts input longer than N bytes when the extra leading bytes only repeat the sign. It returns None when b is empty or the value does not fit in N bytes.
  • ByteArrayReader converts Decimal128 and Decimal256 through one helper that uses try_sign_extend_be. Values that fit read back unchanged. A value that does not fit returns BYTE_ARRAY decimal value of {n} bytes is out of range for {data_type} instead of panicking.
  • StatisticsConverter's from_bytes_to_* functions use try_sign_extend_be and return Option. A min or max that does not fit the Arrow type becomes null, as an INT64 statistic that does not fit a Decimal32 already does (statistics.rs#L296-L299, #L1165).

The path for values of normal width is unchanged apart from one length comparison.

The FIXED_LEN_BYTE_ARRAY reader does not hit this panic: the column width selects an Arrow type that holds it, and a narrower schema hint is rejected with an error.

Are these changes tested?

Yes. New tests in arrow::array_reader::byte_array::tests write BYTE_ARRAY decimal columns and read them back:

Also passed the CI tests on the original PR in my fork: init-js#2

  • sign-extended values for Decimal128 (9 and -9 in 19 bytes, i128::MIN in 17 bytes) and Decimal256 (5 in 34 bytes, i256::MIN in 40 bytes), with nulls;
  • values that do not fit Decimal128 or Decimal256, which now return an error.

A new test in tests/arrow_reader/statistics.rs writes sign-extended BYTE_ARRAY decimals with page statistics, and checks the row group and data page mins and maxes for Decimal32, Decimal64, Decimal128 and Decimal256, including a max that only fits the wider types, and an empty min and max, which become null.

All three fail on main and pass with this change. A unit test for try_sign_extend_be covers the boundary cases, where the dropped bytes look like sign bytes but do not match the sign of the kept bytes.

Are there any significant user-facing changes?

No API change.

AI usage disclosure

The fix started from a patch we wrote and use internally at AWS. An AI assistant (Claude) adapted it to current main, extended it to StatisticsConverter, drafted the tests and this description, and ran the tests against main and this branch. I reviewed the changes and chose the behaviour for values that do not fit: a read error in the reader, and a null statistic.

A BYTE_ARRAY decimal is a big-endian two's complement integer of any
length, and a writer may sign-extend it past the fewest bytes it needs.
The Arrow reader and StatisticsConverter converted each value with
sign_extend_be, which panics on more bytes than the Arrow decimal type
holds, so reading a DECIMAL(38, 0) holding 9 in 19 bytes, or its
statistics, panicked. An empty min or max also panicked.

Both now drop leading bytes that only repeat the sign, so such a value
converts unchanged. A value that does not fit the Arrow type is a read
error in the reader, and a null min or max in StatisticsConverter, as
for INT64 statistics that do not fit a Decimal32. An empty min or max
is null too.

@Subodh-17 Subodh-17 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comprehensive Technical Review of PR #12288

I conducted an independent code audit and edge-case verification of PR #12288 (fix(parquet): read a sign-extended BYTE_ARRAY decimal without panic) at HEAD 8156629098a217c5451030aab5e9306f9d2286eb.

1. Specification Compliance & Root-Cause Resolution

  • Parquet Spec Compliance: Under the Parquet format specification (LogicalTypes.md#L241-L242), variable-length BYTE_ARRAY decimals are encoded as big-endian two's complement integers. Upstream engines (such as Trino and Apache Spark) commonly write values with redundant leading sign bytes (0x00 padding for non-negative values or 0xFF padding for negative values).
  • Elimination of Panic Vectors: Previously, sign_extend_be::<N> executed an unconditional assertion assert!(b.len() <= N), crashing readers with a panic whenever the serialized byte count exceeded target type width (e.g., > 16 bytes for Decimal128 or > 32 bytes for Decimal256). PR #12288 safely resolves this via try_sign_extend_be::<N>.

2. Truncation Safety & Error Propagation

  • Mathematical Correctness of Slicing: Slicing at b.len() - N and asserting prefix.iter().all(|&b| b == sign) where sign = if value[0] & 0x80 == 0 { 0 } else { 0xff } correctly guarantees that discarded bytes carry zero magnitude and match the retained sign bit. Any magnitude overflow or sign flip returns None.
  • Catchable Error in ByteArrayReader: Replacing map(sign_extend_be) with try_from_unary gracefully returns ParquetError::General("BYTE_ARRAY decimal value of {len} bytes is out of range for {data_type}"), replacing fatal process crashes with standard catchable query engine errors.
  • Spec-Compliant Handling in StatisticsConverter: Returning Ok(None) when try_sign_extend_be fails safely demotes unconvertible or out-of-range min/max column statistics to null statistics rather than failing query execution or aborting via panic.

3. Verification

  • Verified all author unit tests and evaluated 8 additional adversarial edge cases via bitwise simulation (including 18 redundant sign bytes for both positive and negative values, boundary values i128::MAX with redundant padding, boundary overflow i128::MAX + 1 returning None, required sign-bit preservation [0x00, 0x80], and zero encodings). All simulation checks passed.
  • Fork CI Rust unit and integration tests passed cleanly.

The fix is robust, mathematically sound, and resolves the issue without introducing regressions. LGTM!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet reader panics on a BYTE_ARRAY decimal with redundant sign-extension bytes

2 participants