Skip to content

perf: scan the values buffer in CheckOverflow when there are no nulls (1.4x faster) - #7658

Open
comphead wants to merge 1 commit into
apache:mainfrom
comphead:checkoverflow-values-scan
Open

comphead wants to merge 1 commit into
apache:mainfrom
comphead:checkoverflow-values-scan

Conversation

@comphead

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

No issue. Found while profiling TPC-H Q1 at SF10. CheckOverflow wraps every decimal + - * there, and its no-overflow scan took about 5% of the native CPU samples.

Rationale for this change

Optimize existing expression. The no-overflow check iterated Options with .iter().flatten().all(). That reads the null bitmap for every slot and branches on every value, so the loop does not unroll.

What changes are included in this PR?

  • When the array has no nulls, scan the values buffer directly. Shifting by the precision's minimum turns the two bounds checks into one unsigned comparison per value. There is no branch inside each 64-value chunk, and the scan still stops at the first chunk that holds an overflow. Arrays with nulls keep the existing scan.
  • Benchmark: add no overflow, dense nulls and ansi no overflow, with nulls shapes.
  • Expression audit: record the tuning pass under CheckOverflow.

How are these changes tested?

Existing tests cover the no-null path for positive and negative overflow, the precision boundary, and ANSI errors. A new test covers an overflow past the first chunk and a sliced array with a non-zero offset.

Criterion, cargo bench --bench check_overflow -- --baseline main, 8192 rows, Apple M-series:

Shape main this PR change
no overflow 4.354 µs 3.039 µs −30.1%
no overflow, with nulls 4.807 µs 4.758 µs −1.2%
no overflow, dense nulls 3.990 µs 3.884 µs −0.9%
sparse overflow 4.315 µs 4.280 µs −0.6%
dense overflow 7.614 µs 7.621 µs −0.1% (p = 0.50)
ansi no overflow 4.405 µs 3.028 µs −31.1%
ansi no overflow, with nulls 4.872 µs 4.747 µs −2.7%

The machine was shared (load average about 4.5), so I took several samples. The two no-null shapes were 29% to 31% faster in every sample. Without #[inline] on the helper, dense nulls read +1.6% to +2.3% in three samples. With it, that shape and every other unchanged shape stayed within −2.7% to −0.1%.

…ls (1.4x faster)

The no-overflow check iterated `Option`s with `.iter().flatten().all()`,
reading the null bitmap for every slot and branching per value. Without
nulls every slot holds a value, so scan the values buffer directly, with
one unsigned comparison per value against the precision bounds shifted
by the minimum, and no branch inside each 64-value chunk. Arrays with
nulls keep the existing scan.

Add dense-null and ANSI-with-nulls shapes to the benchmark, and a test
for an overflow past the first chunk and a sliced array.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead. Reading the values buffer instead of going through the null bitmap per slot is the right fix for this scan.

Comment on lines +98 to +100
c.bench_function("check_overflow: ansi no overflow, with nulls", |b| {
b.iter(|| black_box(ansi.evaluate(black_box(&no_overflow_nulls)).unwrap()))
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a shape for the scan's worst case? Both sparse overflow and dense overflow put an overflow at row 0, because build tests i % overflow_every == 0, so the scan stops in the first chunk. The costliest input for this design is one overflow near the end of the batch: the scan reads every chunk, and then null_if_overflow_precision reads the batch again. apache/arrow-rs#12058 and apache/arrow-rs#12062 use the same one-pass-then-fallback design without stopping early, so every overflow case they benchmark pays for the full pass plus the fallback, and they report that cost. Could you add this shape's numbers to the description? It needs use arrow::compute::concat_batches; at the top.

Suggested change
c.bench_function("check_overflow: ansi no overflow, with nulls", |b| {
b.iter(|| black_box(ansi.evaluate(black_box(&no_overflow_nulls)).unwrap()))
});
c.bench_function("check_overflow: ansi no overflow, with nulls", |b| {
b.iter(|| black_box(ansi.evaluate(black_box(&no_overflow_nulls)).unwrap()))
});
// One overflow in the last row, so the scan reads the whole batch before the slow path runs.
// `sparse overflow` and `dense overflow` overflow at row 0 and stop in the first chunk.
let head = build(size - 1, 0, 0);
let last_row_overflow = concat_batches(&head.schema(), [&head, &build(1, 0, 1)]).unwrap();
c.bench_function("check_overflow: last row overflow", |b| {
b.iter(|| black_box(legacy.evaluate(black_box(&last_row_overflow)).unwrap()))
});

Comment on lines +251 to +280
/// Values per chunk in the no-null scan of [`fits_precision`].
const FITS_PRECISION_CHUNK: usize = 64;

/// Whether every non-null value of `array` fits `precision`, the check that
/// `Decimal128Type::is_valid_decimal_precision` makes for one value.
#[inline]
fn fits_precision(array: &Decimal128Array, precision: u8) -> bool {
let bounds = (
MIN_DECIMAL128_FOR_EACH_PRECISION.get(precision as usize),
MAX_DECIMAL128_FOR_EACH_PRECISION.get(precision as usize),
);
match bounds {
(Some(&min), Some(&max)) if array.null_count() == 0 => {
// Without nulls every slot holds a value, so scan the values buffer directly instead
// of reading the null bitmap per slot. Shifting by `min` turns the two bounds checks
// into one unsigned comparison. The check is branchless within a chunk, which lets
// the compiler unroll it, and the scan still stops at the first chunk that holds an
// overflow.
let range = max.wrapping_sub(min) as u128;
array.values().chunks(FITS_PRECISION_CHUNK).all(|chunk| {
chunk.iter().fold(true, |fits, &v| {
fits & (v.wrapping_sub(min) as u128 <= range)
})
})
}
_ => array
.iter()
.flatten()
.all(|v| Decimal128Type::is_valid_decimal_precision(v, precision)),
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the values scan cover arrays with nulls too? A null slot can hold any value, so when some slot doesn't fit, we can fall back to the existing scan that skips nulls, and only then. apache/arrow-rs#12062 does the same for the checked decimal kernels in unless_valid_overflow: it runs over every slot and reads the validity bits only when something was flagged.

Builders write 0 into null slots, and 0 fits every precision. So the with nulls and dense nulls shapes, which this PR leaves at 4.76 us and 3.88 us, would take the same scan as no overflow. A batch whose null slots hold values that don't fit pays for the values scan up to that chunk, then for today's scan. The last test in my comment on the tests covers that case. Could you rerun the null shapes with this change and add the numbers to the description?

Suggested change
/// Values per chunk in the no-null scan of [`fits_precision`].
const FITS_PRECISION_CHUNK: usize = 64;
/// Whether every non-null value of `array` fits `precision`, the check that
/// `Decimal128Type::is_valid_decimal_precision` makes for one value.
#[inline]
fn fits_precision(array: &Decimal128Array, precision: u8) -> bool {
let bounds = (
MIN_DECIMAL128_FOR_EACH_PRECISION.get(precision as usize),
MAX_DECIMAL128_FOR_EACH_PRECISION.get(precision as usize),
);
match bounds {
(Some(&min), Some(&max)) if array.null_count() == 0 => {
// Without nulls every slot holds a value, so scan the values buffer directly instead
// of reading the null bitmap per slot. Shifting by `min` turns the two bounds checks
// into one unsigned comparison. The check is branchless within a chunk, which lets
// the compiler unroll it, and the scan still stops at the first chunk that holds an
// overflow.
let range = max.wrapping_sub(min) as u128;
array.values().chunks(FITS_PRECISION_CHUNK).all(|chunk| {
chunk.iter().fold(true, |fits, &v| {
fits & (v.wrapping_sub(min) as u128 <= range)
})
})
}
_ => array
.iter()
.flatten()
.all(|v| Decimal128Type::is_valid_decimal_precision(v, precision)),
}
/// Values per chunk in the scan of [`fits_precision`].
const FITS_PRECISION_CHUNK: usize = 64;
/// Whether every non-null value of `array` fits `precision`, the check that
/// `Decimal128Type::is_valid_decimal_precision` makes for one value.
#[inline]
fn fits_precision(array: &Decimal128Array, precision: u8) -> bool {
let bounds = (
MIN_DECIMAL128_FOR_EACH_PRECISION.get(precision as usize),
MAX_DECIMAL128_FOR_EACH_PRECISION.get(precision as usize),
);
let valid_fit = || {
array
.iter()
.flatten()
.all(|v| Decimal128Type::is_valid_decimal_precision(v, precision))
};
match bounds {
(Some(&min), Some(&max)) => {
// Scan every slot of the values buffer, null or not, instead of reading the null
// bitmap per slot. Shifting by `min` turns the two bounds checks into one unsigned
// comparison, and the scan stops at the first chunk that holds a value that doesn't
// fit.
let range = max.wrapping_sub(min) as u128;
let all_fit = array.values().chunks(FITS_PRECISION_CHUNK).all(|chunk| {
chunk.iter().fold(true, |fits, &v| {
fits & (v.wrapping_sub(min) as u128 <= range)
})
});
// A null slot can hold any value, so a value that doesn't fit only counts if the
// scan that skips nulls finds one too.
all_fit || (array.null_count() > 0 && valid_fit())
}
_ => valid_fit(),
}

Comment on lines +256 to +257
#[inline]
fn fits_precision(array: &Decimal128Array, precision: u8) -> bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this scan also go into arrow-rs, in PrimitiveArray::validate_decimal_precision? It makes the same check with the same per-slot is_valid lookup and early return that this PR removes here. The vectorized arithmetic in apache/arrow-rs#12058 and apache/arrow-rs#12062 detects only i128 overflow and doesn't check precision, so it won't replace this check when it lands.

#5104 lists CheckOverflow as hand-rolled on purpose, because ANSI errors need the offending value. This PR splits the scan, which returns only a bool, from the error path that finds the value, so that reason no longer covers the scan.

I'd land the scan here first and settle its design in this PR, including the null handling and loop form from my other comments. Then the arrow-rs change can port the same code instead of reopening those choices. Could you file the arrow-rs issue now and link it from a comment on fits_precision? Once Comet's arrow version includes the upstream change, fits_precision can become validate_decimal_precision(precision).is_ok(), and DataFusion and other arrow-rs callers get the faster check too.

Comment on lines +264 to +274
// Without nulls every slot holds a value, so scan the values buffer directly instead
// of reading the null bitmap per slot. Shifting by `min` turns the two bounds checks
// into one unsigned comparison. The check is branchless within a chunk, which lets
// the compiler unroll it, and the scan still stops at the first chunk that holds an
// overflow.
let range = max.wrapping_sub(min) as u128;
array.values().chunks(FITS_PRECISION_CHUNK).all(|chunk| {
chunk.iter().fold(true, |fits, &v| {
fits & (v.wrapping_sub(min) as u128 <= range)
})
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says the branchless chunk lets the compiler unroll the loop. On aarch64 with Comet's release profile (thin LTO, codegen-units = 1), which cargo bench inherits, LLVM vectorizes the & reduction instead. In the linked check_overflow bench binary at this commit, the main loop takes 32 values per iteration in 396 instructions, and 88 of those are stack loads and stores. Each compare stays scalar, and only its flag moves into a NEON register:

ldr    x8, [x0, #0x80]    ; low half of a value
subs   x8, x8, x9         ; minus the low half of min
str    x8, [sp, #0x108]   ; spilled
...
cmp    x10, x5            ; range against v - min, 128 bits
sbcs   xzr, x11, x26
cset   w8, hs             ; fits
sbfx   x8, x8, #0, #1
dup.2d v4, x8             ; flag into a vector lane for the `&`

That's about 12 instructions per value plus the spills, against 9 in the same binary's one-value remainder loop. On x86-64 the fold unrolls two values per iteration at 11 instructions per value.

Could you benchmark the fold against a plain array.values().iter().all(|&v| v.wrapping_sub(min) as u128 <= range) on both architectures? That compiles to a scalar loop with one branch that isn't taken until a value doesn't fit: 10 instructions per value on aarch64 and 12 on x86-64, with no spills. In the no-overflow case that branch is always predicted correctly, so going branchless doesn't save anything there. If it's as fast, it drops FITS_PRECISION_CHUNK and the fold. Either way, the comment should describe what the compiler does.

Comment on lines +560 to 561
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add tests that move the value that doesn't fit through every row, and that cover the edges of the shift? The compiled scan handles each part of the batch with different code: on aarch64 a 32-value vector loop, an 8-value loop, and a one-value loop, and on x86-64 a two-value unrolled loop and a remainder. The new test checks one position, row 130. apache/arrow-rs#12058 sweeps the overflowing row through every lane for the same reason.

The array tests also all use precision 3, so nothing checks precision 38, where v - min wraps for i128::MAX and i128::MIN. The last test covers a null slot that holds a value that doesn't fit, which the suggestion on fits_precision makes reachable.

Suggested change
}
}
}
#[test]
fn test_array_no_nulls_overflow_at_every_position() {
// The compiled scan handles the main loop, the chunk boundaries, and the remainder
// differently, so move one value that doesn't fit through every row. 203 rows leave a
// last chunk of 11 values.
for overflow in [1000, -1000] {
for i in 0..203 {
let mut values = vec![Some(5); 203];
values[i] = Some(overflow);
let batch = array_batch(values, 38, 0);
let out = eval_array(&array_check_overflow(3, 0, false), &batch);
assert_eq!(out.null_count(), 1, "{overflow} at {i}");
assert!(out.is_null(i), "{overflow} at {i}");
let ansi = array_check_overflow(3, 0, true).evaluate(&batch);
assert!(ansi.is_err(), "{overflow} at {i}");
}
}
}
#[test]
fn test_array_no_nulls_precision_38_extremes() {
// The scan shifts by the precision's minimum and compares unsigned, so check the values
// where that shift wraps, at the widest precision.
let max = MAX_DECIMAL128_FOR_EACH_PRECISION[38];
let min = MIN_DECIMAL128_FOR_EACH_PRECISION[38];
let fits = array_batch(vec![Some(max), Some(min), Some(0)], 38, 0);
let out = eval_array(&array_check_overflow(38, 0, true), &fits);
assert_eq!(out.null_count(), 0);
for v in [max + 1, min - 1, i128::MAX, i128::MIN] {
let batch = array_batch(vec![Some(0), Some(v)], 38, 0);
let out = eval_array(&array_check_overflow(38, 0, false), &batch);
assert!(out.is_null(1), "{v}");
let ansi = array_check_overflow(38, 0, true).evaluate(&batch);
assert!(ansi.is_err(), "{v}");
}
}
#[test]
fn test_array_null_slot_out_of_range_is_ignored() {
// A null slot can hold any value. One that doesn't fit the precision must not null out
// valid values or raise an ANSI error.
use arrow::buffer::NullBuffer;
let nulls = NullBuffer::from(vec![true, false, true]);
let arr = Decimal128Array::new(vec![5, 1000, 7].into(), Some(nulls))
.with_precision_and_scale(38, 0)
.unwrap();
let schema = Schema::new(vec![Field::new("d", arr.data_type().clone(), true)]);
let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(arr)]).unwrap();
for fail_on_error in [false, true] {
let out = eval_array(&array_check_overflow(3, 0, fail_on_error), &batch);
assert_eq!(out.iter().collect::<Vec<_>>(), vec![Some(5), None, Some(7)]);
}
}
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants