Repository navigation
Specialize BatchCoalescer for RunArray - #12218
Rich-T-kid wants to merge 23 commits into
Conversation
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (9c6602c) to cd91f89 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (9c6602c) to cd91f89 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-f0lk0bbd-7e69c37a-5rsc Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
1 similar comment
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (9c6602c) to cd91f89 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (9c6602c) to cd91f89 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (9c6602c) to cd91f89 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-f0lk0bbd-7e69c37a-5rsc Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (9c6602c) to cd91f89 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-f0lk0bbd-28ec8992-mpmz Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
9c6602c to
0207495
Compare
|
run benchmark coalesce_kernels |
1 similar comment
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1ey2y0e2-efdd6ef4-4xr5 Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1ey2y0e2-c6aa1932-89l4 Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1ey2y0e2-efdd6ef4-5vcj Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1ey2y0e2-efdd6ef4-5vcj Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (0207495) to 4c03e90 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1ey2y0e2-efdd6ef4-5vcj Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
0207495 to
a3e3cb5
Compare
|
run benchmark coalesce_kernels env:
BENCH_FILTER: "filter: single_ree"
resources:
arch: amd64 |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (a3e3cb5) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'filter: single_ree'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
Hi @Rich-T-kid, thanks for the request (#12218 (comment)). Usage: Any benchmark name is accepted: Per-side configuration ( env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2GPod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal): resources:
cpu: "16"
memory: "128Gi"
arch: arm64 # arm64 or amd64Independent benchmark shards (optional; 1–8, default 1): run benchmark arrow_writer
shards: 4Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion. File an issue against this benchmark runner |
|
Hi @Rich-T-kid, thanks for the request (#12218 (comment)). Usage: Any benchmark name is accepted: Per-side configuration ( env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2GPod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal): resources:
cpu: "16"
memory: "128Gi"
arch: arm64 # arm64 or amd64Independent benchmark shards (optional; 1–8, default 1): run benchmark arrow_writer
shards: 4Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion. File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (b65ff90) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: arm64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (b65ff90) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (b65ff90) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: arm64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-60ma4f95-6c11b3b1-72z8 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (b65ff90) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1n18gwqc-38f4eb21-k2tl Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (b65ff90) to b3c1346 diff Run configurationrun benchmark coalesce_kernels
baseline:
ref: b3c134666cb6afc59eba9918e761faf2e9945840
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (b65ff90) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (b65ff90) to b3c1346 diff Run configurationrun benchmark coalesce_kernels
baseline:
ref: b3c134666cb6afc59eba9918e761faf2e9945840
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1odthwem-92f17ebb-szgw Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (b65ff90) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-19rx0sh2-c32e1e2a-n95p Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
run benchmark coalesce_kernels |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (ab4370a) to b3c1346 diff Run configurationrun benchmark coalesce_kernels
baseline:
ref: b3c134666cb6afc59eba9918e761faf2e9945840
env:
BENCH_FILTER: single_ree
resources:
arch: arm64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (ab4370a) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (ab4370a) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: arm64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (ab4370a) to b3c1346 diff Run configurationrun benchmark coalesce_kernels
baseline:
ref: b3c134666cb6afc59eba9918e761faf2e9945840
env:
BENCH_FILTER: single_ree
resources:
arch: arm64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-aznwbglv-faa633b0-sxjk Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (ab4370a) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-19rx0sh2-c32e1e2a-n95p Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (ab4370a) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: single_ree
resources:
arch: arm64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-aznwbglv-d7f3a719-pl7l Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing codex/issue-11720 (ab4370a) to b3c1346 diff Run configurationrun benchmark coalesce_kernels
baseline:
ref: b3c134666cb6afc59eba9918e761faf2e9945840
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing codex/issue-11720 (ab4370a) to b3c1346 diff Run configurationrun benchmark coalesce_kernels
baseline:
ref: b3c134666cb6afc59eba9918e761faf2e9945840
env:
BENCH_FILTER: single_ree
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1uht8jxf-63c9b4e9-pvn8 Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Which issue does this PR close?
Rationale for this change
BatchCoalescercurrently uses the generic in-progress array implementation forRunArray. A run-aware accumulator can preserve the encoding while rows are copied and batches are combined.What changes are included in this PR?
Adds
InProgressRunArrayfor run-end encoded columns withInt16,Int32, orInt64run ends.The accumulator handles a logical slice in two steps: it appends clamped, rebased run ends, then retains the corresponding physical value range. Adjacent ranges from the same source are combined. Only the first incoming run can continue the buffered tail, so the boundary check is an O(1) source-and-index comparison.
For example, consider source run ends
[3, 6, 10]with values[A, B, C], and copy the logical slice[2, 7):If the output already contains 4 rows, the run ends become
[5, 8, 9]. IfAis also the buffered tail run, the first existing run end is extended to5and only[B, C]is retained. Otherwise[A, B, C]is retained.Sparse filters use the same accumulator directly. Because filter selections are ordered, the implementation walks the source run ends once instead of materializing a filtered
RunArrayand searching for every selected range. Denser filters retain the existing materialized path, with a lower cutoff for nullable values.Equal values in different physical runs remain distinct, matching concat and interleave semantics.
finishuses Arrow's existing type-specific concat kernels for the recorded value ranges.Are these changes tested?
The shared multi-column coalescer tests include a nullable
RunArrayand exercise sparse and dense filters, empty batches, non-null variants, and the single-column path. The fullarrow-selecttest suite and strict Clippy checks pass.coalesce_kernels/single_reeAre there any significant user-facing changes?
No.