Repository navigation
perf(coalesce): copy sparse byte-array indices directly - #12223
Rich-T-kid wants to merge 1 commit into
Conversation
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (618a419) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (618a419) to 4e04fea (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-lxiegy2m-b02d82b1-lsf8 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
618a419 to
7701041
Compare
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (7701041) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (7701041) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-hysmgqbk-8c5f483c-f2kp Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
7701041 to
6adfdee
Compare
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (6adfdee) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (6adfdee) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-12rv6o5a-dad8edb0-l6rk Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
run benchmark coalesce_kernels |
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#12223 (comment)).
File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (b283975) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: arm64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (b283975) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (b283975) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-e10s86d6-7faabcec-pbjl Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (b283975) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: arm64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-ez8gwx9u-593cc0f5-jvvf Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
b283975 to
510230c
Compare
|
run benchmark coalesce_kernels |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (510230c) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: '^take: (single_utf8|single_binary|mixed_utf8), 8192, nulls: (0|0\.1), selectivity: (0\.001|0\.01|0\.1|0\.8)$'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (510230c) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: '^take: (single_utf8|single_binary|mixed_utf8), 8192, nulls: (0|0\.1), selectivity: (0\.001|0\.01|0\.1|0\.8)$'
resources:
arch: arm64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (510230c) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: '^take: (single_utf8|single_binary|mixed_utf8), 8192, nulls: (0|0\.1), selectivity: (0\.001|0\.01|0\.1|0\.8)$'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-2qaq7be7-fd2ba571-v2rh Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (510230c) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: '^take: (single_utf8|single_binary|mixed_utf8), 8192, nulls: (0|0\.1), selectivity: (0\.001|0\.01|0\.1|0\.8)$'
resources:
arch: arm64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1ftdidtx-5aca7468-7zn4 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
510230c to
3245ba5
Compare
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: '^take: (mixed_utf8|single_utf8), 8192, nulls: 0\.1, selectivity: (0\.1|0\.8)$'
resources:
arch: arm64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-x4c26gr3-90b5c61c-7vhm Instance: c4a-standard-32-lssd (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: '^take: (single_utf8|single_binary|mixed_utf8), 8192, nulls: (0|0\.1), selectivity: (0\.001|0\.01|0\.1|0\.8)$'
resources:
arch: arm64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: '^take: (single_utf8|single_binary|mixed_utf8), 8192, nulls: (0|0\.1), selectivity: (0\.001|0\.01|0\.1|0\.8)$'
resources:
arch: arm64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-x4c26gr3-90b5c61c-7vhm Instance: c4a-standard-32-lssd (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
1 similar comment
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: utf8
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: utf8
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: utf8
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-17veedx0-07303aad-v85s Instance: c4a-highcpu-48 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: utf8
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-17veedx0-07303aad-swjb Instance: c4a-highcpu-48 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
Hi @Rich-T-kid, your benchmark configuration could not be parsed (#12223 (comment)). Error: Usage: Any benchmark name is accepted: Per-side configuration ( env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2GPod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal): resources:
cpu: "16"
memory: "128Gi"
arch: arm64 # arm64 or amd64Independent benchmark shards (optional; 1–8, default 1): run benchmark arrow_writer
shards: 4Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion. File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: arm64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1e3im07h-baf3dd5e-ptd2 Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: arm64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-q1db7b2y-b3409901-thj6 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: arm64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: amd64
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
run benchmark coalesce_kernels |
|
🤖 Benchmark starting (GKE) | trigger Target: coalesce_kernels Sharding factor: 1 (1 workers). Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
id like to attribute the minor regression to noise but lets see 😅 |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: amd64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-1e3im07h-ade29973-sqkt Instance: c4-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-q1db7b2y-cf67b4a9-pqnh Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: coalesce_kernels Comparing rich-T-kid/coalesce-string-indices (8d0d96a) to 4ea4d22 (merge-base) diff Run configurationrun benchmark coalesce_kernels
env:
BENCH_FILTER: 'take: (single_utf8|mixed_utf8|single_binary),'
resources:
arch: arm64
shards: 1
DetailsPer-runner informationcoalesce_kernels — shard 1/1Node: gk3-benchmark-cluster-nap-q1db7b2y-cf67b4a9-hfwx Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench coalesce_kernelsCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
turns out it was noise |
Which issue does this PR close?
Rationale for this change
Indexed byte-array copies currently materialize an intermediate array with
takeand then copy the selected values again into the coalescer. This is especially expensive for sparse selections.What changes are included in this PR?
Sparse
Utf8,LargeUtf8,Binary, andLargeBinaryselections are copied directly into the coalescer. Denser selections use the existingtakeimplementation and adopt its buffers when every schema column is primitive or a variable-width byte array. Other schemas keep the existing coalescing path. Thetakekernel itself is unchanged.I used Codex to help implement the change and run the benchmark matrix. I reviewed the final code and benchmark results.
Are these changes tested?
Tests cover direct copying and fallback behavior, including nulls, slices, repeated and out-of-order indices, and target-batch splitting.
The matrix spans single UTF-8, single binary, and mixed primitive/UTF-8 schemas; nullable and non-null data; and 0.1%, 1%, 10%, and 80% selectivity.
Are there any significant user-facing changes?
No.