Extract common-layer changes and try to add Bolt backend in Gluten

[taiyang-li]

Background

Bolt project: https://github.com/bytedance/bolt

Before proposing Bolt as a supported backend in Gluten, we would like to first extract the related changes in shared modules (for example gluten-substrait and gluten-core) into a set of smaller and independent PRs.

These PRs focus on extending common abstractions / interfaces that are potentially useful to backends in general, instead of binding the changes directly to a specific Bolt backend implementation. The goal is to make sure these common-layer changes can be reviewed on their own and do not change the runtime behavior of the existing Velox / CH backends.

Why split the work this way?

• It allows us to discuss and land backend-agnostic common-layer changes without introducing the full Bolt backend at the same time.
• It makes it easier for the community and Bolt contributors to review whether these changes are reasonable and whether they keep existing Velox / CH backends unaffected.

Related preparatory PRs

Stage 0

• PR [GLUTEN-12462][CORE] Expose shuffle reader metrics iterator delegate #12432
  • Sub-issue [CORE] Expose shuffle reader metrics iterator delegate #12462
• PR [GLUTEN-12463][CORE] Add backend hooks for native config forwarding #12437
  • Sub-issue [CORE] Add backend hooks for native config forwarding #12463
• PR [GLUTEN-12464][CORE] Add TaskContext JNI callback for reading Spark task attempt id from native #12435
  • Sub-issue [CORE] Add TaskContext JNI callback for reading Spark task attempt id from native #12464
• PR [GLUTEN-12465][CORE] Support optional stage InputStats plumbing in scan/input-iterator transformers #12433
  • Sub-issue [CORE] Support optional stage InputStats plumbing in scan/input-iterator transformers #12465
• PR [GLUTEN-12466][CORE] Expose ColumnarBatches.isLightBatch/ensureOffloaded as public #12431
  • Sub-issue [CORE] Expose ColumnarBatches.isLightBatch/ensureOffloaded as public #12466

Stage 1

• PR [WIP] add bolt backend in gluten #12454
  • Sub-issue Try to add bolt backend in gluten #12467

Proposed plan

1. Review and discuss these smaller preparatory PRs in Gluten first.
2. After the common-layer preparation is in place, submit a separate PR to introduce the Bolt backend on top of the latest community code.
3. Start a new dev discussion / vote in the community to decide whether Bolt should be accepted as one of the supported Gluten backends.

[jackylee-ch]

@taiyang-li Consider breaking down issues into smaller ones, with each PR linked to a specific issue; ideally, commit messages should also reference the issue. This makes it easier to understand the context of the changes when tracking PRs later.