[DISCUSS] Proposal to drop support for Spark 2

[yew1eb]

Hi all,

I'd like to propose dropping Spark 2.x support, based on the following key reasons:

• Spark 2.x has been EOL for over 4 years, with near-zero adoption in our user base.
• Maintaining it introduces heavy technical debt (legacy code, CI overhead) and blocks us from adopting modern Spark features.
• Spark 2.x CI tests are frequently unstable and flaky, consuming significant maintenance effort.
• Aligns with ecosystem norms—projects like Iceberg and Hudi have already dropped Spark 2.x support.

Please share your thoughts!

[yew1eb]

cc @SteNicholas