[DNM] Adding Upstream Spark

[zhouyuan]

What changes are proposed in this pull request?

This patch adds a spark-main profile which builds Gluten against Spark nightly builds
TPCH/DS tests added

With this patch Gluten should be able to catch-up with Spark release, e.g., if Spark cut the branch for release Gluten could also move the spark-main shim layer to the target release shim easily

How was this patch tested?

pass GHA

Was this patch authored or co-authored using generative AI tooling?

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[philo-he]

@zhouyuan, thanks for proposing this patch.
Perhaps we should move the Spark main tests to a nightly CI job. Since the Spark snapshot is under active development, keeping these tests in our regular CI may frequently cause stability issues and build failures that possibly blocking PR merging. Thanks.

[baibaichen]

@zhouyuan would you please follow #11352?

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[zhouyuan]

@philo-he @baibaichen Thanks for the inputs. I have monitored this for almost 2 weeks and there are two API changes from Spark in this timeframe which requires modifying the whole shim layers, moving the tests to nightly is a better option.

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86