Add spark-gluten-clickhouse entry (Spark + Gluten with the CH backend)

[alexey-milovidov]

Summary

• Adds a spark-gluten-clickhouse/ entry that runs Apache Spark with Apache Gluten configured to use the ClickHouse backend (spark.gluten.sql.columnar.backend.lib=ch). Gluten loads libch.so (a fork of ClickHouse v23.1) into the Spark executor JVM and runs the columnar physical plan natively through it.
• Complements spark-gluten/ (Velox backend) and the proposed spark-velox/ (Add spark-velox entry (Spark + Velox via Apache Gluten) #858) — this entry exercises a meaningfully different execution path: Catalyst → Substrait → ClickHouse engine, rather than Catalyst → Substrait → Velox.

Build

No pre-built bundle is published for the CH backend (Apache Gluten v1.4.0 ships only the Velox bundle, and Maven Central has nothing). benchmark.sh therefore builds two things from source:

1. libch.so — built from Kyligence/ClickHouse at the branch pinned in gluten/cpp-ch/clickhouse.version (currently rebase_ch/20250326). Uses Clang 18 / cmake / ninja.
2. The Gluten Spark plugin — built via Maven with -Pbackends-clickhouse,spark-3.5,scala-2.12 under JDK 8.

Limitations

• The libch.so compile is essentially a ClickHouse build and is RAM-hungry; Gluten's docs recommend ≥64 GB. On c6a.4xlarge (32 GB) it may OOM — c6a.8xlarge or larger is recommended for a clean run, hence the default machine label in benchmark.sh.
• ARM is untested. Both ClickHouse and the Gluten plugin should compile on aarch64 in principle, but the Gluten CI does not publish CH-backend artifacts for ARM.

Notes

• Queries use ClickHouse-style regex backreferences (\1) rather than Spark's $1, because regex evaluation runs inside libch.so. This was anticipated in the existing spark-gluten/README.md and Gluten issue #7545.
• Memory split between Spark heap and the Gluten off-heap pool is 50/50, identical to the Velox entry — the CH backend also runs off-heap via JNI.

Test plan

• Run on an x86_64 c6a.8xlarge.
• Verify benchmark.sh clones gluten + Kyligence/ClickHouse, builds libch.so and the Spark plugin, runs all 43 queries, and writes results/<machine>.json.

🤖 Generated with Claude Code