Apache Gluten with Velox review: native execution for open-source Spark

Gluten describes itself as a middle layer that offloads the execution of JVM-based SQL engines to native engines. For Spark, the main native backend is Velox, a C++ execution engine created at Meta that its project describes as 'a high-performance, open-source execution engine designed for flexibility and reuse'. ClickHouse is the other supported backend. Gluten was started by Intel and Kyligence in 2022 and became an Apache top-level project in March 2026; contributors listed on the repository include BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, IBM and Google.

Sources: gluten.apache.org, github.com/apache/incubator-gluten, velox-lib.io · Fetched 27 Sep 2026

There are no application code changes. The Velox getting-started page lists the configuration: set spark.plugins to org.apache.gluten.GlutenPlugin, add the Gluten jar to the driver and executor classpath, set spark.memory.offHeap.enabled to true with an off-heap size (the example uses 20g), and set spark.shuffle.manager to the ColumnarShuffleManager. Unsupported operators fall back to vanilla Spark automatically.

spark.plugins=org.apache.gluten.GlutenPlugin
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=20g
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager

Sources: gluten.apache.org/docs/getting-started/velox-backend · Fetched 27 Sep 2026

Sources: apache/incubator-gluten on GitHub · Fetched 27 Sep 2026

The benchmarks are named and repeatable, which is more than most vendors offer, but they are project-published and the most detailed figures are from 2023.

• No license fee and the largest contributor base of the open-source engines here (maturity 4.4).
• Broad Spark version support on x86_64 and aarch64 Linux, on the CPU instances you already run (portability 4.3, highest in set).
• A columnar shuffle manager, so the gain is not limited to the compute stage.

• Memory is yours to size. The off-heap pool has to be set per workload.
• Spill in the Velox backend is documented as experimental, which matters most for exactly the spill-heavy jobs that prompt a search for an accelerator.
• No vendor support contract is described on the project pages; the tuning and operating score (2.6) is tied lowest in the set.
• Managed platforms: the project does not publish EMR, Dataproc or Databricks guides. It installs where you control Spark config and classpath.

Platform teams that self-manage Spark, have engineers who can tune memory and shuffle settings, and want a free engine with a broad contributor base.