ETL CompareETL Compare

Glossary

What is Velox?

In brief

Velox is an open-source C++ execution engine created at Meta, described by its project as “a high-performance, open-source execution engine designed for flexibility and reuse”. It is not a Spark accelerator on its own: Apache Gluten is the layer that plugs Velox into Spark, so that Spark plans run on Velox's native, vectorized operators.

How does Velox relate to Spark?

Spark plans the query; Gluten translates the physical plan for Velox; Velox executes supported operators natively and Gluten falls back to vanilla Spark for the rest. Gluten can also use ClickHouse as a backend.

How is Velox different from DataFusion?

Both are native execution engines used under Spark. Velox is written in C++ and reaches Spark through Gluten. DataFusion is written in Rust on Apache Arrow and reaches Spark through Comet, Auron and Flarion.

Related

Sources