Apache DataFusion under Spark: how Comet, Auron and Flarion share one engine

DataFusion describes itself as an "extensible query engine written in Rust that uses Apache Arrow as its in-memory format". Apache Arrow is a columnar memory layout, so DataFusion is a columnar, vectorized engine: its operators work on batches of column values rather than one row at a time. It is a library for building data systems, not a Spark replacement, which is why Spark accelerators can embed it.

The shared engine does not make the products the same. Comet documents its own memory pool, tuning settings and shuffle manager; Auron documents its own shuffle format and memory management; Flarion describes caching, performance monitoring and anomaly detection. Our Comet review, Auron review and Flarion review score each on its own documentation, and DataFusion Comet vs Apache Auron puts two of them side by side.

Apache DataFusion 55.0.0 was released on 25 August 2026, from 877 commits by 175 contributors. It adds range partitioning, MERGE INTO planner support, runtime row-group pruning and pluggable spill backends. Row-group pruning lets a scan skip blocks of Parquet data that cannot match a query, and spill backends decide how the engine writes data to disk under memory pressure, which matters for spill-heavy ETL.

An engine release does not reach Spark users until the accelerator built on it adopts it. Comet 1.0.0 (7 August 2026) came out before DataFusion 55.0.0, so check the release notes of Comet, Auron or Flarion for the DataFusion version they ship before counting on a new engine feature.

Both are native, vectorized execution engines used under Spark. Velox is written in C++, was created at Meta and reaches Spark through Apache Gluten, which can also use ClickHouse as a backend. DataFusion is written in Rust on Apache Arrow and reaches Spark through Comet, Auron and Flarion. Both communities were active in the period we track: on 25 August 2026, the day DataFusion 55.0.0 shipped, the Velox blog described collecting Delta Lake statistics natively for Gluten.

• Engine features are necessary, not sufficient. A DataFusion feature helps only once the accelerator you run ships it and maps your operators onto it.
• Compare the Spark layer, not only the engine. Plan translation, fallback, shuffle and memory handling are where the three products differ, and they decide how much of your job runs natively.
• Watch both release calendars. For Comet and Auron, the Apache DataFusion and Apache Spark release notes both affect what you can run.
• Test on your own job. Engine-level benchmarks say little about a spill-heavy or shuffle-heavy ETL pipeline; our proof of concept guide shows how to test.