Comet
Apache DataFusion Comet 1.1.0 released
The Apache DataFusion PMC announced version 1.1.0 of the Comet subproject.
Source: Apache DataFusion blog
Rankings · Figures checked September 2026
Best Spark ETL accelerators of 2026, compared
Nine engines and plugins ranked on one buyer question: which one speeds up the Spark ETL you already run, on the platform you already use, without code changes or a team set aside for tuning. The ranking weights adoption effort, stage coverage and tuning burden most.
| Accelerator | Read | Transform | Shuffle | Spill | Write |
|---|---|---|---|---|---|
| Gluten + Velox | Partial | Documented | Documented | Partial | Not stated |
| DualBird | Partial | Documented | Documented | Documented | Partial |
| Comet | Documented | Documented | Documented | Partial | Not stated |
| RAPIDS Accelerator | Documented | Documented | Documented | Not stated | Documented |
| Photon | Documented | Documented | Not stated | Not stated | Documented |
| Auron | Partial | Documented | Documented | Partial | Not stated |
| Flarion | Partial | Documented | Not stated | Not stated | Not stated |
| DataPelago | Documented | Partial | Not stated | Not stated | Not stated |
| Speedata | Partial | Partial | Not stated | Not stated | Not stated |
- Documented: the vendor's own documentation says it addresses this stage
- Partial: indirect or experimental, or covered only by an end-to-end claim
- Not stated: not found on the pages we reviewed
This shows what each vendor says, not what we measured. Sources are listed on each review.
In brief
On the lead question, which accelerator speeds up the Spark ETL you already run, on the platform you already use, without code changes or a dedicated tuning team, Apache Gluten ranks 1st (3.73) and DualBird 2nd (3.72), a gap of 0.01, which we treat as a near tie. DualBird scores highest of the nine on adoption effort and stage coverage, and is tied lowest on cost model transparency and maturity and second lowest on portability and published evidence, because it is young, AWS-only on F2 instances and does not publish prices.
ETL Compare staff · Figures checked September 2026
Which accelerator speeds up the Spark ETL you already run, without code changes or a tuning team?
The ETL fit score is an editorial assessment against seven weighted criteria. It is not a benchmark and we did not run the products. Weights and the reasons behind them: Why these weights.
| Rank | Accelerator | Approach | Runs on | ETL fit score (editorial assessment, 0-5) | Best for |
|---|---|---|---|---|---|
| 1 | Apache Gluten (with Velox) | Open-source plugin that offloads Spark SQL execution to a native C++ engine (Velox or ClickHouse) | Self-managed Spark 3.4 to 4.1 on Linux, on any platform where you control Spark config | 3.73 / 5 ETL fit score (editorial assessment, 0-5) | Open-source native engine for self-managed Spark |
| 2 | DualBird | Commercial Spark plugin paired with Amazon EC2 F2 instances, aimed at spill, skew and shuffle bottlenecks | Apache Spark, Amazon EMR and Amazon EKS, on AWS | 3.72 / 5 ETL fit score (editorial assessment, 0-5) | Best for spill- and shuffle-heavy ETL on EMR and EKS |
| 3 | Apache DataFusion Comet | Open-source Spark plugin that runs supported operators on the Apache DataFusion engine (Rust, Arrow) | Self-managed Spark 3.5, 4.0 and 4.1 (3.4 deprecated) on Linux | 3.67 / 5 ETL fit score (editorial assessment, 0-5) | Best for teams on recent Spark 4.x releases |
| 4 | RAPIDS Accelerator for Apache Spark (now NVIDIA cuDF for Apache Spark) | Open-source NVIDIA plugin that runs supported Spark SQL and DataFrame operations on GPUs | Amazon EMR, Databricks, Dataproc, GKE, Azure Synapse, Kubernetes, on-premises, OCI | 3.65 / 5 ETL fit score (editorial assessment, 0-5) | Best for teams that already run GPU capacity |
| 5 | Databricks Photon | Databricks-native vectorized C++ query engine inside the Databricks Runtime | Databricks only (AWS, Azure, Google Cloud) | 3.46 / 5 ETL fit score (editorial assessment, 0-5) | Best for teams already on Databricks |
| 6 | Apache Auron (formerly Blaze) | Incubating open-source engine that maps Spark physical plans onto DataFusion native execution, with its own shuffle format | Self-managed Spark on JDK 8, 11, 17 or 21 | 3.42 / 5 ETL fit score (editorial assessment, 0-5) | Open-source option with its own shuffle and memory layer |
| 7 | Flarion | Commercial DataFusion-based, Arrow-native execution engine for Spark, Hadoop and Ray | Databricks, AWS EMR, GCP Dataproc, Azure HDInsight, Spark on Kubernetes, on-premises | 3.15 / 5 ETL fit score (editorial assessment, 0-5) | Commercial plugin with a broad managed-platform list |
| 8 | DataPelago Accelerator for Spark | Commercial Spark accelerator (DPA-S) built on the DataPelago Nucleus engine, using CPU, GPU or FPGA hardware | Multiple Spark versions on Google Cloud and AWS; Amazon EMR, EKS and Databricks not named on the DPA-S page | 3.14 / 5 ETL fit score (editorial assessment, 0-5) | Compute-heavy Spark SQL and DataFrame jobs on Parquet or ORC in Google Cloud or AWS |
| 9 | Speedata | Custom analytics silicon: the C200 Analytics Processing Unit (APU) card plus Dash, a Spark plugin that hooks the Catalyst optimizer | Speedata APU servers in your data center (pre-configured 2U server or OEM servers such as HPE); Amazon EMR, EKS and Databricks not mentioned on the product page | 2.61 / 5 ETL fit score (editorial assessment, 0-5) | Teams that can add dedicated analytics hardware on premises |
Measures fit for speeding up existing Spark ETL without rewriting pipelines. It is not a benchmark and not a measure of peak speed.
Apache Gluten and DualBird are separated by 0.01, which is within the margin of our editorial scoring; treat them as a near tie. DualBird is tied lowest in the set on cost model transparency (2.0) and maturity (2.0); second lowest on portability (2.4, AWS and F2 only) and published evidence (2.8). Per-criterion facts for every accelerator, without scores: Facts by criterion.
| Accelerator | Adoption 20% | Stages 20% | Operating 20% | Portability 10% | Cost model 5% | Evidence 15% | Maturity 10% | Total |
|---|---|---|---|---|---|---|---|---|
| Gluten + Velox | 3.8 | 3.8 | 2.6lowest in set | 4.3highest in set | 5.0highest in set | 3.8highest in set | 4.4 | 3.73 |
| DualBird | 4.6highest in set | 4.8highest in set | 4.4 | 2.4 | 2.0lowest in set | 2.8 | 2.0lowest in set | 3.72 |
| Comet | 3.8 | 3.6 | 2.8 | 4.2 | 5.0highest in set | 3.8highest in set | 3.9 | 3.67 |
| RAPIDS Accelerator | 3.4 | 4.2 | 3.0 | 3.2 | 4.2 | 3.6 | 4.6highest in set | 3.65 |
| Photon | 2.6lowest in set | 3.8 | 4.6highest in set | 1.6lowest in set | 3.2 | 3.2 | 4.6highest in set | 3.46 |
| Auron | 3.4 | 3.8 | 2.6lowest in set | 3.8 | 5.0highest in set | 3.4 | 3.2 | 3.42 |
| Flarion | 4.2 | 2.8 | 3.8 | 4.2 | 2.0lowest in set | 1.8lowest in set | 2.0lowest in set | 3.15 |
| DataPelago | 3.8 | 3.0 | 3.0 | 3.2 | 3.0 | 3.0 | 2.6 | 3.14 |
| Speedata | 2.6lowest in set | 2.6lowest in set | 3.0 | 1.6lowest in set | 2.0lowest in set | 3.0 | 2.6 | 2.61 |
Criterion scores are shown to one decimal and totals to two; ranks use the unrounded totals. Weights and reasons: see How we score.
Which accelerator fits which situation?
| Situation | Consider first | Why | Watch for |
|---|---|---|---|
| You run Spark on EMR or EKS and your slow stages are shuffle and disk spill | DualBird | Its product page targets spills, skew and shuffle directly, and setup is an instance-type change plus a plugin (DualBird states) | AWS only; F2 instances; no public price; vendor-published evidence only |
| You self-manage Spark 3.4 to 4.1 and want a free native engine | Apache Gluten with Velox | Largest contributor base of the open-source options, columnar shuffle, broad Spark version support | Off-heap memory sizing and experimental spill are your team's job |
| You are on Spark 3.5 or 4.x and want a lighter open-source plugin | Apache DataFusion Comet | Prebuilt Linux jars on Maven Central, native shuffle, a published TPC-DS benchmark | Tuning guide warns memory accounting is not exact |
| You already have NVIDIA GPU capacity or budget | RAPIDS Accelerator | The broadest platform list and a Qualification Tool that reads your own event logs | Every node must be a GPU instance |
| Your jobs already run on Databricks | Databricks Photon | One setting, no plugin to maintain, default on serverless | Only runs on Databricks; DBU consumption changes |
| You want an open-source engine with its own shuffle and memory layer and can accept incubating status | Apache Auron | Compacted shuffle format and multi-level memory management | Still incubating; supported Spark versions not listed on its homepage |
| You need one commercial plugin across Databricks, EMR, Dataproc and HDInsight | Flarion | The broadest managed-platform list among the commercial options | Few published details on method or pricing |
Not sure which stage is slow? Profile a slow Spark ETL job before buying anything.
Where do accelerators sit relative to EMR, Databricks and other platforms?
An accelerator is not a replacement for the platform your Spark jobs run on. Amazon EMR, Databricks, Google Cloud's managed Spark service (Dataproc), AWS Glue and self-managed Spark on Kubernetes are the platform layer: they provision clusters, schedule jobs and ship a Spark runtime. Accelerators sit underneath the Spark API on that platform and replace parts of how the physical plan executes. That is why none of the platforms appear in the ranking.
The practical question is which accelerator is documented for the platform you already use. Most of the products here keep you on your platform. Photon is the exception: it only exists inside Databricks, so for a team on EMR it implies a platform move, which is why it scores lowest on adoption effort for this site's use case.
| Accelerator | EMR | Databricks | Google Cloud (Dataproc) | AWS Glue | Self-managed Spark / Kubernetes |
|---|---|---|---|---|---|
| Gluten + Velox | Not documented(self-install) | Not documented | Not documented(self-install) | Not documented | Documented |
| DualBird | Documented | Not documented | Not documented(AWS only) | Not documented | Documented(Apache Spark, Amazon EKS) |
| Comet | Not documented(self-install) | Not documented | Not documented(self-install) | Not documented | Documented |
| RAPIDS Accelerator | Documented | Documented | Documented | Not documented | Documented |
| Photon | Not available | Only platform | Not available | Not available | Not available |
| Auron | Not documented(self-install) | Not documented | Not documented(self-install) | Not documented | Documented |
| Flarion | Documented | Documented | Documented | Not documented | Documented(on-premises) |
| DataPelago | Not documented(AWS stated; EMR not named) | Not documented | Not documented(Google Cloud stated; service not named) | Not documented | Not documented |
| Speedata | Not documented | Not documented | Not documented | Not documented | Documented(on Speedata APU servers, on premises) |
Documented means the vendor or project lists the platform on the pages we reviewed. Self-install means you can usually add an open-source plugin to a platform that lets you set Spark configuration and classpath, but the project does not publish a guide for that platform. None of the vendor pages we reviewed list AWS Glue.
What do the vendors claim, and how should you read the claims?
Every vendor in this guide publishes a speed or cost figure. The figures come from different workloads, baselines and cluster sizes, so they cannot be compared with each other. We quote them so you know what each vendor promises, then score how much evidence stands behind the figure.
| Accelerator | What the vendor or project states | Baseline named | Source |
|---|---|---|---|
| DualBird | "10x-30x faster data processing performance with 50%-90% lower costs. No changes required." | Not named on homepage; Iceberg benchmark compares vanilla Spark and "state-of-the-art C++ accelerated Spark" | dualbird.io |
| Apache Gluten | 3.3x on TPC-H and 3.0x on TPC-DS, up to 23x on a single query | Vanilla Spark | gluten.apache.org |
| Apache Auron | About 2x faster than Spark 3.5 on TPC-DS, about 50% cluster resources saved | Spark 3.5 | auron.apache.org |
| Flarion | 3x performance, 60% cost reduction | Not stated | flarion.io |
| Apache DataFusion Comet | TPC-DS at 1 TB speedup over stock Spark, per-query breakdown in the Benchmarking Guide | Stock Apache Spark | datafusion.apache.org/comet |
| RAPIDS Accelerator | Not reviewed for this edition; the Qualification Tool estimates speedup from your own event logs | Your own jobs | docs.nvidia.com |
| Databricks Photon | No single headline multiple on the documentation page; documents where it does not help (queries under 2 seconds, UDFs) | Databricks Runtime without Photon | docs.databricks.com |
Vendor figures. Not measured by ETL Compare.
Also check the baseline. AWS states that Amazon EMR's Spark runtime is up to 5.4x faster than open-source Apache Spark, so an accelerator measured against open-source Spark and one measured against the EMR runtime are not starting from the same place.
What should you check before you buy any of them?
Before you buy
- Find the slow stage first. Open the Spark UI Stages tab and compare Shuffle Read Size, Shuffle spill (disk) and task duration spread. An accelerator that speeds up compute will not help a job that is waiting on shuffle fetches.
- Price the whole job, not the hour. A 3x speedup on an instance that costs 2x per hour halves the saving. See Spark cost per job, explained.
- Test on one real pipeline at two or three cluster sizes. A single benchmark point can hide whether a product scales or only wins at one configuration.
Head-to-head comparisons
DualBird vs RAPIDS Accelerator
Two ways to change the execution hardware under Spark: F2 instances on AWS or NVIDIA GPU instances on most platforms.
DualBird vs Apache Gluten
A commercial, AWS-only product aimed at spill and shuffle against the most widely contributed open-source native engine.
DualBird vs DataFusion Comet
Vendor-supported and tuning-light against free and tuning-heavy, for teams on EMR or EKS.
Apache Gluten vs DataFusion Comet
The two main open-source native engines for self-managed Spark, and how to pick between them.
All 36 head-to-head pages, alternatives pages and a side-by-side table: Compare Spark accelerators. To apply your own weights: calculator.
What is new in Spark acceleration?
Flarion
Flarion argues Spark will keep its API while its engine is replaced
Flarion's Ran Reichman writes that Spark remains the main data processing platform because of its scale, network effects and the cost of migrating away, and expects its execution layer to be replaced underneath a stable API.
Source: Flarion blog
DualBird
DualBird explains how its pipeline handles skew-driven spill
In the second part of its "Faster Spark" series, DualBird co-founder and chief architect Ehud Eliaz explains how data skew causes spills and straggler tasks in Spark. He states that DualBird keeps only a handful of partitions live at once, so each one has a larger memory budget.
Source: DualBird blog
Background: University · Notes
What have we published lately?
A buyer's checklist for Spark ETL accelerators
Twenty questions, in five groups, to answer before the first trial run.
What Spark accelerators cost: what each vendor publishes about price
Some products have a cost you can model from public numbers, and some need a quote. Here is what is published, and how to compare the rest.
How to run a Spark accelerator proof of concept on your own ETL pipeline
A trial plan that produces a decision, not a demo: one real pipeline, a logged baseline, three cluster sizes and a threshold set in advance.
Frequently asked questions
What is the best Spark ETL accelerator in 2026?
On our lead ranking, which asks which accelerator speeds up the Spark ETL you already run, on the platform you already use, without code changes or a dedicated tuning team, Apache Gluten (3.73), DualBird (3.72), DataFusion Comet (3.67) and RAPIDS Accelerator (3.65) are the top four. The best choice depends on your platform and your slowest stage: Photon if you are already on Databricks, DualBird if you run on EMR or EKS and your jobs are dominated by shuffle and disk spill, the RAPIDS Accelerator if you already run GPUs.
Do Spark accelerators require code changes?
None of the nine require changes to your Spark SQL or DataFrame code, according to their own documentation. They differ in setup: open-source plugins need jars, off-heap memory and shuffle-manager configuration; the RAPIDS Accelerator needs GPU instances; DualBird states you change the instance type and add its plugin; Photon is a setting inside Databricks.
Is Amazon EMR a Spark accelerator?
No. Amazon EMR is a platform that runs Spark and ships its own optimized Spark runtime. Accelerators such as DualBird, the RAPIDS Accelerator and Flarion are documented to run on EMR, and open-source plugins can usually be installed on it. You do not have to leave EMR to use them.
Which accelerators run on Amazon EMR?
On the pages we reviewed, DualBird, the RAPIDS Accelerator and Flarion list Amazon EMR. Gluten, Comet and Auron do not publish EMR guides, but as Spark plugins they can generally be added where you control Spark configuration. Photon runs only on Databricks.
What makes a Spark ETL job slow?
Usually one of four things: shuffle (data moved between stages), disk spill (tasks running out of execution memory), skew (a few partitions much larger than the rest) and small files (too many tiny input or output files). The profiling guide shows how to tell which one you have from the Spark UI.
How were these scores produced?
From desk research on public vendor and project pages, fetched in September 2026, scored against seven published criteria with one published weight profile. We did not run the products. See How we score (/method).
More questions: Spark accelerator FAQ