How to profile a slow Spark ETL job before buying anything

Open the Spark UI for one representative run of the job (the Spark History Server for a finished run). In the Jobs tab, find the job that takes most of the wall-clock time, then open its stages. Sort the stages by duration. In most ETL jobs one or two stages account for most of the runtime; everything after this step is about those stages.

Record four numbers per slow stage for your baseline: duration, number of tasks, input size, and shuffle read and write size. You will compare against them after every change, including any accelerator trial.

Open the slow stage and look at the task metrics the Spark UI reports: Shuffle Read Size / Records, Shuffle Read Fetch Wait Time, Shuffle Remote Reads and Shuffle Write Time.

What an accelerator can do: several products in this guide document shuffle work (Gluten's ColumnarShuffleManager, Comet's native shuffle, the RAPIDS Shuffle Manager, Auron's compacted shuffle format, and DualBird, which states it reduces shuffle data). None of them removes a shuffle that the query plan requires. If an avoidable join is shuffling, fix the plan first.

Sources: spark.apache.org/docs/latest/web-ui.html, spark.apache.org/docs/latest/sql-performance-tuning.html · Fetched 27 Sep 2026

In the same stage view, check Shuffle spill (memory) and Shuffle spill (disk). The Spark UI defines Shuffle spill (memory) as the size of the deserialized form of the shuffled data in memory, and Shuffle spill (disk) as the size of the serialized form of the data on disk. Any non-zero disk spill on a slow stage means tasks ran out of execution memory and wrote intermediate data to local disk, then read it back.

Check Peak execution memory and GC time next. High GC time alongside spill points to memory pressure on the executors.

Why spill keeps coming back after you add memory: executors share execution memory among the tasks running on them, so adding cores per executor can reduce memory per task even when total memory grows. DualBird's article 'A Simpler Spark' walks through this sharing rule with a 4-task, 16 GiB example.

Common fixes without new software: fewer concurrent tasks per executor (fewer cores per executor or more memory per core), more shuffle partitions so each task handles less data, and removing skew (step 4). What an accelerator can do: DualBird states it eliminates disk spills; Gluten and Comet document native spill with limits (experimental in Gluten's Velox backend, bounded per task in Comet); Auron documents multi-level memory management. Check these claims on your worst-spilling stage.

Sources: spark.apache.org/docs/latest/web-ui.html, dualbird.io/resources/blog/a-simpler-spark · Fetched 27 Sep 2026

Open the summary metrics for the slow stage and compare the median and max task duration, and the median and max shuffle read size. If the max is many times the median, a few partitions hold far more data than the rest, and the stage waits on those few tasks (stragglers).

Adaptive query execution handles one common case. With spark.sql.adaptive.skewJoin.enabled (default true), Spark splits skewed partitions in sort-merge joins. A partition counts as skewed when it is larger than spark.sql.adaptive.skewJoin.skewedPartitionFactor (default 5.0) times the median partition size and also larger than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes (default 256MB). Skew in aggregations or in joins below those thresholds is not covered.

Other fixes: salting hot keys, filtering or pre-aggregating hot keys separately, and broadcasting the small side of a join. Skew and spill often come together: the oversized task is also the one that spills. An accelerator does not change the distribution of your keys; the question to ask a vendor is what happens to the oversized task.

Sources: spark.apache.org/docs/latest/sql-performance-tuning.html · Fetched 27 Sep 2026

Look at the first (read) stage: if the number of tasks is very high and the input per task is tiny, the job is spending its time opening files and planning, not processing. spark.sql.files.maxPartitionBytes (default 128 MB) sets the maximum bytes packed into one read partition for file sources such as Parquet, JSON and ORC.

On the write side, count the files a job produces per partition. Thousands of small output files slow every downstream reader. Fixes: coalesce or repartition before writing, and for table formats such as Apache Iceberg, run compaction. Compaction is itself a heavy Spark job; DualBird's public benchmark happens to be a sort-based Iceberg compaction job, which makes it a relevant workload to ask vendors about.

1. Which of my slow stages does your product address, and what falls back to standard Spark?
2. Does it need code changes, a platform move or a different instance type?
3. What is the cost per job on my workload, including your fee and any change in instance price?
4. Can we test on one of my real pipelines, at two or three cluster sizes, against my current runtime (for example the Amazon EMR runtime, not open-source Spark)?
5. Who supports a failed job at 3 a.m., and what happens if we remove the plugin?

Run the trial against the baseline you recorded in step 1. Compare stage by stage: if the accelerator speeds up a stage that was not your bottleneck, the total will barely move.