Iceberg compaction on Spark: why it is expensive and what to measure

Frequent appends and streaming writes leave tables with many small files, and every reader then pays to open and plan them (see the small files problem). Iceberg ships Spark procedures for table maintenance. rewrite_data_files combines data files using a binpack, sort or zorder strategy; rewrite_manifests rewrites manifest files "for better query performance and metadata efficiency"; and expire_snapshots removes expired snapshot metadata. Compaction in this note means rewrite_data_files.

Binpack groups small files into larger ones. Sort and zorder also reorder rows by chosen columns, which helps later reads skip data but requires Spark to sort the data it rewrites.

In Spark terms a sort-based compaction is a full ETL job: a scan stage that reads and decodes Parquet, an exchange that shuffles rows to sort them, a sort that can spill to disk when a partition is larger than the memory its task gets, and a write stage that compresses and commits new files. DualBird's benchmark post describes sort-based compaction as one of the most compute-intensive jobs in a lakehouse because it "reads, decodes, sorts, shuffles, and re-encodes potentially hundreds of gigabytes of Parquet data". Because it runs on a schedule across many tables, its cost recurs.

The cost also grows with how far maintenance falls behind. A table that is compacted rarely has more small files per partition, so each run reads more files and rewrites more data, and a sort over a large partition is more likely to run out of task memory. Smaller, more frequent rewrites spread the work, but every run still pays the scan and write cost, which is why cost per TB rewritten is the number to track.

The Iceberg Spark procedures page lists the options and their defaults. These are the ones that most affect how much work a run does and how much memory it needs:

Two more options change the scale and risk of a run: partial-progress.enabled lets a long rewrite commit in parts, so a failure loses less work, and rewrite-all forces every matching file to be rewritten, which is the most expensive choice.

Test the strategy you actually run. A binpack rewrite and a sort rewrite of the same table stress different stages, and a result for one does not carry over to the other.

Of the seven products we track, DualBird is the one whose pages we reviewed include a compaction-specific benchmark. Its June 2026 post covers sort-based compaction of a 100 GB dataset of about 370 million rows and 64 mixed-type columns with ZSTD level 3 compression, against vanilla Spark and "state-of-the-art C++ accelerated Spark".

The post is written by DualBird staff. As noted in our DualBird review, cluster sizes and Spark versions are not stated and we found no third-party results.

Other products publish Iceberg results that are not about compaction. Comet 1.0.0 (7 August 2026) adds Iceberg table format V3 support with table encryption, and the AWS engineers' June 2026 post reports Comet 37% faster overall on Iceberg tables for their TPC-DS queries. The RAPIDS Accelerator's 26.08.0 release adds Iceberg 1.11 support with accelerated incremental append scans. We found no compaction-specific figures from Gluten, Auron, Photon or Flarion on the pages we reviewed.

1. Pick one real table and a fixed set of partitions to compact, and copy it so baseline and trial start from the same files.
2. Record files and bytes before and after, and the TB rewritten.
3. Compute cost per TB: cost per run divided by TB rewritten, using cost per job.
4. Record the stage profile: shuffle write, Shuffle spill (disk) and the slowest task.
5. Run sort and binpack separately if you use both, at two cluster sizes.
6. Check the result: row counts match and the new snapshot commits cleanly.

For the full trial method, see how to run a Spark accelerator proof of concept.