ETL CompareETL Compare

University · Running it · Lesson 9

Running a Spark accelerator in production: versions, fallback checks and rollback

ETL Compare staff · Published 13 September 2026 · 3 min read

In brief

Once an accelerator is in production, the work shifts from proving a speedup to keeping it: match every Spark, JDK and platform upgrade to the versions the product supports, watch for operators that start falling back, validate output after each change, and keep a tested way to switch the product off. Re-check cost per run every quarter, because instance prices, data volumes and releases all move.

How do Spark upgrades affect an accelerator?

Accelerators hook into Spark below its public API, so each Spark release needs a matching accelerator release. The 2026 calendar shows the spread. Apache Spark 4.2.0 shipped on 14 July 2026. NVIDIA cuDF for Apache Spark 26.08.0 lists Spark 4.2.0 support. Comet 1.0.0 lists Spark 4.2 as experimental and deprecates Spark 3.4 and JDK 11, with removal planned in 1.1.0. Gluten documents Spark up to 4.1. On managed platforms the Spark version comes with the platform release, so check the platform's release notes as well. Our note on Spark 4.x support tracks where each product stands.

How do you catch fallback creeping in?

Every product in this set falls back to standard Spark for work it cannot run natively, so a job keeps working when coverage drops. That is good for reliability and bad for visibility: after a code change, a new data type or an upgrade, a heavy stage can quietly move back to the JVM. Check the physical plan of your most expensive jobs after each change, and alert on runtime or cost per run rising past a threshold, not only on failures.

How do you validate output?

Compare row counts and checksums of output tables between an accelerated run and a standard Spark run on the same input, at least at every upgrade. In an August 2026 post on Spark upgrades, Flarion argues that proving output data is unchanged, not changing code, is the hardest part of a migration. The same discipline applies when you change the engine underneath Spark.

What does a rollback plan look like?

  • Keep the Spark configuration from before the accelerator in version control, next to the accelerated one
  • Know which settings to remove: plugin jars, spark.plugins or extension settings, the shuffle manager and off-heap sizing
  • For products tied to instance types, keep the capacity to run the old instance type, such as standard CPU nodes instead of GPU or F2 instances
  • Test the rollback once, on a real job, before you need it

Because the products sit under the Spark API, rolling back does not need code changes, but it does change runtime and cost, so plan the capacity for it.

What should you review every quarter?

  • Cost per run on your most expensive jobs, with current instance and license prices (cost per job)
  • The share of runtime in the stages the product addresses (profiling guide)
  • Supported Spark, JDK and platform versions against your upgrade plan
  • Support tickets and engineering time spent on tuning
  • Whether another product now fits your priorities better (calculator)

Related

Sources