Setting JVM Options in sbt to Avoid Spark Test OOMs

Spark spins up a full local mini-cluster inside your test JVM — drivers, executors, shuffle infrastructure — which needs far more heap than sbt's default. Two settings in build.sbt fix this: forking the test JVM and sizing it properly.

Why Spark Tests Run Out of Memory

When you run sbt test, all your test code runs inside the sbt process itself by default. That process was launched with whatever heap sbt started with — often 256MB or less. A SparkSession in local mode still initializes the full driver, starts an in-process executor, and allocates shuffle and broadcast buffers. On a modest dataset that works fine. As your test suite grows, or as your tests use larger DataFrames, you'll start seeing:

java.lang.OutOfMemoryError: Java heap space

or the subtler:

java.lang.OutOfMemoryError: GC overhead limit exceeded

The GC overhead error is often worse — it means Spark is technically running but spending the majority of its time garbage collecting instead of doing work. Tests become unpredictably slow before they finally crash.

The fix is not to pass memory flags to sbt on the command line (though you can). The right approach is to encode the configuration in build.sbt so everyone on the team gets the same behavior without needing to remember flags.

Step 1: Fork the Test JVM

By default, javaOptions in sbt has no effect on tests — those options are only applied to forked processes. You must enable forking first:

// build.sbt
Test / fork := true

With fork := true, sbt launches a fresh JVM process to run your tests instead of running them inside the sbt shell process. This is the prerequisite for every other JVM option to take effect.

Forking also has a useful side effect: each test run starts with a clean JVM, which prevents state leaking between runs. Static state in Spark's internal registries can cause confusing intermittent failures in long sbt sessions — forking eliminates that.

Step 2: Set the Heap Size

Now that tests run in a forked process, you can set the heap size for that process:

// build.sbt
Test / fork := true
Test / javaOptions ++= Seq(
  "-Xms512m",
  "-Xmx4g",
)

-Xms512m sets the initial heap. -Xmx4g sets the maximum. Spark will use what it needs up to that ceiling.

How much to allocate: For a typical Spark unit test suite with small DataFrames (hundreds of rows), 2g is usually enough. If your tests use DataFrames with millions of rows or run many tests in sequence, 4g is a safer starting point. If you're still seeing OOMs after raising -Xmx, the next section explains why.

Step 3: Disable Parallel Test Execution

sbt runs test suites in parallel by default. With Spark, this compounds the memory problem: each test suite that creates a SparkSession is running a separate in-process cluster simultaneously. Three parallel test suites means three sets of Spark infrastructure all competing for the same heap.

Disable parallel execution:

// build.sbt
Test / fork := true
Test / javaOptions ++= Seq(
  "-Xms512m",
  "-Xmx4g",
)
Test / parallelExecution := false

With parallelExecution := false, sbt runs one test suite at a time. This often makes tests faster overall even though they're sequential — because each suite isn't fighting for memory and triggering GC pauses.

If you're using a shared SparkSession (via a SparkSessionWrapper trait or similar), parallel execution is also unsafe for correctness reasons: global Spark configuration set in one test can affect another. Sequential execution prevents this entire class of bug.

Putting It Together

A complete build.sbt for a Spark Scala project:

name := "myproject"

scalaVersion := "2.13.11"
val sparkVersion = "3.4.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "com.lihaoyi" %% "utest" % "0.8.1" % "test",
)

testFrameworks += new TestFramework("utest.runner.Framework")

Test / fork := true
Test / javaOptions ++= Seq(
  "-Xms512m",
  "-Xmx4g",
)
Test / parallelExecution := false

These three settings — fork, javaOptions, parallelExecution — are the minimum viable configuration for a Spark test suite that doesn't randomly OOM.

Additional JVM Flags for Spark

For larger test suites or if you're still seeing GC pressure after sizing the heap, two additional flags are worth knowing:

Test / javaOptions ++= Seq(
  "-Xms512m",
  "-Xmx4g",
  "-XX:+UseG1GC",
  "-XX:G1HeapRegionSize=32m",
)

G1GC handles mixed workloads with short-lived allocations (typical of Spark's internal processing) better than the default GC in many JVM versions. The region size hint helps G1 avoid treating Spark's larger internal buffers as humongous objects, which can cause promotion failures.

These are worth trying if your tests are slow due to GC but not necessarily OOMing. For most projects the three-setting baseline is enough and you can add GC tuning only if you have evidence of a problem.

Verifying the Configuration Works

Run your test suite and check that sbt is actually forking:

sbt test
// [info] Forking test run to separate JVM
// ...

The Forking test run to separate JVM line confirms that fork := true is in effect and your javaOptions are being applied.

If you want to run a single test suite rather than the full suite, see running a specific test with sbt — the same fork and memory settings apply.