Using Option[T] with toDF to Create Nullable Typed Columns in Spark Scala

The toDF implicit is the fastest way to hand-build DataFrames for tests, but the moment you need a numeric or boolean column that can hold a missing value, a bare null stops compiling. This tutorial shows how Scala's Option[T] fills that gap — making columns nullable, keeping their type, and letting None stand in for missing data — plus the one type-inference trap to watch for.

The Baseline: Plain Values Are Non-Nullable

When you build a DataFrame from a Seq of tuples, Spark derives each column's type — and its nullability — from the Scala types you hand it. A plain Int can never be null in Scala, so Spark marks the column nullable = false.

val df = Seq(
  ("Alice", 30),
  ("Bob", 25),
  ("Carol", 41),
).toDF("name", "age")

println("-------------- Schema --------------")
df.printSchema()

// root
//  |-- name: string (nullable = true)
//  |-- age: integer (nullable = false)

println("-------------- Result --------------")
df.show(false)

// +-----+---+
// |name |age|
// +-----+---+
// |Alice|30 |
// |Bob  |25 |
// |Carol|41 |
// +-----+---+

Notice the asymmetry: name is nullable = true because String is a reference type that can be null in Scala, but age is nullable = false. This is usually fine — until you need a row where the age is genuinely unknown.

Why a Bare null Won't Work

The instinct is to drop a null into the missing slot, the way you would for a string column. It doesn't compile:

val df = Seq(
  ("Alice", 30),
  ("Bob", null),   // <- does not compile
  ("Carol", 41),
).toDF("name", "age")

Scala has to infer a single element type for the second position across all three rows. Two of them are Int and one is Null, and the only common supertype is Any. Spark has no encoder for Any, so you get a compile-time error like this:

[ENCODER_NOT_FOUND] Not found an encoder of the type Any to Spark SQL internal
representation. Consider to change the input type to one of supported at...

The fix is to give the column a type that already includes the idea of "absent" — which is exactly what Option[T] is for.

The Fix: Wrap Values in Option[T]

Option[Int] has two shapes: Some(30) for a present value and None for a missing one. Because both are subtypes of Option[Int], Scala infers the column type as Option[Int], and Spark maps that to a nullable integer column with None rendered as null.

val df = Seq(
  ("Alice", Some(30)),
  ("Bob", None),
  ("Carol", Some(41)),
).toDF("name", "age")

println("-------------- Schema --------------")
df.printSchema()

// root
//  |-- name: string (nullable = true)
//  |-- age: integer (nullable = true)

println("-------------- Result --------------")
df.show(false)

// +-----+----+
// |name |age |
// +-----+----+
// |Alice|30  |
// |Bob  |null|
// |Carol|41  |
// +-----+----+

Two things changed compared to the baseline. The schema now reports age as nullable = true, and the Bob row shows null instead of failing to compile. The column is still a real integer — you didn't have to fall back to strings to represent the gap.

This is the whole trick, and it works for any encodable type: Option[Double], Option[Boolean], Option[java.sql.Timestamp], and so on.

Wider Rows: Case Classes with Option Fields

Tuples get unreadable fast once a row has more than three or four fields — you lose track of which position means what, and every nullable column adds another Some(...) wrapper to the noise. A case class with Option fields names each column at the type level and reads far better.

case class Employee(name: String, age: Option[Int], department: Option[String])

val df = Seq(
  Employee("Alice", Some(30), Some("Engineering")),
  Employee("Bob", None, Some("Sales")),
  Employee("Carol", Some(41), None),
).toDF()

println("-------------- Schema --------------")
df.printSchema()

// root
//  |-- name: string (nullable = true)
//  |-- age: integer (nullable = true)
//  |-- department: string (nullable = true)

println("-------------- Result --------------")
df.show(false)

// +-----+----+-----------+
// |name |age |department |
// +-----+----+-----------+
// |Alice|30  |Engineering|
// |Bob  |null|Sales      |
// |Carol|41  |null       |
// +-----+----+-----------+

The column names come straight from the case class fields, so you can call .toDF() with no arguments. Wrapping department in Option[String] is optional here — a plain String field is already nullable — but doing it consistently makes "this field can be missing" explicit at the type level, which is worth it in a schema definition you'll reuse across tests.

The Trap: A Column That Is All None

There's one case where Option alone isn't enough. If every value in a column is None, Scala has nothing concrete to infer the element type from. None on its own has type None.type, and the best Scala can do across a column of them is Option[Nothing] — which Spark can't encode, so it fails to compile the same way a bare null did.

The fix is to state the element type explicitly. Option.empty[Int] is a None that already knows it's an Option[Int]:

val df = Seq(
  ("Alice", Option.empty[Int]),
  ("Bob", Option.empty[Int]),
  ("Carol", Option.empty[Int]),
).toDF("name", "age")

println("-------------- Schema --------------")
df.printSchema()

// root
//  |-- name: string (nullable = true)
//  |-- age: integer (nullable = true)

println("-------------- Result --------------")
df.show(false)

// +-----+----+
// |name |age |
// +-----+----+
// |Alice|null|
// |Bob  |null|
// |Carol|null|
// +-----+----+

(None: Option[Int]) works identically if you prefer that spelling. Either way you get a properly typed nullable integer column, even with no non-null value anywhere in it. You'll hit this most often when writing a test for the "column is entirely absent" edge case — annotate one value and the whole column resolves.

When to Use Which

Plain values — for columns that genuinely can't be missing. Keeps nullable = false in the schema, which documents the invariant.
Option[T] in tuples — the quick fix for a nullable numeric, boolean, or date column in a small hand-built DataFrame.
Case class with Option fields — once rows get wide, or when you want a named, reusable schema across a test suite.
Option.empty[T] / (None: Option[T]) — whenever a column has no non-null value for Scala to infer the type from.

For the broader picture on building test DataFrames — single-column sequences, tuples, and naming columns — see creating DataFrames in Spark Scala for testing with toDF. Once you've built your expected and actual DataFrames, comparing two DataFrames covers asserting they match. And because nullable columns are where three-valued logic bites, it's worth knowing why null =!= null returns null, not true before you write comparisons against these columns.