The VARIANT Data Type in Spark 4.0: Semi-Structured Data Without Schema Headaches

Spark 4.0 added a native VARIANT type (SPARK-45827) for storing semi-structured data in a compact binary format you can query directly — no upfront schema, no from_json ceremony on every read. The published benchmarks show roughly 8x faster reads than storing the same payload as a JSON string column, and Spark 4.1 adds shredding to push that further. This article shows how the Scala API works, when to reach for VARIANT, and when you still want a strongly typed StructType.

For the broader release picture, see What's New in Spark 4.0 for Scala Developers and Upgrading from Spark 3.x to Spark 4.0: A Practical Guide.

Why VARIANT Exists

Before Spark 4.0, every team handling JSON-shaped data ended up at the same fork in the road:

• Store JSON as StringType and parse on read. Cheap to write, painful to query. Every from_json call needs a schema, and every analyst either invents one inline or pastes one from a doc that's two months out of date. The string column is opaque to the optimizer.
• Pin a schema with StructType up front. Fast and type-safe, but every new field upstream is a schema migration. If the producer adds device.os.locale on Thursday, your from_json either silently drops it (returns null) or fails the job depending on your columnNameOfCorruptRecord settings.

VARIANT is the third option. It stores semi-structured data in a binary encoding that preserves the full nested structure, lets you query paths directly with dot/bracket syntax, and does not require you to declare a schema. The encoding is an open specification shared with Parquet and the Delta Lake project, so the format is not Spark-proprietary.

The performance gain comes from the fact that VARIANT is parsed once at ingest, not on every query. Databricks' published numbers (Runtime 15.0 with Photon) show 8x improvement over equivalent String columns for both nested and flat schemas. Spark 4.0 ships the core type and functions; Spark 4.1 adds shredding — projecting frequently-accessed paths into columnar storage so common queries skip the binary decode entirely.

The Core Functions

Spark 4.0 ships a small, focused set of VARIANT functions. They are all available from SQL; in Scala you reach them through expr() or selectExpr(), since most do not have direct Column method equivalents.

There is also the : colon-path syntax (raw:store.bicycle.price) which is shorthand for variant_get — handy in SQL, less useful from the Scala DSL.

Reading JSON into VARIANT

The natural entry point is parse_json. Given a string column of JSON, one call gets you a queryable VARIANT.

import org.apache.spark.sql.functions._
import spark.implicits._

val rawEvents = Seq(
  """{"user": "alice", "action": "click", "target": {"id": "btn_save", "page": "/checkout"}}""",
  """{"user": "bob",   "action": "purchase", "amount": 49.99, "items": ["widget_a"]}""",
  """{"user": "carol", "action": "view", "target": {"id": "card_3", "page": "/home"}, "tags": ["promo", "new_user"]}"""
).toDF("json")

val events = rawEvents.withColumn("payload", expr("parse_json(json)"))

events.printSchema()
// root
//  |-- json: string (nullable = true)
//  |-- payload: variant (nullable = true)

Note the schema: payload is now variant, a first-class type — not a struct, not a string. The three rows have three different shapes and that is fine. The VARIANT column does not need them to agree.

If your input is already non-string JSON (e.g., from a Kafka source where the value is parsed by the connector), you can build VARIANT values from arbitrary Spark expressions; parse_json is just the most common path.

Querying Nested Fields

This is where VARIANT pays off. You query paths directly, with optional type casts.

import org.apache.spark.sql.functions._

events.selectExpr(
  "variant_get(payload, '$.user', 'string')        AS user",
  "variant_get(payload, '$.action', 'string')      AS action",
  "variant_get(payload, '$.target.id', 'string')   AS target_id",
  "variant_get(payload, '$.amount', 'double')      AS amount"
).show(false)

// +-----+--------+---------+------+
// |user |action  |target_id|amount|
// +-----+--------+---------+------+
// |alice|click   |btn_save |NULL  |
// |bob  |purchase|NULL     |49.99 |
// |carol|view    |card_3   |NULL  |
// +-----+--------+---------+------+

A few things to notice:

• Missing paths return null. There is no exception when amount is absent on a click event or target.id is absent on a purchase. That's the point — VARIANT is for data where presence is not guaranteed.
• The path uses JSONPath-style $.foo.bar syntax. Bracket indexing works for arrays: $.items[0].
• The third argument is a Spark SQL type, not a Scala type. Pass it as a string ('string', 'double', 'array<string>', 'struct<id: string, page: string>').

If a value is present but cannot be cast to the requested type, variant_get throws under ANSI mode (which is now the default in Spark 4.0). When the source data is messy, swap to try_variant_get:

// Some rows have amount as a string like "49.99", some as a number, some absent.
events.selectExpr(
  "try_variant_get(payload, '$.amount', 'double') AS amount"
).show()

try_variant_get is to variant_get what try_cast is to cast. Reach for it at the expression level where null is the genuine semantic; keep variant_get everywhere else so the optimizer (and your future self) can see strict type expectations.

Extracting a Struct in One Shot

Plucking individual fields gets verbose. When you know the shape you want, cast the whole VARIANT (or a subpath) to a StructType:

events.selectExpr(
  "variant_get(payload, '$.target', 'struct<id: string, page: string>') AS target"
).show(false)

// +------------------+
// |target            |
// +------------------+
// |{btn_save, /checkout}|
// |NULL              |
// |{card_3, /home}   |
// +------------------+

This is the seam between schema-less and schema-bound. Store the data as VARIANT, then project the well-known parts of the payload into a typed struct at query time. New fields that the producer adds later are still in the underlying VARIANT — you just have not asked for them yet.

Inferring a Schema

When you don't know the shape, ask Spark. schema_of_variant_agg walks every row and returns a unified DDL string.

events.select(
  expr("schema_of_variant_agg(payload)").as("inferred")
).show(false)

// inferred:
// STRUCT<action: STRING, amount: DOUBLE,
//        items: ARRAY,
//        tags: ARRAY,
//        target: STRUCT<id: STRING, page: STRING>,
//        user: STRING>
</code></pre>  
</div>



This is a notebook tool, not a production pattern. Running `schema_of_variant_agg` over a 10 TB table is expensive. The right workflow is: ingest into VARIANT, run `schema_of_variant_agg` over a sample, paste the result into a `variant_get(..., 'struct<...>')` call, and ship that.

---

## SQL `null` vs JSON `null`

Two distinct concepts collide here. A column can be SQL `NULL` (the value is absent). A VARIANT value can encode a JSON `null` (the value is *present* and explicitly `null`). They are not the same and `is_variant_null` lets you tell them apart.


  

    

val mixed = Seq(
  """{"flag": null}""",
  """{"other": 1}""",
  null
).toDF("json")
  .withColumn("payload", expr("try_parse_json(json)"))
  .withColumn("flag", expr("variant_get(payload, '$.flag', 'string')"))

mixed.selectExpr(
  "flag",
  "flag IS NULL                                    AS sql_is_null",
  "is_variant_null(variant_get(payload, '$.flag')) AS variant_is_null"
).show()

// +----+-----------+---------------+
// |flag|sql_is_null|variant_is_null|
// +----+-----------+---------------+
// |NULL|       true|           true|   <- explicit JSON null
// |NULL|       true|          false|   <- key absent
// |NULL|       true|           NULL|   <- whole row was SQL NULL
// +----+-----------+---------------+