Convert a Spark Scala Struct to a JSON String

Using a struct type in Spark Scala DataFrames offers different benefits, from type safety, more flexible logical structures, hierarchical data and of course working with structured data.

Sometimes however it can be really useful to convert structs to json. The benefits of using a struct can sometimes be a negative. Specifically if you need to use 'arbitrary' structures within a column. When this is the case JSON can be a lot more flexible.

For example:

val df = Seq(
  ("example"),
).toDF("example")

val df2 = df
  .withColumn("struct", struct(lit("hi").as("greeting"), lit("later").as("farewell")))
  .withColumn("json", to_json(col("struct")))

df2.show(false)
// +-------+-----------+------------------------------------+
// |example|struct     |json                                |
// +-------+-----------+------------------------------------+
// |example|{hi, later}|{"greeting":"hi","farewell":"later"}|
// +-------+-----------+------------------------------------+

I will typically use this technique when I need to merge together structs with different definitions. When you encounter a situation like this JSON is much more flexible.