Spark Scala Skewness and Kurtosis

skewness and kurtosis are aggregate functions that describe the shape of a numeric distribution rather than its center or spread. Skewness measures how asymmetric the values are, and kurtosis measures how heavy the tails are. Both are handy for spotting outliers and non-normal data before you rely on averages or standard deviations.

Skewness: measuring asymmetry

skewness returns the sample skewness of the values in a group. It comes in two overloads — one taking a column name as a string and one taking a Column:

def skewness(columnName: String): Column

def skewness(e: Column): Column

The result is a Double you read by its sign:

• 0 — the distribution is symmetric around its mean.
• Positive — the distribution has a longer tail on the right; a few large values pull the mean above the bulk of the data.
• Negative — the distribution has a longer tail on the left; a few small values pull the mean below the bulk of the data.

Here are three groups shaped to make the difference obvious — one symmetric, one with a single large value, and one with a single small value:

val df = Seq(
  ("symmetric",  10.0),
  ("symmetric",  20.0),
  ("symmetric",  30.0),
  ("symmetric",  40.0),
  ("symmetric",  50.0),
  ("right_skew", 10.0),
  ("right_skew", 12.0),
  ("right_skew", 14.0),
  ("right_skew", 16.0),
  ("right_skew", 90.0),
  ("left_skew",  10.0),
  ("left_skew",  84.0),
  ("left_skew",  86.0),
  ("left_skew",  88.0),
  ("left_skew",  90.0),
).toDF("shape", "value")

val df2 = df
  .groupBy("shape")
  .agg(skewness(col("value")).as("skewness"))
  .orderBy("shape")

df2.show(false)
// +----------+-------------------+
// |shape     |skewness           |
// +----------+-------------------+
// |left_skew |-1.4842774436056696|
// |right_skew|1.4842774436056696 |
// |symmetric |0.0                |
// +----------+-------------------+

The symmetric group is evenly spaced, so its skewness is exactly 0. The right_skew and left_skew groups are mirror images of each other, so they produce the same magnitude with opposite signs.

Kurtosis: measuring tailedness

kurtosis measures how much of a distribution's variance comes from extreme, tail-end values versus values near the mean. Like skewness, it has both a string and a Column overload:

def kurtosis(columnName: String): Column

def kurtosis(e: Column): Column

One thing to know up front: Spark returns excess kurtosis, meaning the kurtosis of a normal distribution is defined as 0 rather than 3. That makes the sign easy to interpret:

• Positive — heavy tails: values cluster tightly around the mean but with a few far-out outliers (leptokurtic).
• Negative — light tails: values are spread out fairly evenly with no strong peak (platykurtic).

The heavy_tails group below is mostly a single repeated value with two outliers, while flat is an evenly spread set:

val df = Seq(
  ("heavy_tails", 50.0),
  ("heavy_tails", 50.0),
  ("heavy_tails", 50.0),
  ("heavy_tails", 50.0),
  ("heavy_tails", 50.0),
  ("heavy_tails", 50.0),
  ("heavy_tails", 50.0),
  ("heavy_tails", 1.0),
  ("heavy_tails", 100.0),
  ("flat",        10.0),
  ("flat",        30.0),
  ("flat",        50.0),
  ("flat",        70.0),
  ("flat",        90.0),
).toDF("shape", "value")

val df2 = df
  .groupBy("shape")
  .agg(kurtosis(col("value")).as("kurtosis"))
  .orderBy("shape")

df2.show(false)
// +-----------+------------------+
// |shape      |kurtosis          |
// +-----------+------------------+
// |flat       |-1.3              |
// |heavy_tails|1.5009521218637687|
// +-----------+------------------+

The tightly clustered heavy_tails group has positive kurtosis because its two outliers dominate the tails, while the evenly spread flat group comes back negative.

Using them together

In practice you'll often compute both alongside your usual aggregates to sanity-check a column before trusting its mean. Here we compare two stores' daily order values — one has a single unusually large order, the other is tightly consistent:

val df = Seq(
  ("Store A", 20.0),
  ("Store A", 25.0),
  ("Store A", 30.0),
  ("Store A", 22.0),
  ("Store A", 200.0),
  ("Store B", 48.0),
  ("Store B", 52.0),
  ("Store B", 50.0),
  ("Store B", 49.0),
  ("Store B", 51.0),
).toDF("store", "order_value")

val df2 = df
  .groupBy("store")
  .agg(
    skewness(col("order_value")).as("skewness"),
    kurtosis(col("order_value")).as("kurtosis"),
  )
  .orderBy("store")

df2.show(false)
// +-------+------------------+-------------------+
// |store  |skewness          |kurtosis           |
// +-------+------------------+-------------------+
// |Store A|1.4914731113016217|0.23844695794772797|
// |Store B|0.0               |-1.3               |
// +-------+------------------+-------------------+

Store A's lone $200 order gives it a clearly positive skewness — a signal that its mean order value is being dragged up by an outlier and may not represent a typical order. Store B is symmetric and evenly spread, so both measures sit near zero.

Nulls and single-value groups

Two edge cases are worth knowing before you rely on these functions in production.

First, nulls are skipped, just like the other aggregates — they don't count toward the group and don't affect the result.

Second, both functions need enough spread to be defined. A group with a single value has zero standard deviation, so the calculation divides by zero and Spark returns null rather than raising an error.

val df = Seq(
  ("Store A", Some(20.0)),
  ("Store A", Some(25.0)),
  ("Store A", None),
  ("Store A", Some(30.0)),
  ("Store A", Some(200.0)),
  ("Store B", Some(48.0)),
  ("Store C", Some(50.0)),
).toDF("store", "order_value")

val df2 = df
  .groupBy("store")
  .agg(
    count("order_value").as("non_null_values"),
    skewness(col("order_value")).as("skewness"),
    kurtosis(col("order_value")).as("kurtosis"),
  )
  .orderBy("store")

df2.show(false)
// +-------+---------------+------------------+-------------------+
// |store  |non_null_values|skewness          |kurtosis           |
// +-------+---------------+------------------+-------------------+
// |Store A|4              |1.1471821489008156|-0.6724480369293677|
// |Store B|1              |null              |null               |
// |Store C|1              |null              |null               |
// +-------+---------------+------------------+-------------------+

In Store A, the null order value is ignored, leaving four values — enough to compute both measures. Stores B and C each have a single non-null value, so both skewness and kurtosis come back as null. If you'd rather see 0 than null for these groups, wrap the result with coalesce: coalesce(skewness(col("order_value")), lit(0.0)).

Related Functions

Skewness and kurtosis pair naturally with the other summary statistics. For the spread of a column, see standard deviation and variance. For the center, see avg and mean. For the smallest and largest values in a group, see min and max.