Spark Scala Mode
The mode function is an aggregate that returns the most frequently occurring value within each group. It's handy for questions like "what's the best-selling product per store?" or "what's the most common rating on each survey?".
def mode(e: Column): ColumnThe mode function first appeared in version 3.4.0 and is defined as:
Aggregate function: returns the most frequent value in a group.
Use it inside an agg after a groupBy. It counts how often each value appears within the group and returns the one that appears most often.
val df = Seq(
("Store A", "coffee"),
("Store A", "coffee"),
("Store A", "tea"),
("Store B", "tea"),
("Store B", "tea"),
("Store B", "juice"),
("Store B", "tea"),
).toDF("store", "top_seller")
val df2 = df
.groupBy("store")
.agg(mode(col("top_seller")).as("most_common"))
df2.show(false)
// +-------+-----------+
// |store |most_common|
// +-------+-----------+
// |Store B|tea |
// |Store A|coffee |
// +-------+-----------+
Store A sold coffee twice and tea once, so its mode is coffee. Store B sold tea three times, beating juice, so its mode is tea.
mode works with any comparable type, not just strings. Here it finds the most common integer rating:
val df = Seq(
("survey", 5),
("survey", 5),
("survey", 3),
("survey", 3),
("survey", 1),
).toDF("form", "rating")
val df2 = df
.groupBy("form")
.agg(mode(col("rating")).as("most_common_rating"))
df2.show(false)
// +------+------------------+
// |form |most_common_rating|
// +------+------------------+
// |survey|3 |
// +------+------------------+
Ties
Notice the previous example is actually a tie: both 5 and 3 appear twice. When there's a tie for the most frequent value, mode returns one of the tied values — but which one is not guaranteed. Don't rely on mode to break ties in a particular way. If deterministic tie-breaking matters, count the values explicitly and order them yourself:
val df = Seq(
("survey", 5),
("survey", 5),
("survey", 3),
("survey", 3),
("survey", 1),
).toDF("form", "rating")
val df2 = df
.groupBy("form", "rating")
.count()
.orderBy(col("count").desc, col("rating").desc)
df2.show(false)
// +------+------+-----+
// |form |rating|count|
// +------+------+-----+
// |survey|5 |2 |
// |survey|3 |2 |
// |survey|1 |1 |
// +------+------+-----+
Handling Nulls
mode ignores null values when counting. Nulls never win, even if they're the most common entry in the group:
val df = Seq(
("Store A", Some("coffee")),
("Store A", None),
("Store A", None),
("Store A", Some("coffee")),
("Store A", Some("tea")),
).toDF("store", "top_seller")
val df2 = df
.groupBy("store")
.agg(mode(col("top_seller")).as("most_common"))
df2.show(false)
// +-------+-----------+
// |store |most_common|
// +-------+-----------+
// |Store A|coffee |
// +-------+-----------+
Even though null appears twice — as often as coffee — the nulls are excluded from the count, so coffee is returned as the mode. If a group contains only nulls, mode returns null.
Related Functions
For finding the most and least extreme values in a group, see min and max. For counting values and distinct values within a group, see count and countDistinct.