Spark Scala Covariance and Correlation

Covariance and correlation both measure how two numeric columns move together. Spark provides corr for the Pearson correlation coefficient (a normalized value between -1 and 1) and two covariance functions — covar_samp and covar_pop — that differ only in whether they treat your data as a sample or the whole population.

Correlation with corr

corr is the one you'll reach for most often. It returns the Pearson correlation coefficient between two columns — a single number between -1 and 1 that describes both the direction and strength of a linear relationship. It has two overloads, one taking column names as strings and one taking Column values:

def corr(columnName1: String, columnName2: String): Column

def corr(column1: Column, column2: Column): Column

Here we measure how strongly advertising spend and revenue move together within each region:

val df = Seq(
  ("North", 10.0, 100.0),
  ("North", 20.0, 210.0),
  ("North", 30.0, 290.0),
  ("South", 5.0, 50.0),
  ("South", 15.0, 130.0),
  ("South", 25.0, 240.0),
).toDF("region", "ad_spend", "revenue")

val df2 = df
  .groupBy("region")
  .agg(
    corr(col("ad_spend"), col("revenue")).as("corr"),
  )
  .orderBy("region")

df2.show(false)
// +------+------------------+
// |region|corr              |
// +------+------------------+
// |North |0.9958705948858224|
// |South |0.9958705948858224|
// +------+------------------+

Both regions come back close to 1.0, telling us that as ad spend goes up, revenue goes up almost in lockstep. The result is always a Double.

Reading the correlation coefficient

The value corr returns is easiest to understand at its extremes:

• 1.0 — a perfect positive linear relationship. When one column rises, the other rises proportionally.
• -1.0 — a perfect negative linear relationship. When one column rises, the other falls proportionally.
• 0.0 — no linear relationship at all.

This example builds three groups, each with a different kind of relationship, so you can see all three landmarks at once:

val df = Seq(
  ("Positive", 1.0, 1.0),
  ("Positive", 2.0, 2.0),
  ("Positive", 3.0, 3.0),
  ("Positive", 4.0, 4.0),
  ("Negative", 1.0, 4.0),
  ("Negative", 2.0, 3.0),
  ("Negative", 3.0, 2.0),
  ("Negative", 4.0, 1.0),
  ("None", 1.0, 3.0),
  ("None", 2.0, 1.0),
  ("None", 3.0, 4.0),
  ("None", 4.0, 2.0),
).toDF("relationship", "x", "y")

val df2 = df
  .groupBy("relationship")
  .agg(
    corr(col("x"), col("y")).as("corr"),
  )
  .orderBy("relationship")

df2.show(false)
// +------------+----+
// |relationship|corr|
// +------------+----+
// |Negative    |-1.0|
// |None        |0.0 |
// |Positive    |1.0 |
// +------------+----+

Because corr is normalized, it's directly comparable across columns measured on completely different scales — a correlation of 0.8 means the same strength of relationship whether your columns are dollars, degrees, or counts. That normalization is what makes it more useful than raw covariance for most analysis.

Sample vs population covariance: covar_samp and covar_pop

Covariance measures the same "do these move together" idea as correlation, but unnormalized — its magnitude depends on the scale of the data, so it's harder to compare across different column pairs. A positive covariance means the two columns tend to rise together; a negative one means they move in opposite directions.

Like the standard deviation and variance functions, covariance comes in a sample form and a population form that differ only in their denominator:

• covar_samp divides by n - 1. Use it when your rows are a sample drawn from a larger population.
• covar_pop divides by n. Use it when your rows are the entire population you care about.

def covar_samp(columnName1: String, columnName2: String): Column

def covar_pop(columnName1: String, columnName2: String): Column

Both also have Column-based overloads. Because the sample form divides by a smaller number, it always produces a larger magnitude than the population form for the same data:

val df = Seq(
  ("North", 10.0, 100.0),
  ("North", 20.0, 210.0),
  ("North", 30.0, 290.0),
  ("South", 5.0, 50.0),
  ("South", 15.0, 130.0),
  ("South", 25.0, 240.0),
).toDF("region", "ad_spend", "revenue")

val df2 = df
  .groupBy("region")
  .agg(
    covar_samp(col("ad_spend"), col("revenue")).as("covar_samp"),
    covar_pop(col("ad_spend"), col("revenue")).as("covar_pop"),
  )
  .orderBy("region")

df2.show(false)
// +------+----------+-----------------+
// |region|covar_samp|covar_pop        |
// +------+----------+-----------------+
// |North |950.0     |633.3333333333334|
// |South |950.0     |633.3333333333334|
// +------+----------+-----------------+

The covariance is a large positive number here, confirming that ad spend and revenue rise together — but its actual value (950.0) is tied to the scale of the columns and isn't meaningful on its own. That's exactly why corr, which normalizes covariance by the standard deviations of both columns, is usually the more practical choice when you want to compare relationships.

Nulls and single-value groups

All three functions skip rows where either column is null — a pair only counts if both values are present. Two more edge cases are worth knowing before relying on these in production:

val df = Seq(
  ("North", Some(10.0), Some(100.0)),
  ("North", Some(20.0), None),
  ("North", Some(30.0), Some(290.0)),
  ("South", Some(5.0),  Some(50.0)),
  ("Solo",  Some(40.0), Some(400.0)),
).toDF("region", "ad_spend", "revenue")

val df2 = df
  .groupBy("region")
  .agg(
    count(when(col("ad_spend").isNotNull && col("revenue").isNotNull, 1)).as("complete_pairs"),
    corr(col("ad_spend"), col("revenue")).as("corr"),
    covar_samp(col("ad_spend"), col("revenue")).as("covar_samp"),
    covar_pop(col("ad_spend"), col("revenue")).as("covar_pop"),
  )
  .orderBy("region")

df2.show(false)
// +------+--------------+----+----------+---------+
// |region|complete_pairs|corr|covar_samp|covar_pop|
// +------+--------------+----+----------+---------+
// |North |2             |1.0 |1900.0    |950.0    |
// |Solo  |1             |null|null      |0.0      |
// |South |1             |null|null      |0.0      |
// +------+--------------+----+----------+---------+

In the North group, the middle row has a null revenue, so it's dropped — only two complete pairs remain, and those two points sit on a perfect line, giving a corr of 1.0.

The Solo and South groups each have a single complete pair, which exposes the second edge case. corr and covar_samp come back as null: correlation needs at least two points to describe a relationship, and covar_samp divides by n - 1, which is zero for a single row. covar_pop, dividing by n, returns 0.0 — a single point has no spread from its own mean. If you'd rather see 0 than null, wrap the result with coalesce: coalesce(corr(col("ad_spend"), col("revenue")), lit(0.0)).

Related Functions

Correlation and covariance are built on the spread of each column, so they pair naturally with stddev and variance. For the center of a column, see avg and mean; for the shape of its distribution, see skewness and kurtosis.