The Reference You Need
Spark Scala Examples
Simple spark scala examples to help you quickly complete your data etl pipelines. Save time digging through the spark scala function api and instead get right to the code you need...
Page 1 of 9
-
mode in Spark Scala: Find the Most Frequent Value in a DataFrame
The mode function is an aggregate that returns the most frequently occurring value within each group. It's handy for questions like "what's the best-selling product per store?" or "what's the most common rating on each survey?".
-
percentile_approx, median, and approx_count_distinct in Spark Scala DataFrames
percentile_approx, median, and approx_count_distinct are aggregate functions for summarizing distributions in a Spark Scala DataFrame. percentile_approx estimates one or more percentiles, median returns the middle value, and approx_count_distinct estimates the number of unique values — trading a little accuracy for a lot of speed on large datasets.
-
Covariance and Correlation in Spark Scala: corr, covar_samp, and covar_pop in a DataFrame
Covariance and correlation both measure how two numeric columns move together. Spark provides corr for the Pearson correlation coefficient (a normalized value between -1 and 1) and two covariance functions — covar_samp and covar_pop — that differ only in whether they treat your data as a sample or the whole population.
-
Skewness and Kurtosis in Spark Scala: Measure Distribution Shape in a DataFrame
skewness and kurtosis are aggregate functions that describe the shape of a numeric distribution rather than its center or spread. Skewness measures how asymmetric the values are, and kurtosis measures how heavy the tails are. Both are handy for spotting outliers and non-normal data before you rely on averages or standard deviations.
-
Standard Deviation and Variance in Spark Scala: stddev, variance, and Their Population Variants in a DataFrame
Standard deviation and variance measure how spread out the values in a numeric column are. Spark provides a full family of aggregate functions for both — stddev, stddev_samp, and stddev_pop for standard deviation, and variance, var_samp, and var_pop for variance. The only real decision you have to make is whether you want the sample or population form.
-
collect_list, collect_set, array_agg in Spark Scala: Aggregate Rows into an Array
collect_list and collect_set are aggregate functions that gather the values in each group into an array. collect_list keeps every value, including duplicates; collect_set keeps only the distinct ones. array_agg is a SQL alias for collect_list. They're the go-to tools when you want to roll many rows up into a single row that holds all their values.
-
first, last, first_value, last_value, any_value in Spark Scala: Picking Values from a Group
first and last are aggregate functions that return the earliest or latest value in a group — but only when the data is ordered. They're most useful as window functions paired with an orderBy clause, where "first" and "last" actually mean something specific. first_value and last_value are SQL-only synonyms, and any_value returns an arbitrary value when you don't care which one you get.
-
min, max, min_by, max_by in Spark Scala: Find Extremes in a DataFrame
min and max are aggregate functions that return the smallest and largest values in a column. min_by and max_by go a step further — they return the value of one column at the row where another column reaches its extreme, so you can answer questions like "which employee earns the highest salary?" rather than just "what is the highest salary?".
-
avg and mean in Spark Scala: Compute Column Averages in a DataFrame
avg and mean are aggregate functions that compute the arithmetic mean of values in a numeric column. They're the workhorse functions for summarizing data — average salary by department, average order size by region, moving average of a stock price. The two functions are identical; mean is just an alias for avg.
-
sum in Spark Scala: Aggregate Column Totals in a DataFrame
sum is an aggregate function that totals the values in a numeric column. It's one of the most common operations in Spark Scala — used to roll up sales totals, computed metrics, running balances, and just about any group-level numeric summary.