The Reference You Need
Spark Scala Examples
Simple spark scala examples to help you quickly complete your data etl pipelines. Save time digging through the spark scala function api and instead get right to the code you need...
Page 2 of 9
-
count and countDistinct in Spark Scala: Aggregate Row and Distinct Value Counts in a DataFrame
count, countDistinct, and count_if are aggregate functions for counting rows in a Spark Scala DataFrame. count counts rows or non-null values, countDistinct counts unique values, and count_if counts rows that match a condition.
-
Shift Functions in Spark Scala: shiftleft, shiftright, and shiftrightunsigned in a DataFrame
The bitwise shift functions move the bits of an integer column left or right by a fixed number of positions. They're useful for packing and unpacking flags, fast multiplication or division by powers of two, and working with binary protocol data.
-
e and pi in Spark Scala: Euler's Number and Pi Constants in a DataFrame
e() returns Euler's number (≈ 2.71828) and pi() returns π (≈ 3.14159). They're handy when a Spark expression needs one of these mathematical constants — for circle math, exponential growth, trigonometry, and so on — without you having to hard-code the literal value.
-
width_bucket in Spark Scala: Equiwidth Histogram Buckets in a DataFrame
width_bucket assigns a numeric value to an equiwidth histogram bucket given a range and a bucket count. It's the right tool when you need to bin continuous values into fixed-size groups — age brackets, price tiers, score ranges — without writing a chain of when expressions.
-
pmod in Spark Scala: Positive Modulo for DataFrame Columns
The pmod function returns the positive remainder of dividing one column by another. Unlike the standard % operator, which mirrors the sign of the dividend, pmod keeps the result non-negative whenever the divisor is positive. That makes it the right tool for hash bucketing and any cyclic indexing where a negative remainder would point you at the wrong bucket.
-
hypot in Spark Scala: Compute the Hypotenuse of Two DataFrame Columns
The hypot function computes sqrt(a² + b²) for two numeric inputs without the intermediate overflow or underflow that a naive implementation would produce. It's the standard tool for distances between points, vector magnitudes, and anywhere the Pythagorean theorem applies.
-
signum and sign in Spark Scala: Get the Sign of a Numeric DataFrame Column
signum returns -1.0 for negative numbers, 0.0 for zero, and 1.0 for positive numbers. It's useful when you care about the direction of a value but not its magnitude — flagging gains vs. losses, classifying deltas, or branching on the sign of a difference. Spark also exposes the SQL-only aliases sign, negative, and positive for working with the sign of a column.
-
factorial in Spark Scala: Compute the Factorial of an Integer Column in a DataFrame
The factorial function returns the factorial of an integer column — the product of all positive integers up to and including the input value (n! = n × (n-1) × ... × 2 × 1). It's useful anywhere you need to compute permutations, combinations, or other counting expressions inline in a DataFrame.
-
degrees and radians in Spark Scala: Convert Between Angle Units on DataFrame Columns
The degrees and radians functions convert DataFrame columns between the two ways of measuring angles. radians turns degrees into radians; degrees does the reverse. They're the unit-conversion helpers you reach for whenever your data is in degrees but you need to feed it into Spark's trig functions, which all expect radians.
-
Trigonometric Functions in Spark Scala: sin, cos, tan and More on DataFrame Columns
Spark Scala exposes the full set of trigonometric functions from java.lang.Math as DataFrame column functions: the basics (sin, cos, tan), their inverses (asin, acos, atan, atan2), the reciprocals (cot, csc, sec), and the hyperbolic versions of all of them. Every input and output is in radians, not degrees — use the radians function to convert if your data is in degrees.