The Reference You Need
Spark Scala Examples
Simple spark scala examples to help you quickly complete your data etl pipelines. Save time digging through the spark scala function api and instead get right to the code you need...
Page 3 of 9
-
ceil, floor, and rint in Spark Scala: Rounding to Integers in a DataFrame
The ceil, floor, and rint functions round a numeric column to an integer. ceil rounds up toward positive infinity, floor rounds down toward negative infinity, and rint rounds to the nearest integer using banker's rounding for exact halves. ceil and floor also accept a scale argument to round to a specific number of decimal places.
-
round and bround in Spark Scala: Rounding Numeric Columns in a DataFrame
The round and bround functions round numeric columns to a given number of decimal places. They differ in how they handle exact halves: round rounds half away from zero (the most common convention), while bround uses banker's rounding, which rounds half to the nearest even number to reduce bias in large aggregations.
-
abs in Spark Scala: Compute the Absolute Value of a Numeric Column in a DataFrame
The abs function returns the absolute value of a numeric column — the value with its sign stripped. It works on any numeric type (integers, longs, doubles, decimals) and is most often used when you care about the magnitude of a number but not its direction.
-
Time Windows in Spark Scala: window, session_window, and window_time for DataFrame Aggregations
When you need to aggregate event-time data — page views per 5 minutes, average price every minute, user activity per session — Spark provides window, session_window, and window_time. These functions bucket timestamps into intervals so you can group and aggregate them with the regular DataFrame API. They work on both batch and streaming DataFrames; the examples below all use batch.
-
make_interval, make_dt_interval, and make_ym_interval in Spark Scala: Build Intervals in a DataFrame
The make_interval family builds interval values from numeric columns. Use make_interval for general-purpose intervals that mix years, months, days, and time-of-day components; use make_ym_interval when you need a strict year-month interval (e.g., subscription terms); and use make_dt_interval when you need a strict day-time interval (e.g., job durations). All three are Spark SQL functions called through expr().
-
make_date and make_timestamp in Spark Scala: Build Dates and Timestamps from Parts in a DataFrame
The make_date and make_timestamp family of functions construct date and timestamp values from separate numeric columns — year, month, day, and (for timestamps) hour, minute, and second. They're the inverse of extracting parts from a date: instead of pulling fields out, you assemble fields into a single value.
-
months_between in Spark Scala: Months Between Two Dates in a DataFrame
months_between returns the number of months between two date or timestamp columns as a Double. It's the right tool when you want a months gap rather than a days gap — tenure in months, age of a record, billing cycles, anything where calendar months matter more than raw day counts.
-
last_day and next_day in Spark Scala: Month-End and Next Weekday in a DataFrame
last_day returns the last day of the month that a given date falls in, and next_day returns the first date after a given date that lands on a particular weekday. Both are handy for building reporting periods, billing cycles, and scheduling logic.
-
date_from_unix_date and unix_date in Spark Scala: Convert Between Dates and Day Counts
date_from_unix_date converts a day count (days since 1970-01-01) into a calendar date, and unix_date does the reverse — turning a date into the number of days since the epoch. They're useful when your data uses integer day offsets for compact storage or interoperability with other systems.
-
from_utc_timestamp, to_utc_timestamp, and convert_timezone in Spark Scala: Shift Timestamps Between Time Zones in a DataFrame
from_utc_timestamp, to_utc_timestamp, and convert_timezone shift a timestamp column between time zones. Use them when your data is stored in one zone (almost always UTC for warehouse data) but you need to report, filter, or join in another — they handle daylight saving transitions for you, so you don't have to do offset math by hand.