The Reference You Need
Spark Scala Examples
Simple spark scala examples to help you quickly complete your data etl pipelines. Save time digging through the spark scala function api and instead get right to the code you need...
Page 7 of 9
-
translate in Spark Scala: Replace or Delete Characters in a DataFrame Column
translate performs character-by-character substitution within a string column. Each character in matchingString is replaced by the corresponding character in replaceString. If replaceString is shorter than matchingString, characters without a replacement are deleted entirely.
-
like, ilike, and rlike in Spark Scala DataFrames
like, ilike, and rlike are Column methods that match a string column against a pattern — like uses SQL LIKE wildcards, ilike does the same case-insensitively, and rlike matches against a full regular expression. All three return a boolean column.
-
contains, startsWith, and endsWith in Spark Scala DataFrames
contains, startsWith, and endsWith are Column methods that check whether a string column contains a substring, begins with a prefix, or ends with a suffix. They each return a boolean column — true, false, or null for null input.
-
instr and locate in Spark Scala: Find Substring Position in a DataFrame
instr and locate both find the position of a substring within a string column. They return the same result — the difference is just argument order and the fact that locate has an optional start-position parameter for finding occurrences beyond the first.
-
String Length Functions in Spark Scala: length, bit_length, and octet_length
Spark provides three functions for measuring string size: length counts characters, octet_length counts bytes, and bit_length counts bits. For ASCII text they all agree, but they diverge once you have Unicode characters — which matters whenever you're validating input lengths or working with encoded data.
-
Split Function: Split Strings into Arrays in Spark Scala DataFrames
split breaks a string column on a delimiter or regular expression pattern and returns an ArrayType column. It's the go-to function whenever you need to turn a delimited string — like a CSV field, a tag list, or a log line — into individual elements you can work with.
-
Substring Functions: substring and substring_index in Spark Scala DataFrames
substring and substring_index are two complementary ways to extract a portion of a string column. Use substring when you know the character position; use substring_index when you want to split on a delimiter.
-
Lpad and Rpad: Pad Strings in Spark Scala DataFrames
The lpad and rpad functions pad a string column to a specified length by adding characters to the left or right side. They're commonly used for zero-padding numbers, aligning text output, and formatting fixed-width fields.
-
InitCap: Convert Strings to Title Case in Spark Scala DataFrames
The initcap function converts a string column to title case — capitalizing the first letter of each word and lowercasing the rest. It's useful for normalizing names, addresses, and other text where consistent capitalization matters.
-
Lower and Upper: Convert String Case in Spark Scala DataFrames
The lower and upper functions convert string columns to lowercase and uppercase respectively. They're commonly used to normalize data for case-insensitive comparisons and consistent formatting.