The Reference You Need
Spark Scala Examples
Simple spark scala examples to help you quickly complete your data etl pipelines. Save time digging through the spark scala function api and instead get right to the code you need...
Page 6 of 9
-
decode and encode in Spark Scala: Convert Between Strings and Binary in DataFrames
encode converts a string column to binary using a specified character set. decode does the reverse — it converts binary data back to a string. Together they let you move between string and binary representations, which is useful when working with systems that expect raw bytes or when you need to control the character encoding explicitly.
-
Format String in Spark Scala: Printf-Style Column Formatting for DataFrames
The format_string function formats column values into a string using printf-style patterns. It's useful for building human-readable labels, combining columns into structured text, or formatting numbers inline without changing their type first.
-
find_in_set in Spark Scala: Search Comma-Delimited Strings in a DataFrame
find_in_set returns the 1-based position of a string within a comma-delimited list stored in another column. It returns 0 if the string isn't found and null if either input is null.
-
Sentences: Tokenize Text into Words and Sentences in Spark Scala DataFrames
sentences splits a string into an array of sentences, where each sentence is an array of words. It's useful for text analysis tasks like counting sentences, extracting individual words, or preparing text for downstream NLP processing.
-
levenshtein in Spark Scala: Measure String Distance in a DataFrame
The levenshtein function computes the Levenshtein distance between two string columns — the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform one string into the other. It's useful for fuzzy matching, typo detection, and deduplication.
-
soundex in Spark Scala: Phonetic Matching in a DataFrame Column
The soundex function returns the soundex code of a string column — a four-character phonetic encoding that groups similar-sounding names together. It's useful for fuzzy matching, deduplication, and search where exact spelling varies.
-
hex and unhex in Spark Scala: Hexadecimal Conversion in DataFrames
hex converts an integer or string column to its hexadecimal representation. unhex does the reverse — it decodes a hex string back to binary. These are useful when working with low-level data formats, color codes, or any system that uses hex encoding.
-
base64 and unbase64 in Spark Scala: Encode and Decode Binary Data in DataFrames
base64 encodes a binary or string column into a Base64-encoded string. unbase64 does the reverse — it decodes a Base64 string back into binary. Together they let you safely represent binary data as printable text, which is useful when passing data through systems that only handle strings.
-
ASCII and Char: Convert Between Characters and Code Points in Spark Scala DataFrames
The ascii function returns the numeric code point of the first character in a string column. The chr and char SQL functions do the reverse — they convert an integer code point back to a character. Together they let you move between characters and their numeric representations.
-
reverse in Spark Scala: Reverse Strings in a DataFrame Column
The reverse function reverses the character order of a string column. It also works on array columns, reversing the element order — but this article focuses on string usage.