The Reference You Need
Spark Scala Examples
Simple spark scala examples to help you quickly complete your data etl pipelines. Save time digging through the spark scala function api and instead get right to the code you need...
Page 6 of 9
-
Sentences: Tokenize Text into Words and Sentences in Spark Scala DataFrames
sentences splits a string into an array of sentences, where each sentence is an array of words. It's useful for text analysis tasks like counting sentences, extracting individual words, or preparing text for downstream NLP processing.
-
levenshtein in Spark Scala: Measure String Distance in a DataFrame
The levenshtein function computes the Levenshtein distance between two string columns — the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform one string into the other. It's useful for fuzzy matching, typo detection, and deduplication.
-
soundex in Spark Scala: Phonetic Matching in a DataFrame Column
The soundex function returns the soundex code of a string column — a four-character phonetic encoding that groups similar-sounding names together. It's useful for fuzzy matching, deduplication, and search where exact spelling varies.
-
hex and unhex in Spark Scala: Hexadecimal Conversion in DataFrames
hex converts an integer or string column to its hexadecimal representation. unhex does the reverse — it decodes a hex string back to binary. These are useful when working with low-level data formats, color codes, or any system that uses hex encoding.
-
base64 and unbase64 in Spark Scala: Encode and Decode Binary Data in DataFrames
base64 encodes a binary or string column into a Base64-encoded string. unbase64 does the reverse — it decodes a Base64 string back into binary. Together they let you safely represent binary data as printable text, which is useful when passing data through systems that only handle strings.
-
ASCII and Char: Convert Between Characters and Code Points in Spark Scala DataFrames
The ascii function returns the numeric code point of the first character in a string column. The chr and char SQL functions do the reverse — they convert an integer code point back to a character. Together they let you move between characters and their numeric representations.
-
reverse in Spark Scala: Reverse Strings in a DataFrame Column
The reverse function reverses the character order of a string column. It also works on array columns, reversing the element order — but this article focuses on string usage.
-
Repeat and Space: Duplicate Strings in Spark Scala DataFrames
The repeat function duplicates a string column a specified number of times. The space SQL function generates a string of spaces — it's shorthand for repeat(" ", n). Together they cover common needs like building separators, indenting text, and padding output.
-
replace in Spark Scala: Replace Substrings in a DataFrame Column
replace substitutes all occurrences of a substring within a string column. It's the straightforward choice when you need a literal find-and-replace without regular expressions.
-
overlay in Spark Scala: Replace or Insert Characters by Position in a DataFrame Column
overlay replaces a portion of a string column starting at a given position with a replacement string. It works like the SQL standard OVERLAY function and is useful for masking, patching, or inserting text at a specific character position.