Helping you Learn Spark Scala.

df.count() feels like it should be free — it's just counting rows. And on a plain table backed by Parquet or Iceberg, it more or less is: Spark can pull row counts straight out of the file footers. The moment you add a .filter() in front of it, that shortcut disappears. Spark has to read the filter column for every row to decide which ones to count, and every additional .count() you write re-runs the whole pipeline from scratch.

A MapType column is convenient for carrying a bag of key/value attributes through a pipeline, but most downstream work — filtering, joining, aggregating, writing tabular output — is easier when each key has its own column. This tutorial shows how to pull values out of a map with getItem, the apply() shorthand, what happens when a key is missing, and how to expand a map whose keys you don't know until runtime.

Event streams, webhook dumps, and log files rarely have a uniform shape — one record carries x and y coordinates, the next has a price, the next an email. Spark's JSON reader handles this gracefully, but the rules for how it handles it are worth knowing before you ship a pipeline. This tutorial walks through schema inference's union behavior, what happens when the same field has different types across records, supplying an explicit schema to stay defensive, capturing malformed rows with _corrupt_record, and the "keep it as raw JSON" escape hatch for genuinely heterogeneous nested payloads.

collect() is one of the most innocent-looking ways to crash a Spark job. It returns an Array[Row], which feels harmless — until you remember that the array has to fit in driver memory. This tutorial shows what collect() actually does, why it falls over on real datasets, and which APIs to reach for instead.

You join two DataFrames on what you thought was a primary key, expect the same number of rows you started with, and instead get more. This isn't a bug in Spark — it's how SQL joins are defined. Every row on the left gets matched to every matching row on the right, and if either side has duplicate keys, the output multiplies.

Everyone coming to Spark from Scala (or from Pandas) tries col("x") == "value" at least once. It looks right, it sometimes even compiles, and then nothing matches. Spark uses === for column equality — not == — and the reason traces back to a hard limit in the Scala language itself.

When a Spark job filters bad rows out of a dataset, you usually want to know how many — and ideally why. Accumulators are Spark's built-in tool for collecting metrics from worker tasks back to the driver. The built-in LongAccumulator handles a single counter; for per-reason breakdowns you need to write a custom one by extending AccumulatorV2.

JSON events, Protobuf payloads, and Avro records often arrive in Spark as DataFrames with structs nested several levels deep. Most analytical work is easier on a flat schema. This tutorial walks through three approaches — explicit dotted paths, the .* star expansion, and a recursive flatten function — and finishes with the name-collision trap that catches most people the first time.

Diffing two DataFrames — yesterday's snapshot against today's, or the output of a pipeline against an expected baseline — is one of the most useful patterns in Spark Scala. This tutorial walks through three approaches: exceptAll for full-row diffs, anti-joins for keyed adds and removes, and a full_outer join that classifies every row as ADDED, REMOVED, CHANGED, or UNCHANGED.

Once your team is sharing internal Spark libraries through a private Nexus, Artifactory, or CodeArtifact, you need a reliable way to publish new versions. Doing it from developer laptops causes version drift and credential sprawl. GitHub Actions can give you tagged, reproducible publishes — but wiring publishTo, credentials, and versioning into a workflow has a few sharp edges.

SQL Scripting reached GA and is enabled by default in Spark 4.1 (SPARK-54499), bringing variables, IF/CASE, WHILE/REPEAT/FOR loops, and DECLARE ... HANDLER error handling to pure SQL. Here's what it does, when it beats reaching for Scala, and how to run a script from a Spark Scala application.

There are three credible paths to native C++ execution for Spark in 2026: Databricks Photon (proprietary, mature, locked in), Apache Gluten with Velox (open source, newly graduated to ASF Top-Level Project), and Microsoft Fabric's Native Execution Engine (Gluten-based, managed). The choice maps cleanly onto your deployment model — here's how Scala teams should think about it.

Kubernetes finally has an official Apache Spark operator, a credible batch scheduler in YuniKorn, and most of the cloud vendor weight behind it. YARN still works, still runs on most of the on-prem Hadoop clusters, and isn't going anywhere this decade. Here's an honest framework for choosing between them when your Scala team has the option.

UNKNOWN

DuckLake is a new open lakehouse format from DuckDB Labs that puts table metadata in a standard SQL database — Postgres, MySQL, SQLite, or DuckDB — instead of writing thousands of small Avro and JSON files like Iceberg and Delta Lake do. v1.0 shipped in April 2026 under the MIT license. For Spark Scala teams the immediate story is not "rip out Iceberg," but the metadata-in-SQL idea is interesting enough to be worth understanding now.

MLflow 3.0 (released June 2025) rebuilt the platform around LoggedModel as a first-class entity, added GenAI tracing on top of OpenTelemetry, and reorganized how artifacts are stored. Most of the headline features land in the Python and TypeScript SDKs, but the JVM tracking client is still the path Scala teams use to log Spark ML runs from production code — and the changes underneath it are worth knowing before you upgrade.

Delta Lake's Universal Format generates Iceberg metadata alongside the Delta transaction log, against the same Parquet files, so Iceberg-native engines can read your Delta tables without conversion or duplication. With Delta 4.0.1 restoring Iceberg compat for Spark 4.0, UniForm is once again a usable option for Scala teams that need cross-engine reads — provided you understand the limitations.

Apache Polaris graduated to a top-level ASF project in February 2026 and is consolidating as the default open implementation of the Iceberg REST Catalog spec. For Spark Scala teams, it's the piece that lets Spark, Trino, and Flink work against the same Iceberg tables with one source of truth — without Hive Metastore, without per-engine catalog plumbing, and without vendor lock-in.

The official Apache Spark Kubernetes Operator launched as an ASF subproject in May 2025, built from scratch instead of forking the aging Kubeflow operator. A year of rapid releases later, it's at 0.9.0 and is the path the Spark community is steering toward for running Scala jobs on Kubernetes.

Spark 4.1's Real-Time Mode delivers single-digit millisecond latency for stateless queries and targets sub-300ms p99 on the Databricks runtime, putting Spark within striking distance of Flink for the first time. For most analytics streaming, CDC, and ML feature workloads, RTM closes the gap. For sub-10ms requirements, complex event processing, and true event-at-a-time semantics, Flink still wins — and probably will for a long time.