Iceberg v3 is the first format-version bump since 2021 and finally lands the features Spark Scala teams have been working around for years: deletion vectors that replace position-delete files, mandatory row lineage for cheap CDC, a VARIANT type shared with Spark and Delta, default column values, geospatial types, and nanosecond timestamps. Here's what each feature actually does, which pieces are production-ready on Spark 4.0 today, and what to watch out for if you flip a table to format-version = 3.

Apache Gluten graduated to an ASF Top-Level Project in March 2026. It pushes Spark's physical operators down to native C++ engines (Velox or ClickHouse) via Substrait and JNI, keeping the JVM in charge of scheduling while the heavy lifting happens off-heap. Here's the architecture, how to wire it up on a Scala workload, and where the rough edges are.

Spark Connect decouples the application from the cluster with a gRPC protocol, and as of Spark 4.0 the Scala client has near-complete DataFrame and Dataset API parity with classic mode. Here's the architecture, how to wire it up from sbt, and what still doesn't work.

Spark 4.0 added a native VARIANT type (SPARK-45827) for storing semi-structured data in a compact binary format you can query directly — no upfront schema, no from_json ceremony on every read. The published benchmarks show roughly 8x faster reads than storing the same payload as a JSON string column, and Spark 4.1 adds shredding to push that further. This article shows how the Scala API works, when to reach for VARIANT, and when you still want a strongly typed StructType.

Spark 4.0 flipped spark.sql.ansi.enabled from false to true, so invalid casts, arithmetic overflow, divide-by-zero, and bad array indices that used to silently return null now throw runtime errors. This guide catalogs each failure mode with the exception you'll see and the try_* function that fixes it without falling back to legacy mode.

DuckDB is an embedded, in-process columnar OLAP engine that runs inside your application — no cluster, no JVM serialization tax, sub-second startup. For Spark Scala developers, the entry point is the DuckDB JDBC driver on Maven Central, and the Scala 3 duck4s wrapper if you want a more idiomatic API. This is not a Spark replacement — it's a complementary tool that fits the gaps Spark was never designed to fill.

The "PySpark won, Scala is legacy" narrative is half right and half lazy. PySpark genuinely owns notebooks, ML, and the hiring funnel — but Spark itself still runs on the JVM, and Scala code still executes on the engine without a serialization boundary. Here's an honest look at where each language wins in 2026, and why Scala remains the right call for a meaningful slice of production work.

The Databricks vs open-source Spark debate is usually framed as a feature comparison, but for Scala teams shipping production pipelines it's really a question about who owns operational complexity. Here's a practical decision guide for 2026 — where the gap has narrowed, where it persists, and what should actually drive the call.

Five years after Alex Birsan's original dependency confusion research collected more than $130,000 in bug bounties from Apple, Microsoft, PayPal, and Shopify, the same class of supply-chain attack is still landing in JVM builds. Spark Scala teams are an especially easy target — and sbt's default resolver behavior, combined with repository managers that auto-proxy Maven Central, makes the attack surprisingly close to a one-line publish on Sonatype's side.

Most comparison guides for artifact repositories are written for Java teams using Maven or Gradle. If your team builds Spark Scala applications with sbt, the landscape looks different — and some popular options have sharp edges that only show up with sbt's dependency resolution.