Every JVM team eventually needs a place to publish internal JARs — shared UDF libraries, custom connectors, standardized schemas. In 2026 the options span three distinct models: self-hosted repository managers, SaaS registries, and build-from-git services. Here's the map, so a Spark Scala team can figure out which tier fits before getting lost in feature checklists.

SQL Scripting reached GA and is enabled by default in Spark 4.1 (SPARK-54499), bringing variables, IF/CASE, WHILE/REPEAT/FOR loops, and DECLARE ... HANDLER error handling to pure SQL. Here's what it does, when it beats reaching for Scala, and how to run a script from a Spark Scala application.

There are three credible paths to native C++ execution for Spark in 2026: Databricks Photon (proprietary, mature, locked in), Apache Gluten with Velox (open source, newly graduated to ASF Top-Level Project), and Microsoft Fabric's Native Execution Engine (Gluten-based, managed). The choice maps cleanly onto your deployment model — here's how Scala teams should think about it.

Kubernetes finally has an official Apache Spark operator, a credible batch scheduler in YuniKorn, and most of the cloud vendor weight behind it. YARN still works, still runs on most of the on-prem Hadoop clusters, and isn't going anywhere this decade. Here's an honest framework for choosing between them when your Scala team has the option.

UNKNOWN

DuckLake is a new open lakehouse format from DuckDB Labs that puts table metadata in a standard SQL database — Postgres, MySQL, SQLite, or DuckDB — instead of writing thousands of small Avro and JSON files like Iceberg and Delta Lake do. v1.0 shipped in April 2026 under the MIT license. For Spark Scala teams the immediate story is not "rip out Iceberg," but the metadata-in-SQL idea is interesting enough to be worth understanding now.

MLflow 3.0 (released June 2025) rebuilt the platform around LoggedModel as a first-class entity, added GenAI tracing on top of OpenTelemetry, and reorganized how artifacts are stored. Most of the headline features land in the Python and TypeScript SDKs, but the JVM tracking client is still the path Scala teams use to log Spark ML runs from production code — and the changes underneath it are worth knowing before you upgrade.

Delta Lake's Universal Format generates Iceberg metadata alongside the Delta transaction log, against the same Parquet files, so Iceberg-native engines can read your Delta tables without conversion or duplication. With Delta 4.0.1 restoring Iceberg compat for Spark 4.0, UniForm is once again a usable option for Scala teams that need cross-engine reads — provided you understand the limitations.

Apache Polaris graduated to a top-level ASF project in February 2026 and is consolidating as the default open implementation of the Iceberg REST Catalog spec. For Spark Scala teams, it's the piece that lets Spark, Trino, and Flink work against the same Iceberg tables with one source of truth — without Hive Metastore, without per-engine catalog plumbing, and without vendor lock-in.

The official Apache Spark Kubernetes Operator launched as an ASF subproject in May 2025, built from scratch instead of forking the aging Kubeflow operator. A year of rapid releases later, it's at 0.9.0 and is the path the Spark community is steering toward for running Scala jobs on Kubernetes.