Spark 4.1 lands with 1,800+ resolved tickets from 230+ contributors and two headline features: Structured Streaming Real-Time Mode for sub-second latency and Spark Declarative Pipelines for managed ETL pipelines. Here's what matters if you're writing or maintaining Spark Scala applications.

Spark 4.0 brings real breaking changes that will likely affect your existing Scala pipelines — ANSI mode on by default, Scala 2.12 dropped, JDK 17 required, and infrastructure changes to shuffle and event logging. This guide walks through each one with before/after context and the config knob to fall back if you need time to migrate.

Spark 4.0 is the biggest release in years — over 5,100 resolved tickets from 390+ contributors. Here's what matters most if you're writing or maintaining Spark Scala applications.

UNKNOWN

In the evolving landscape of big data engineering and analytics, staying up-to-date with the latest tools and technologies is a chore. Also, with the growing adoption of pyspark, Spark Scala seems to be taking a back seat in the ecosystem. That's where SparkingScala comes to the rescue! Created by experienced data engineers who have been developing and maintaining spark scala applications for years. We aim to create a simple resource for Spark Scala.