Introducing SparkingScala: A Spark Scala Resource

In the evolving landscape of big data engineering and analytics, staying up-to-date with the latest tools and technologies is a chore. Also, with the growing adoption of pyspark, Spark Scala seems to be taking a back seat in the ecosystem. That's where SparkingScala comes to the rescue! Created by experienced data engineers who have been developing and maintaining spark scala applications for years. We aim to create a simple resource for Spark Scala.

There is a Gap

One of the main pain points we keep facings is the lack of simple, concise, practical examples for Spark Scala functions. There is a lack of examples out there and the ones that exist are either poor, incorrect or the websites bombard you with so many ads that you can't even see the examples! We are trying to correct that. Offering straightforward explanations and code snippets that demonstrate the correct usage of these spark scala functions and higher level tutorials on how to approach certain problems you will face as a data engineer.

We Come From the Data Engineering Trenches

At SparkingScala, we want to share knowledge gained from painful real-world experiences, we've learned the hard way. We've already run up the cluster costs with our mistakes! You're welcome! We hope to provide you with some tips and tricks that help reduce your cluster compute costs before they happen.

Spark Scala Applications Are Still Maintained

While pyspark has become increasingly popular, many organizations still have spark scala programs as their data ETL pipelines. SparkingScala wants to help data engineers who are maintaing these pipelines. We understand that these pipelines are the backbone of their company's data infrastructure, provide massive value and we want to help you maintain these systems – doing a major re-write is usually not the best answer!

As the data engineering landscape evolves, we see the shift towards PySpark. However, we firmly believe that Spark Scala has its place. Rebuilding a complex etl pipeline just to swap out a language is rarely the best business choice. There will be a need for spark scala experts who can maintain and optimize existing systems. If you are struggling in maintaining such a system and need help we do offer spark scala consulting.

Let's Get Started

So thank you for stopping by. We hope the tutorials and examples help you deliver value faster.