A Little Background

The big data space is under rapid disruption. When Apache Spark came onto the scene it greatly simplified old Hadoop systems and big data developers and engineers rejoiced!

But oh how the time has gone. Spark Scala was a huge hit with Data Scientists, Machine Learning Engineers and anyone who now had much better tools at their fingertips for working with big data. This brought about the need for pyspark as python with pandas is the most popular tech stack in the space. Now pyspark has taken over the Spark ecosystem and we continue to see data etl (extract transform and load) pipelines make the switch.

As early adopters of Spark Scala we still maintain many older applications and data pipelines that use scala as the main language.

As this shift takes places we wanted to create a place for us older 'spark scala devs' to quickly find easy to use examples, tutorials and latest big data industry news to help them maintain their older data pipelines. It doesn't always make sense to re-write these!

We hope you find the information helpful as you maintain these older applications.