Spark-Scala-Maven Intellij

Setup your Spark(3.0)-Scala-Maven project with spark tests in IntelliJ IDEA

Objective

Setup your local environment with Maven, Scala, Spark 3.0. The setup takes hours and I believe this tutorial will get through the setup easily.

Prerequisites

java 11

Scala 2.12

apache-spark 3.0

Alternatively, Click here to download spark-poc and get started.

1. Install Apache-spark 3.0 or update

brew upgrade && brew update // updates your spark
  or
brew install apache-spark

brew upgrades apache-spark to 3.0

Alternatively, download here and untar it

2. Create a new maven project

Open Idea-Intellij, file -> new -> project

Choose Maven, Project SDK 11

Name the project -> Next and finish

name the project

3.0 Install the scala plugin

4. Update your pom.xml

Open pom.xml.

Copy the gist

https://gist.github.com/mahen-github/02ec3c756dd11656dd2fb2f90fce82c

Make sure groupId and artifactId is YOUR PROJECT specific

4.1 Verify build is successful

5. Add Scala framework support

Right-click on your project and “Add Scala framework support”

Add scala framework support

Choose Scala-Sdk2.12

5.1 Verify Project SDK and Global Libraries

Verify the Java-SDK version is 11

Verify version of Scala-SDK is2.12

6. Create a package and your first Scala object

Next Chapter

Index

1. Setup Spark-Scala-Maven In Intellij IDEA [You are here]
2. Chapter 1 : Get Started
3. Chapter 1.1 : Configure Spark Web UI in Local

Comment the issues/feedback below. I went through lot of issues to set it up in way that i could run the spark-scala tests in local in debug mode.