In order to run our application with a Spark distribution, we need to package our compiled classes and their dependencies in a JAR file. This is what we call an assembly JAR or fat JAR: its size can be quite large if you have many dependencies. To do that, we have to modify our SBT build files.

First, we need to enable the assembly plugin. Add a new file, assembly.sbt, in the bitcoin-analyser/project folder and type the following:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.7"

We also need to exclude all the Spark dependencies from our assembly JAR. There is no point in having them as they are already present in the Spark distribution. Excluding them will save space and build time. For this, we had already scoped these dependencies to % Provided in build.sbt. This will make sure that the dependencies are present for compiling the project and running the tests, but are excluded when building the assembly JAR.

Then we have to add a few configuration options at the end of build.sbt:

assemblyOption in assembly := (assemblyOption in   
  assembly).value.copy(includeScala = false)
test in assembly := {}
mainClass in assembly := Some("coinyser.BatchProducerAppSpark")

Here is a short explanation talking about what line performs what action:

• The first line excludes all Scala runtime JARs.
• The second line tells SBT to skip the tests when running the assembly task
• The last line declares what is our main class. This declaration will end up in the MANIFEST.MF file, and will be used by Spark to bootstrap our program.

Our build files are ready; we can run the assembly task. Open the SBT shell in IntelliJ (Ctrl + Shift + S), and type assembly after the sbt> prompt. This should compile the project and package the assembly JAR. The output of your SBT shell should look like this:

[IJ]sbt:bitcoin-analyser> assembly
[info] Strategy 'discard' was applied to 3 files (Run the task at debug level to see details)[info] Packaging /tmp/sbt/bitcoin-analyser/scala-2.11/bitcoin-analyser-assembly-0.1.jar 
...
[info] Done packaging.

The assembly JAR is ready; we can submit it to Spark.