Play with Spark: Building Apache Spark with Play Framework

Nowadays, Play framework is being used a lot, for building Scala Applications. It is easy to use & it is Typesafe. So, in this post, we will see how to build a Spark Application in Play 2.2.x.

Although Play also uses sbt to build an application but building a Spark Application in Play is totally different.

Before start building this application follow the instructions of building Spark given here.

After building Spark, we are ready to build the Application. To build the application follow these steps:

1). Run play new PlaySparkApp.

This will create a new Play application.

2). Add following dependencies in build.sbt file

Although, Akka is inbuilt in Spark & Play both but due to mismatch of Akka versions in Play & Spark, application will not build properly. So, a common version of Akka is added in the application.

3). Add these lines in Application.scala file

This is just a demo application in Spark. You can build any application of Spark here.

4). Then go to PlaySparkApp directory.

5). Run  play clean compile dist

6). Run play run

The application is ready to be displayed on localhost:9000.

To download a Demo Application click here.

Note: Whenever a change is made in the application, then run play dist command otherwise changes in the Spark files/functions will be not be reflected.

Written by 

Himanshu Gupta is a lead consultant having more than 4 years of experience. He is always keen to learn new technologies. He not only likes programming languages but Data Analytics too. He has sound knowledge of "Machine Learning" and "Pattern Recognition".He believes that best result comes when everyone works as a team. He likes listening to Coding ,music, watch movies, and read science fiction books in his free time.

5 thoughts on “Play with Spark: Building Apache Spark with Play Framework

  1. playsparkapp_2.10-1.0-SNAPSHOT.jar, this jar how to get? i want to take a project with spark and play?i do not know if I want submit my porject,i just like in spark make a jar and submit ,or use play run project-name?

    1. Hi Bin Zhang, if you want to get playsparkapp_2.10-1.0-SNAPSHOT.jar then use ‘play dist’ command and you will get this jar in the target folder.

      But if you want to run your project just by typing ‘play run project_name’ & not provide the project’s jar file then you should use a custom configuration of Spark. To know how to build a custom configuration of Spark you can visit this link – http://blog.knoldus.com/2014/07/14/play-with-spark-building-apache-spark-with-play-framework-part-3/.

  2. You should move `val sc = new SparkContext(“local”, “Application”, “/path/to/spark-0.9.0-incubating”, List(“target/scala-2.10/playsparkapp_2.10-1.0-SNAPSHOT.jar”))` into new object, because each time you call controller new SparkContext being created.
    “`
    object SparkConnector {
    val sc = new SparkContext(“local”, “Application”, “/path/to/spark-0.9.0-incubating”, List(“target/scala-2.10/playsparkapp_2.10-1.0-SNAPSHOT.jar”))
    }
    “`
    and then

    `val logData = SparkConnector.sc.textFile(logFile, 2).cache()`

Leave a Reply

%d bloggers like this: