Play with Spark: Building Apache Spark with Play Framework

Table of contents

Reading Time: < 1 minute

Nowadays, Play framework is being used a lot, for building Scala Applications. It is easy to use & it is Typesafe. So, in this post, we will see how to build a Spark Application in Play 2.2.x.

Although Play also uses sbt to build an application but building a Spark Application in Play is totally different.

Before start building this application follow the instructions of building Spark given here.

After building Spark, we are ready to build the Application. To build the application follow these steps:

1). Run play new PlaySparkApp.

This will create a new Play application.

2). Add following dependencies in build.sbt file

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "0.9.1",
"com.typesafe.akka" %% "akka-actor" % "2.2.3",
"com.typesafe.akka" %% "akka-slf4j" % "2.2.3"
)

Although, Akka is inbuilt in Spark & Play both but due to mismatch of Akka versions in Play & Spark, application will not build properly. So, a common version of Akka is added in the application.

3). Add these lines in Application.scala file

package controllers

import play.api._
import play.api.mvc._
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

object Application extends Controller {

def index = Action {
val logFile = "public/data/README.md" // Should be some file on your system
val sc = new SparkContext("local", "Application", "/path/to/spark-0.9.0-incubating", List("target/scala-2.10/playsparkapp_2.10-1.0-SNAPSHOT.jar"))
val logData = sc.textFile(logFile, 2).cache()
val numSparks = logData.filter(line => line.contains("Spark")).count()
Ok(views.html.index("Lines with Spark: " + numSparks))
}

}

This is just a demo application in Spark. You can build any application of Spark here.

4). Then go to PlaySparkApp directory.

5). Run play clean compile dist

6). Run play run

The application is ready to be displayed on localhost:9000.

To download a Demo Application click here.

Note: Whenever a change is made in the application, then run play dist command otherwise changes in the Spark files/functions will be not be reflected.

5 thoughts on “Play with Spark: Building Apache Spark with Play Framework2 min read”

Pingback: Play with Spark: Building Apache Spark with Play Framework – (Part – 2) | Knoldus

playsparkapp_2.10-1.0-SNAPSHOT.jar, this jar how to get? i want to take a project with spark and play?i do not know if I want submit my porject,i just like in spark make a jar and submit ,or use play run project-name?

Himanshu Gupta says:

August 21, 2014 at 11:07 AM

Hi Bin Zhang, if you want to get playsparkapp_2.10-1.0-SNAPSHOT.jar then use ‘play dist’ command and you will get this jar in the target folder.

But if you want to run your project just by typing ‘play run project_name’ & not provide the project’s jar file then you should use a custom configuration of Spark. To know how to build a custom configuration of Spark you can visit this link – http://blog.knoldus.com/2014/07/14/play-with-spark-building-apache-spark-with-play-framework-part-3/.

You should move `val sc = new SparkContext(“local”, “Application”, “/path/to/spark-0.9.0-incubating”, List(“target/scala-2.10/playsparkapp_2.10-1.0-SNAPSHOT.jar”))` into new object, because each time you call controller new SparkContext being created.
“`
object SparkConnector {
val sc = new SparkContext(“local”, “Application”, “/path/to/spark-0.9.0-incubating”, List(“target/scala-2.10/playsparkapp_2.10-1.0-SNAPSHOT.jar”))
}
“`
and then

`val logData = SparkConnector.sc.textFile(logFile, 2).cache()`