Play with Spark: Building Apache Spark with Play Framework


Nowadays, Play framework is being used a lot, for building Scala Applications. It is easy to use & it is Typesafe. So, in this post, we will see how to build a Spark Application in Play 2.2.x.

Although Play also uses sbt to build an application but building a Spark Application in Play is totally different.

Before start building this application follow the instructions of building Spark given here.

After building Spark, we are ready to build the Application. To build the application follow these steps:

1). Run play new PlaySparkApp.

This will create a new Play application.

2). Add following dependencies in build.sbt file

libraryDependencies ++= Seq(
 "org.apache.spark" %% "spark-core" % "0.9.1",
 "com.typesafe.akka" %% "akka-actor" % "2.2.3",
 "com.typesafe.akka" %% "akka-slf4j" % "2.2.3"
)

Although, Akka is inbuilt in Spark & Play both but due to mismatch of Akka versions in Play & Spark, application will not build properly. So, a common version of Akka is added in the application.

3). Add these lines in Application.scala file

package controllers

import play.api._
import play.api.mvc._
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

object Application extends Controller {

 def index = Action {
 val logFile = "public/data/README.md" // Should be some file on your system
 val sc = new SparkContext("local", "Application", "/path/to/spark-0.9.0-incubating", List("target/scala-2.10/playsparkapp_2.10-1.0-SNAPSHOT.jar"))
 val logData = sc.textFile(logFile, 2).cache()
 val numSparks = logData.filter(line => line.contains("Spark")).count()
 Ok(views.html.index("Lines with Spark: " + numSparks))
 }

}

This is just a demo application in Spark. You can build any application of Spark here.

4). Then go to PlaySparkApp directory.

5). Run  play clean compile dist

6). Run play run

The application is ready to be displayed on localhost:9000.

To download a Demo Application click here.

Note: Whenever a change is made in the application, then run play dist command otherwise changes in the Spark files/functions will be not be reflected.

Advertisements
This entry was posted in Agile, Akka, Play Framework, Scala, Spark, Web and tagged , , . Bookmark the permalink.

5 Responses to Play with Spark: Building Apache Spark with Play Framework

  1. Pingback: Play with Spark: Building Apache Spark with Play Framework – (Part – 2) | Knoldus

  2. Bin Zhang says:

    playsparkapp_2.10-1.0-SNAPSHOT.jar, this jar how to get? i want to take a project with spark and play?i do not know if I want submit my porject,i just like in spark make a jar and submit ,or use play run project-name?

  3. superczynski says:

    You should move `val sc = new SparkContext(“local”, “Application”, “/path/to/spark-0.9.0-incubating”, List(“target/scala-2.10/playsparkapp_2.10-1.0-SNAPSHOT.jar”))` into new object, because each time you call controller new SparkContext being created.
    “`
    object SparkConnector {
    val sc = new SparkContext(“local”, “Application”, “/path/to/spark-0.9.0-incubating”, List(“target/scala-2.10/playsparkapp_2.10-1.0-SNAPSHOT.jar”))
    }
    “`
    and then

    `val logData = SparkConnector.sc.textFile(logFile, 2).cache()`

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s