Upgrade your Spark REST Server with Akka HTTP & Spark 2.0

About an year ago, in one of our blog – Spark with Spray Starter Kit we explained about creating REST Services with Spark and Spray. But, from past one year there has not been much development on Spray which tells us that soon Spray will be out-of-phase.

So, we decided to Upgrade our Spark REST Server with Akka HTTP and Spark 2.0. Akka HTTP is a suite of libraries that are used to implement a full server- and client-side HTTP stack on top of akka-actor and akka-stream. It is not a web-framework but rather a more general toolkit for providing and consuming HTTP-based services.

Now, lets take a look at some code. First lets see the dependencies which we are going to need for building our application:

"org.apache.spark" %% "spark-core" % "2.0.0-preview",
"com.typesafe.akka" %% "akka-actor" % "2.4.6",
"com.typesafe.akka" %% "akka-stream" % "2.4.6",
"com.typesafe.akka" %% "akka-http-experimental" % "2.4.6",
"com.typesafe.akka" %% "akka-http-spray-json-experimental" % "2.4.6",
"com.typesafe.akka" %% "akka-http-testkit" % "2.4.6"

Since, Spark 2.0 is still in preview phase, so we will be using 2.0.0-preview version of Spark.

Next comes a standard template of an Akka HTTP Server, which looks like following in code:

object SparkServer extends App with SparkServices {
implicit val system = ActorSystem("spark-akka-http")
implicit val materializer = ActorMaterializer()
implicit val executionContext = system.dispatcher

val bindingFuture = Http().bindAndHandle(route, "localhost", 8001)
}

The above code will create and bind a HTTP Server with localhost at port number 8001.

At last we define SparkServices, which contains Spark Service routes:

trait SparkServices {
val sparkConf = new SparkConf().setAppName("spark-akka-http").setMaster("local")
val sc = new SparkContext(sparkConf)

val route =
path("version") {
get {
complete(HttpEntity(ContentTypes.`text/html(UTF-8)`, s"
<h1>Spark version in this template is: ${sc.version}</h1>
"))
}
}
}

Now when we are done with coding, lets start the server and test it. So, type in command sbt run on terminal and hit http://localhost:8001/version on any browser. We can see following result on browser:

For complete code of this template click here.