Play with Spark: Building Apache Spark with Play Framework – (Part – 2)

Last week, we saw how to build a Simple Spark Application in Play using Scala. Now in this blog we will see how to add Spark’s Twitter Streaming feature in a Play Scala application.

Spark Streaming is a powerful tool of Spark. It runs on top of Spark. It gives the ability to process and analyze real-time streaming data (in batches) along with fault-tolerant characteristics of Spark.

To add Spark’s Twitter Streaming feature in a Play Scala application follow these steps:

1). Add following dependencies in build.sbt file

As you can see we have upgraded our application to Spark 1.0.0. The dependency – “org.apache.spark”  %% “spark-streaming-twitter” % “1.0.0” is specific to Twitter Streaming. If you want to use Flume, Kafka, etc, then import respective dependency given here.

2). Create a file app/utils/TwitterPopularTags.scala & add following code to it

This function will get the most Popular Tags from Twitter of matching “filters” from Streaming Tweets using Spark Streaming. Also, Spark Streaming needs its own Port to stream in data, so, provide a separate port to it other than 9000 or the port on which Play is running.

3). Add Twitter App credentials in conf/application.conf like this

4). At last add following code in app/controllers/Application.conf

5). Now the application is ready to be run on localhost:9000.

To download a Demo Application click here.

Written by 

Himanshu Gupta is a lead consultant having more than 4 years of experience. He is always keen to learn new technologies. He not only likes programming languages but Data Analytics too. He has sound knowledge of "Machine Learning" and "Pattern Recognition".He believes that best result comes when everyone works as a team. He likes listening to Coding ,music, watch movies, and read science fiction books in his free time.

2 thoughts on “Play with Spark: Building Apache Spark with Play Framework – (Part – 2)

  1. Hi, thanks for the post. But if you put the sparkstreaming job inside of Future, it is going to skip over the spark job and go directly to views.html.index page, right?

    I wonder if there is a way to start the spark streaming job in the background and still serve the views.html.index page under play framework.

    Thank you.

Leave a Reply

%d bloggers like this: