Play with Spark: Building Spark MLLib in a Play Spark Application

In our last post of Play with Spark! series, we saw how to integrate Spark SQL in a Play Scala application. Now in this blog we will see how to add Spark MLLib feature in a Play Scala application.

Spark MLLib is a new component under active development. It was first released with Spark 0.8.0. It contains some common machine learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as some optimization primitives. For detailed list of available algorithms click here.

To add Spark MLLib feature in a Play Scala application follow these steps:

1). Add following dependencies in build.sbt file

The dependency – “org.apache.spark”  %% “spark-mllib” % “1.0.1” is specific to Spark MLLib.

As you can see that we have upgraded to Spark 1.0.1 (latest release of Apache Spark).

2). Create a file app/utils/SparkMLLibUtility.scala & add following code to it

In above code we have used Naive Bayes algorithm as an example.

3). In above code you can notice that we have parsed data into Vectors object of Spark.

Reason for using Vectors object of Spark instead of Vector class of Scala is that, Vectors object of Spark contains both Dense & Sparse methods for parsing both dense & sparse type of data. This allows us to analyze data according to its properties.

4). Next we observe that we have split data in 2 parts – 60% for training & 40% for testing.

5). Then we trained our model using Naive Bayes algorithm & training data.

6). At last we used our model to predict the labels/class of test data.

Then to find how good our model is, we calculated the Accuracy of the predicted labels.

So, we see that how easy it is to use any algorithm available in Spark MLLib to perform predictive analytics on data. For more information on Spark MLLib click here.

To download a Demo Application click here.

Written by 

Himanshu Gupta is a lead consultant having more than 4 years of experience. He is always keen to learn new technologies. He not only likes programming languages but Data Analytics too. He has sound knowledge of "Machine Learning" and "Pattern Recognition".He believes that best result comes when everyone works as a team. He likes listening to Coding ,music, watch movies, and read science fiction books in his free time.

6 thoughts on “Play with Spark: Building Spark MLLib in a Play Spark Application

    1. Leon Radley (@LeonRadley) Play & Spark uses different version of Akka. That is why, we get a lot of strange errors. So, we have to use a common version of Akka.
      Using Akka 2.2.3 can solve this problem.

Leave a Reply

%d bloggers like this: