Spark – IoT : Combining Big Data Analysis with IoT

Welcome back , folks !

Time for some new gig ! I think that last series i.e. Scala – IOT was pretty amazing , which got an overwhelming response from you all which resulted in pumping up the idea of this new web-series Spark-IOT.

So let’s get started,

What was the motivation ?

I have been active in the IoT community here, and I found a gap in the community. As far as I have seen here in Delhi/NCR, India the IoT community is basically divided into two sets:

1. The ones who understand hardware, sensor, protocols, sensor data.
2. The ones who understand software, what to analyse , how to analyse and so on.

Maybe you can relate this into your own community 😉 .

I am the guy who understands software very well, and we have been working on the analysis part for quite a while. Hence I thought why not combine the two fields and put the big data analysis on the streaming data generated from an IoT device.

Follow – Up !!

If you do not know anything about IoT and protocols , I would suggest you check out the Scala – IoT series first , so that you can get a basic understanding of the protocols , MQTT and all. Here is the link to all the Scala – IoT blogs:

1. Scala-IOT : Introduction to Internet Of Things.

2. Scala-IOT : What is Mqtt ? How it is lightweight ?

3. Scala-IOT: Getting started with RaspberryPi without Monitor or Screen.

4. Scala – IOT : First basic IOT application using Scala on RaspberryPi

If you know a little about Apache Spark, I would like you to go through these , so that you have the basic understanding of what it is and then maybe you can move onto the official documentation. Here is the link.

What this web-series will be about ?

This web series will be about how to analyse the data generated from IoT device . In this case, it would be Raspberry Pi. So I have moulded the series like this :

1. Spark – IoT: Setting up Apache Spark Cluster on RaspberryPi.
   

   In this, we will be talking about how to setup a basic Spark Cluster on Raspberry Pi. In which one Raspberry Pi would be master and the other one would be a slave. And we would be running the Spark Shell.

2. Spark – IoT: Developing your first basic Streaming Application for analysis.

   In this , we will be developing a basic Spark Streaming Application for the data analysis , the data producer will be raspberryPi itself as we have done it in the first Scala -IoT Application.

3. Spark – IoT: Deploying your application on RaspberryPi-Spark Cluster.

   In this, we will be deploying this application on the RaspberryPi-Spark Cluster for the data processing.

This is the overall idea,  but things can be changed so if you have any suggestions please drop it in the comments and maybe we can work on that too along with you 🙂

If you are interested in this web series , please let us know ! You can subscribe to our newsletter here !

And if you have any questions please feel free to contact me here or on Twitter : @shiv4nsh

Till then !

Happy hAKKAing ! 🙂