Tag Archives: apache spark

Partition-Aware Data Loading in Spark SQL


Data loading, in Spark SQL, means loading data in memory/cache of Spark worker nodes. For which we use to write following code: val connectionProperties = new Properties() connectionProperties.put(“user”, “username”) connectionProperties.put(“password”, “password”) val jdbcDF = spark.read .jdbc(“jdbc:postgresql:dbserver”, “schema.table”, connectionProperties) In here we are … Continue reading

Posted in Scala, Spark | Tagged , , , | 2 Comments

Migration From Spark 1.x to Spark 2.x


Hello Folks, As we know that we have latest release of Spark 2.0, with to much enhancement and new features. If you are using Spark 1.x and now you want to move your application with Spark 2.0 that time you … Continue reading

Posted in Scala | Tagged | 2 Comments

Spark – LDA : A Complete example of clustering algorithm for topic discovery.


In this blog we will be demonstrating the functionality of applying the full ML pipeline over a set of documents which in this case we are using 10 books from the internet. So lets start with first thing first.. What … Continue reading

Posted in apache spark, Scala, Spark | Tagged , , , , , , , , , , , , , , , , , , , , , , | 5 Comments

Spark – IoT : Combining Big Data Analysis with IoT


Welcome back , folks ! Time for some new gig ! I think that last series i.e. Scala – IOT was pretty amazing , which got an overwhelming response from you all which resulted in pumping up the idea of … Continue reading

Posted in apache spark, IOT, Scala, Spark | Tagged , , , , , , , , , , , | 1 Comment

Streaming with Apache Spark Custom Receiver


Hello inqisitor. In previous blog we have seen about the predefined Stream receiver of Spark. In this blog we are going to discuss about Custom receiver of spark so that we can source the data from any . So if … Continue reading

Posted in apache spark, big data, Scala | Tagged , | 1 Comment

Streaming with Apache Spark 2.0


Hello geeks we were discussed about Apache Spark 2.0 with hive in earlier blog. Now i am going to describe how can we use spark to stream the data   . At first we need to understand this new Spark Streaming architecture … Continue reading

Posted in apache spark, big data, Scala | Tagged , | 2 Comments

KnolX: Introduction to Apache Spark 2.0


Knoldus organized a KnolX session on Friday, 23 September 2016. In that one hour session we got an introduction of Apache Spark 2.0 and its API(s). Spark 2.0 is a major release of Apache Spark. This release has brought many … Continue reading

Posted in Scala, Spark | Tagged , , , | 1 Comment

Introduction to Apache Hadoop: The Need


In this Blog we will read about the Hadoop fundamentals. After reading this blog we will be able to understand why we need Apache Hadoop, So lets starts with the problem. Whats the Problem :- The problem is simple: the … Continue reading

Posted in Scala | Tagged , , | Leave a comment

Scala – IOT : First basic IOT application using Scala on RaspberryPi


Let’s start our journey for making the first IoT application to make world a better place 😉 (I would never miss a chance to mock Hooli ! 😉 ) In this blog finally the two technologies SCALA and IOT  will … Continue reading

Posted in Scala | Tagged , , , , , , , , , , , , , , , , , , , , , , , , | 2 Comments

Scala-IOT : Introduction to Internet Of Things.


Recently this word IOT is gaining lot of popularity. And we see a lot of news on it like the world is moving towards IOT , and its the next big thing and smart cities are no longer a fiction  … Continue reading

Posted in IOT, Scala | Tagged , , , , , , , , , , , | 11 Comments