Tag Archives: apache spark

Spark Structured Streaming: A Simple Definition

Posted on June 18, 2017 by

“Structured Streaming”, nowadays we are hearing this term in Apache Spark ecosystem quite a lot, as it is being preached as next big thing in scalable big data world. Although, we all know that Structured Streaming means a stream having … Continue reading →

Posted in Scala, Spark, Streaming | Tagged , , , , | 2 Comments

Getting Started with Apache Spark

Posted on May 9, 2017 by

Introduction Apache Spark is an open source big data processing framework built around speed, ease of use, and sophisticated analytics. It was originally developed in 2009 in UC Berkeley’s AMPLab, and open sourced in 2010 as an Apache project. Spark … Continue reading →

Posted in apache spark, Scala, Spark | Tagged , , , , , , | 1 Comment

The Dominant APIs of Spark: Datasets, DataFrames and RDDs

Posted on April 16, 2017 by

While working with Spark often we come across the three APIs: DataFrames, Datasets and RDDs.  In this blog I will discuss the three in terms of use case, performance and optimization.  It is essential to keep in mind that there … Continue reading →

Posted in Spark | Tagged , , , , , , | 1 Comment

Partition-Aware Data Loading in Spark SQL

Posted on February 9, 2017 by

Data loading, in Spark SQL, means loading data in memory/cache of Spark worker nodes. For which we use to write following code: val connectionProperties = new Properties() connectionProperties.put(“user”, “username”) connectionProperties.put(“password”, “password”) val jdbcDF = spark.read .jdbc(“jdbc:postgresql:dbserver”, “schema.table”, connectionProperties) In here we are … Continue reading →

Posted in Scala, Spark | Tagged , , , | 7 Comments

Migration From Spark 1.x to Spark 2.x

Posted on December 30, 2016 by

Hello Folks, As we know that we have latest release of Spark 2.0, with to much enhancement and new features. If you are using Spark 1.x and now you want to move your application with Spark 2.0 that time you … Continue reading →

Posted in Scala | Tagged | 2 Comments

Spark – LDA : A Complete example of clustering algorithm for topic discovery.

Posted on October 8, 2016 by

In this blog we will be demonstrating the functionality of applying the full ML pipeline over a set of documents which in this case we are using 10 books from the internet. So lets start with first thing first.. What … Continue reading →

Posted in apache spark, Scala, Spark | Tagged , , , , , , , , , , , , , , , , , , , , , , | 7 Comments

Spark – IoT : Combining Big Data Analysis with IoT

Posted on October 1, 2016 by

Welcome back , folks ! Time for some new gig ! I think that last series i.e. Scala – IOT was pretty amazing , which got an overwhelming response from you all which resulted in pumping up the idea of … Continue reading →

Posted in apache spark, IOT, Scala, Spark | Tagged , , , , , , , , , , , | 2 Comments

Streaming with Apache Spark Custom Receiver

Posted on September 30, 2016 by

Hello inqisitor. In previous blog we have seen about the predefined Stream receiver of Spark. In this blog we are going to discuss about Custom receiver of spark so that we can source the data from any . So if … Continue reading →

Posted in apache spark, big data, Scala | Tagged , | 1 Comment

Streaming with Apache Spark 2.0

Posted on September 30, 2016 by

Hello geeks we were discussed about Apache Spark 2.0 with hive in earlier blog. Now i am going to describe how can we use spark to stream the data   . At first we need to understand this new Spark Streaming architecture … Continue reading →

Posted in apache spark, big data, Scala | Tagged , | 2 Comments

KnolX: Introduction to Apache Spark 2.0

Posted on September 28, 2016 by

Knoldus organized a KnolX session on Friday, 23 September 2016. In that one hour session we got an introduction of Apache Spark 2.0 and its API(s). Spark 2.0 is a major release of Apache Spark. This release has brought many … Continue reading →

Posted in Scala, Spark | Tagged , , , | 1 Comment