1 comment on “Kafka And Spark Streams: The happily ever after !!”

Kafka And Spark Streams: The happily ever after !!


Hi everyone, Today we are going to understand a bit about using the spark streaming to transform and transport data between Kafka topics. The demand for stream processing is increasing every day. The reason is that often, processing big volumes…

3 comments on “Spark Streaming: Unit Testing DStreams”

Spark Streaming: Unit Testing DStreams


Frankly, I don't think there's any need of telling us, "The Developers", the need for proper testing or Unit testing to be correct(QAs, Don't be flattered :P). The unit test cases are the quickest way to know there's something wrong…

2 comments on “Assimilation of Spark Streaming With Kafka”

Assimilation of Spark Streaming With Kafka


As we know Spark is used at a wide range of organizations to process large datasets. It seems like spark becoming main stream. In this blog we will talk about Integration of Kafka with Spark Streaming. So, lets get started. How Kafka…

5 comments on “What’s new in Apache Spark 2.2”

What’s new in Apache Spark 2.2


Apache recently released a newer version of Spark i.e Apache Spark2.2. The new version comes with new improvements as well as the addition of new functionalities. The major addition to this release is Structured Streaming. It has been marked as production…

10 comments on “Basic Example for Spark Structured Streaming & Kafka Integration”

Basic Example for Spark Structured Streaming & Kafka Integration


The Spark Streaming integration for Kafka 0.10 is similar in design to the 0.8 Direct Stream approach. It provides simple parallelism, 1:1 correspondence between Kafka partitions and Spark partitions, and access to offsets and metadata. However, because the newer integration…

2 comments on “Spark Streaming vs Kafka Stream”

Spark Streaming vs Kafka Stream


The demand for stream processing is increasing a lot these days. The reason is that often processing big volumes of data is not enough. Data has to be processed fast, so that a firm can react to changing business conditions…

1 comment on “Getting Started with Apache Spark”

Getting Started with Apache Spark


Introduction Apache Spark is an open source big data processing framework built around speed, ease of use, and sophisticated analytics. It was originally developed in 2009 in UC Berkeley’s AMPLab, and open sourced in 2010 as an Apache project. Spark…

1 comment on “Streaming with Apache Spark Custom Receiver”

Streaming with Apache Spark Custom Receiver


Hello inqisitor. In previous blog we have seen about the predefined Stream receiver of Spark. In this blog we are going to discuss about Custom receiver of spark so that we can source the data from any . So if…

2 comments on “Streaming with Apache Spark 2.0”

Streaming with Apache Spark 2.0


Hello geeks we were discussed about Apache Spark 2.0 with hive in earlier blog. Now i am going to describe how can we use spark to stream the data   . At first we need to understand this new Spark Streaming architecture…

0 comments on “MeetUp on “An Overview of Spark DataFrames with Scala””

MeetUp on “An Overview of Spark DataFrames with Scala”


Knoldus is organizing an one hour session on 18th Nov 2015 at 6:00 PM. Topic would be An Overview of Spark DataFrames with Scala. All of you are invited to join this session. Address:- 30/29, First Floor, Above UCO Bank, Near Rajendra…