1 comment on “Kafka And Spark Streams: The happily ever after !!”

Kafka And Spark Streams: The happily ever after !!


Hi everyone, Today we are going to understand a bit about using the spark streaming to transform and transport data between Kafka topics. The demand for stream processing is increasing every day. The reason is that often, processing big volumes…

0 comments on “They said Spark Streaming simply means Discretized Stream”

They said Spark Streaming simply means Discretized Stream


I am working in a company (Knoldus Software LLP) where Apache Spark is literally running into people's blood means there are certain people who are really good at it. If you ever visit our blogging page and search for stuff…

2 comments on “Developers Needs SDKMAN Not Super-Man”

Developers Needs SDKMAN Not Super-Man


Every developer has pain for setup development environment to his/her machine with lots of the setups. Sometimes, the pain goes beyond while we need to test same application on multiple versions of SDKs or virtual machines. If you are a…

3 comments on “Apache Hadoop vs Apache Spark”

Apache Hadoop vs Apache Spark


The term Big Data has created a lot of hype already in the business world. Hadoop and Spark are both Big Data frameworks – they provide some of the most popular tools used to carry out common Big Data-related tasks.…

5 comments on “What’s new in Apache Spark 2.2”

What’s new in Apache Spark 2.2


Apache recently released a newer version of Spark i.e Apache Spark2.2. The new version comes with new improvements as well as the addition of new functionalities. The major addition to this release is Structured Streaming. It has been marked as production…

2 comments on “Having Issue How To Order Streamed Dataframe ?”

Having Issue How To Order Streamed Dataframe ?


A few days ago, i have to perform aggregation on streaming dataframe. And the moment, i apply groupBy for aggregation, data gets shuffled. Now the situation arises how to maintain order? Yes, i can use orderBy with streaming dataframe using…

6 comments on “Difference between RDD , DF and DS in Spark”

Difference between RDD , DF and DS in Spark


In this blog I try to cover the difference between RDD, DF and DS. much of you have a little bit confused about RDD, DF and DS. so don't worry after this blog everything will be clear. With Spark2.0 release,…

1 comment on “RealTimeProcessing of Data using kafka and Spark”

RealTimeProcessing of Data using kafka and Spark


Before Starting it you should know about kafka, spark and what is Real time processing of Data.so let's do some brief introduction about it. Real Time Processing - Processing the Data that appears to take place instead of storing the data and then…

4 comments on “Integrating Kafka With Spark Structure Streaming”

Integrating Kafka With Spark Structure Streaming


Kafka is a messaging broker system which facilitates the passing of messages between producer and consumer whereas Spark Structure streaming consumes static and streaming data from various sources like kafka, flume, twitter or any other socket which can be processed…

1 comment on “Exploring Spark Structured Streaming”

Exploring Spark Structured Streaming


Hello Spark Enthusiasts, Streaming apps are growing more complex. And it is getting difficult to do with current distributed streaming engines. Why streaming is hard ? Streaming computations don't run in isolation. Data arriving out of time order is a…