Category Archives: Apache Kafka

Self-Learning Kafka Streams with Scala – #1


A few days ago, I came across a situation where I wanted to do a stateful operation on the streaming data. So, I started finding possible solutions for it. I came across many solutions which were using different technologies like … Continue reading

Posted in Apache Kafka, Scala, Streaming | Tagged | Leave a comment

Kafka Streams : More Than Just a Dumb Storage


Whenever we hear the word Kafka, all we think about it as a messaging system with a publisher-subscriber model that we use for our streaming applications as a source and a sink. So we can say that Kafka is just a … Continue reading

Posted in Apache Kafka, Scala | Tagged , , , | Leave a comment

Unit Testing Of Kafka


Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another. Generally, data is published to topic via Producer API … Continue reading

Posted in Apache Kafka, Scala, scalatest, Streaming, testing | Tagged , , , , | 1 Comment

Setting It Up: KAFKA Multi-Broker System


In this blog, I am going to cover up the leftovers of my last blog: “A Beginners Approach To KAFKA” in which I tried to explain the details of Kafka, like its terminologies, advantages and demonstrated like how to set up the Kafka environment … Continue reading

Posted in Apache Kafka, Scala | Tagged , , , | Leave a comment

Meetup: Stream processing using Kafka


Knoldus organized a Meetup on Friday, 7th April 2017 at 4:00 PM which was presented by Himani Arora and me(Prabhat Kashyap). Topics which were covered in this meetup: What is Stream processing Advantages of stream processing Type of stream processing What are KStreams Use cases … Continue reading

Posted in Apache Kafka, Scala, Streaming | Tagged , , | Leave a comment

Integrating Kafka With Spark Structure Streaming


Kafka is a messaging broker system which facilitates the passing of messages between producer and consumer whereas Spark Structure streaming consumes static and streaming data from various sources like kafka, flume, twitter or any other socket which can be processed … Continue reading

Posted in Apache Kafka, apache spark, Scala, Streaming | Tagged , , | 1 Comment

Spark Streaming vs Kafka Stream


The demand for stream processing is increasing a lot these days. The reason is that often processing big volumes of data is not enough. Data has to be processed fast, so that a firm can react to changing business conditions … Continue reading

Posted in Apache Kafka, apache spark, big data, Scala, Streaming | Tagged , | 1 Comment

Streaming in Spark, Flink and Kafka


There is a lot of buzz going on between when to use use spark, when to use flink, and when to use Kafka. Both spark streaming and flink provides exactly once guarantee that every record will be processed exactly once … Continue reading

Posted in Apache Flink, Apache Kafka, apache spark, Streaming | Tagged , , , | Leave a comment

Twitter’s tweets analysis using Lambda Architecture


Hello Folks, In this blog i will explain  twitter’s tweets analysis with lambda architecture. So first we need to understand  what is lambda architecture,about its component and usage. According to Wikipedia, Lambda architecture is a data processing architecture designed to handle … Continue reading

Posted in Akka, akka-http, Apache Kafka, apache spark, Architecture, Batch, big data, Cassandra, Scala, Spark, Streaming | 6 Comments

Kafka – Sending Object as a message


Kafka lets us publish and subscribe to streams of records and the records can be of any type, it can be JSON, String, POJO, etc. Kafka gives user the ability to creates our own serializer and deserializer so that we … Continue reading

Posted in Apache Kafka, Java | 1 Comment