Tag Archives: kafka

Join Semantics in Kafka Streams


Introduction to core concepts:   Apache Kafka is a distributed streaming platform which enables you to publish and subscribe to a stream of records also letting you process this stream of records as it occurs. Kafka Streams is a client … Continue reading

Posted in Apache Kafka, Scala | Tagged , , , , , , | 2 Comments

Basic Example for Spark Structured Streaming & Kafka Integration


The Spark Streaming integration for Kafka 0.10 is similar in design to the 0.8 Direct Stream approach. It provides simple parallelism, 1:1 correspondence between Kafka partitions and Spark partitions, and access to offsets and metadata. However, because the newer integration … Continue reading

Posted in Scala, Spark, Streaming | Tagged , | 6 Comments

RealTimeProcessing of Data using kafka and Spark


Before Starting it you should know about kafka, spark and what is Real time processing of Data.so let’s do some brief introduction about it. Real Time Processing – Processing the Data that appears to take place instead of storing the data and then … Continue reading

Posted in Scala | Tagged , , , , | 1 Comment

Unit Testing Of Kafka


Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another. Generally, data is published to topic via Producer API … Continue reading

Posted in Apache Kafka, Scala, scalatest, Streaming, testing | Tagged , , , , | 3 Comments

Setting It Up: KAFKA Multi-Broker System


In this blog, I am going to cover up the leftovers of my last blog: “A Beginners Approach To KAFKA” in which I tried to explain the details of Kafka, like its terminologies, advantages and demonstrated like how to set up the Kafka environment … Continue reading

Posted in Apache Kafka, Scala | Tagged , , , | Leave a comment

A Beginners Approach To “KAFKA”


Heavy Data Load? Kafka Is Here For You. In this blog, I am going to get into the details like: What is Kafka? Getting familiar with Kafka. Learning some basics in Kafka. Creating a general Single Broker Cluster. So let’s … Continue reading

Posted in Scala | Tagged , | 2 Comments

Meetup: Stream processing using Kafka


Knoldus organized a Meetup on Friday, 7th April 2017 at 4:00 PM which was presented by Himani Arora and me(Prabhat Kashyap). Topics which were covered in this meetup: What is Stream processing Advantages of stream processing Type of stream processing What are KStreams Use cases … Continue reading

Posted in Apache Kafka, Scala, Streaming | Tagged , , | Leave a comment

Integrating Kafka With Spark Structure Streaming


Kafka is a messaging broker system which facilitates the passing of messages between producer and consumer whereas Spark Structure streaming consumes static and streaming data from various sources like kafka, flume, twitter or any other socket which can be processed … Continue reading

Posted in Apache Kafka, apache spark, Scala, Streaming | Tagged , , | 1 Comment

Streaming in Spark, Flink and Kafka


There is a lot of buzz going on between when to use use spark, when to use flink, and when to use Kafka. Both spark streaming and flink provides exactly once guarantee that every record will be processed exactly once … Continue reading

Posted in Apache Flink, Apache Kafka, apache spark, Streaming | Tagged , , , | Leave a comment

Introducing Kafka Streams: Processing made easy


If you are working on huge amount of data, you might have heard about Kafka. At a very high level, Kafka is a fault tolerant, distributed publish-subscribe messaging system that is designed for fast processing of data and the ability … Continue reading

Posted in big data, Java, Streaming | Tagged , , , | 1 Comment