Category Archives: Apache Kafka

KnolX: Learning Kafka Streams with Scala


Hello everyone, Knoldus organized a session on 22nd September 2017. The topic was “Learning Kafka Streams with Scala”. Many people attended and enjoyed the session. In this blog post, I am going to share the slides & video of the session. Slides: … Continue reading

Posted in Apache Kafka, Scala | Tagged , , | Leave a comment

Case Study to understand Kafka Consumer and its offsets


In this blog post, we will discuss mainly Kafka Consumer and its Offsets. We will understand this using a case study implemented in Scala. This blog post assumes that you are aware of basic Kafka terminology. CASE STUDY: The Producer … Continue reading

Posted in Apache Kafka, big data, Functional Programming, knoldus, Scala, Streaming | 3 Comments

Join Semantics in Kafka Streams


Introduction to core concepts:   Apache Kafka is a distributed streaming platform which enables you to publish and subscribe to a stream of records also letting you process this stream of records as it occurs. Kafka Streams is a client … Continue reading

Posted in Apache Kafka, Scala | Tagged , , , , , , | 2 Comments

A Java Lagom service which only consumes from Kafka topic (Subscriber only service)


Subscriber only service means an application which only consumes, does not produce. We have generally seen the applications which both produces and consumes data from a Kafka topic but sometimes we need to write an application which only consumes data … Continue reading

Posted in Akka, Apache Kafka, Architecture, Best Practices, big data, Functional Programming, github, Java, MessagesAPI, Microservices, Scala | Leave a comment

Having Issue How To Order Streamed Dataframe ?


A few days ago, i have to perform aggregation on streaming dataframe. And the moment, i apply groupBy for aggregation, data gets shuffled. Now the situation arises how to maintain order? Yes, i can use orderBy with streaming dataframe using … Continue reading

Posted in Apache Kafka, apache spark, big data, Scala, Spark, Streaming | Tagged , , , , , , , , , , | 1 Comment

Self-Learning Kafka Streams with Scala – #2


In our previous blog – Self-Learning Kafka Streams with Scala – #1, we saw how to create a simple KStream in Scala. In this blog, we will see how to transform a KStream and create a new Stream from it. But, … Continue reading

Posted in Apache Kafka, Scala | Tagged | Leave a comment

Kafka Streams: Unit Testing


Kafka Streams is a powerful API. In Kafka, we can only store our data for consumers to consume. But we always needed a processor with which we can process the data without going to an external tool like Spark, Storm … Continue reading

Posted in Apache Kafka, Scala | Tagged , , | 1 Comment

Self-Learning Kafka Streams with Scala – #1


A few days ago, I came across a situation where I wanted to do a stateful operation on the streaming data. So, I started finding possible solutions for it. I came across many solutions which were using different technologies like … Continue reading

Posted in Apache Kafka, Scala, Streaming | Tagged | 2 Comments

Kafka Streams : More Than Just a Dumb Storage


Whenever we hear the word Kafka, all we think about it as a messaging system with a publisher-subscriber model that we use for our streaming applications as a source and a sink. So we can say that Kafka is just a … Continue reading

Posted in Apache Kafka, Scala | Tagged , , , | 3 Comments

Unit Testing Of Kafka


Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another. Generally, data is published to topic via Producer API … Continue reading

Posted in Apache Kafka, Scala, scalatest, Streaming, testing | Tagged , , , , | 3 Comments