Category Archives: Apache Kafka

A Beginner’s Guide to Deploying a Lagom Service Without ConductR


How to deploy a Lagom Service without ConductR? This question has been asked and answered by many, on different forums. For example, take a look at this question on StackOverflow – Lagom without ConductR? Here the user is trying to know whether … Continue reading

Posted in Akka, Apache Kafka, Cassandra, cluster, Microservices, Scala | Tagged , | 1 Comment

Kafka And Spark Streams: The happily ever after !!


Hi everyone, Today we are going to understand a bit about using the spark streaming to transform and transport data between Kafka topics. The demand for stream processing is increasing every day. The reason is that often, processing big volumes … Continue reading

Posted in Apache Kafka, apache spark, Scala, Spark, Streaming | Tagged , , , , , | 1 Comment

Error Registering Avro Schema | Multiple Schemas In One Topic


org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {“type”:”record”,”name”:”schema1″,”namespace”:”test”,”fields”:[{“name”:”Name”,”type”:”string”},{“name”:”Age”,”type”:”int”},{“name”:”Location”,”type”:”string”}]} Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema being registered is incompatible with an earlier schema; error code: 409 at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:170) at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:188) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:245) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:237) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:232) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:59) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:91) at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:72) at io.confluent.kafka.formatter.AvroMessageReader.readMessage(AvroMessageReader.java:158) at … Continue reading

Posted in Apache Kafka, Java, Scala | Tagged , , , , | 1 Comment

Assimilation of Spark Streaming With Kafka


As we know Spark is used at a wide range of organizations to process large datasets. It seems like spark becoming main stream. In this blog we will talk about Integration of Kafka with Spark Streaming. So, lets get started. How Kafka … Continue reading

Posted in Apache Kafka, apache spark, Scala | Tagged , , , | 2 Comments

KnolX: Learning Kafka Streams with Scala


Hello everyone, Knoldus organized a session on 22nd September 2017. The topic was “Learning Kafka Streams with Scala”. Many people attended and enjoyed the session. In this blog post, I am going to share the slides & video of the session. Slides: … Continue reading

Posted in Apache Kafka, Scala | Tagged , , | 1 Comment

Case Study to understand Kafka Consumer and its offsets


In this blog post, we will discuss mainly Kafka Consumer and its Offsets. We will understand this using a case study implemented in Scala. This blog post assumes that you are aware of basic Kafka terminology. CASE STUDY: The Producer … Continue reading

Posted in Apache Kafka, big data, Functional Programming, knoldus, Scala, Streaming | 4 Comments

Join Semantics in Kafka Streams


Introduction to core concepts:   Apache Kafka is a distributed streaming platform which enables you to publish and subscribe to a stream of records also letting you process this stream of records as it occurs. Kafka Streams is a client … Continue reading

Posted in Apache Kafka, Scala | Tagged , , , , , , | 3 Comments

A Java Lagom service which only consumes from Kafka topic (Subscriber only service)


Subscriber only service means an application which only consumes, does not produce. We have generally seen the applications which both produces and consumes data from a Kafka topic but sometimes we need to write an application which only consumes data … Continue reading

Posted in Akka, Apache Kafka, Architecture, Best Practices, big data, Functional Programming, github, Java, MessagesAPI, Microservices, Scala | 1 Comment

Having Issue How To Order Streamed Dataframe ?


A few days ago, i have to perform aggregation on streaming dataframe. And the moment, i apply groupBy for aggregation, data gets shuffled. Now the situation arises how to maintain order? Yes, i can use orderBy with streaming dataframe using … Continue reading

Posted in Apache Kafka, apache spark, big data, Scala, Spark, Streaming | Tagged , , , , , , , , , , | 2 Comments

Self-Learning Kafka Streams with Scala – #2


In our previous blog – Self-Learning Kafka Streams with Scala – #1, we saw how to create a simple KStream in Scala. In this blog, we will see how to transform a KStream and create a new Stream from it. But, … Continue reading

Posted in Apache Kafka, Scala | Tagged | 1 Comment