Knoldus Blogs

Which is better EventStoreDB or Kafka

December 25, 2022December 29, 2022EventStoreDB, Tech Blogseventstoredb, kafka

Reading Time: 4 minutes Introduction EventStoreDB is a database management system optimized for storing and processing event data. It is often used in event-driven architectures, where the database is a central hub for storing and processing data from various sources. EventStoreDB is designed to be scalable, resilient, and fast, making it well-suited for high-volume applications that require real-time data processing. Building real-time data pipelines and streaming applications uses the Continue Reading

Hacker hands using laptop computer to code

Connecting Kafka With Micronaut

November 18, 2022November 21, 2022Micronaut, Tech Blogskafka, Micronaut

Reading Time: 2 minutes connecting micronaut with kafka

Visualizing data - blue matrix with data, electronic, digital, abstract, dark blue, data science

Sink Connector: The MarkLogic Kafka Connector

November 16, 2022November 21, 2022Javajava, kafka, Kafka connect, marklogic, streaming data

Reading Time: 2 minutes The MarkLogic Kafka connector is a sink connector for receiving messages from Kafka and writing them to a MarkLogic database. The sink pulls messages from the Kafka topics to store in MarkLogic as JSON documents. This acquires messages using Kafka from numerous brokers and then writes to marklogic with no coding required. The connector uses the MarkLogic Data Movement SDK (DMSDK) to store those messages in a Continue Reading

Getting started with Apache Kafka and Quarkus

September 29, 2022October 19, 2022JavaAsynchronous messaging, java, kafka, Quarkus, reactive microservices

Reading Time: 4 minutes Overview The ideal framework for continuous delivery and greater resilience is provided by micro-service-oriented architecture. In addition to boosting developer efficiency and improving real-time scalability, they encourage quicker innovation to respond to shifting market conditions. In order to build scalable architectures, we need event-driven and asynchronous integration between microservices. We have multiple options available for asynchronous integration. Kafka is the most popular platform offering characteristics Continue Reading

Axon vs Kafka

July 19, 2022July 22, 2022Axon, kafkaAxon, kafka

Reading Time: 4 minutes Introduction One of the most common discussion points that come up regularly in interactions with Customers/Prospects/Forums is how does Axon compare to Apache Kafka? Can one do the job of the other? Are they complementary to each other? Can they work together? Does Axon provide any capabilities to work with Kafka? Apache Kafka is a very popular system for publishing and consuming events. Its architecture Continue Reading

How Kafka Relates to Axon Framework?

July 12, 2022July 30, 2022Apache Kafka, CQRS, event, Java, kafka, Streaming, Streaming SolutionsAxon, kafka

Reading Time: 3 minutes Axon and Kafka are used for different purposes, Axon is used for Event-Driven Architecture and provides the application-level support for domain modeling and Event Sourcing, as well as the routing of Commands and Queries, while Kafka serves as an Event Streaming system. The basic fundamental of Axon is to implement CQRS and Event Sourcing-based architecture. With the help of this, we can design & develop Continue Reading

Kafka Connect example: Mysql to Elastic Search

June 21, 2022June 22, 2022Java, Kafka Connect / Connector, Tech BlogsClean code, confluent, connect, connector, Elastic search, elasticsearch, Java8, jdbc, kafka, Kafka connect

Reading Time: 3 minutes Overview: Hello everyone, in this blog, we will see an example of Kafka connect in which we will take a MySQL table, stream it to a Kafka topic, and from there load it to Elasticsearch and index its content. Installation: Now first of all we will install MySQL and Elastic search to our local system. For installing simply run: The next step is to make Continue Reading

Deep dive into Kafka Connect

June 21, 2022June 21, 2022Kafka Connect / Connectorkafka, Kafka connect

Reading Time: 6 minutes Hello! In this article we will continue our journey of understanding Kafka Connect. We will try to understand the architecture and internals of it. We’ve seen that Kafka Connect is a pluggable component that helps to fed data into or from Kafka and hence provides flexible integration pipelines. It is inherently fault tolerant and sacalable. To work with any software component and get the most Continue Reading

Streaming Kafka Messages to Google Cloud Pub/Sub

May 25, 2022May 25, 2022Apache Kafka, GCPkafka, pub-sub

Reading Time: 3 minutes In this blog post i present an example that creates a pipeline to read data from a single topic/multiple topics from Apache Kafka and write data into a topic in Google Pub/Sub. The example provides code samples to implement simple yet powerful pipelines.also provides an out-of-the-box solution that you can just ” compatiable.This consicutive example is build in Apache Beam.And it can be downloaded here.So, we hope you will find this Continue Reading

Kafka connector with MongoDB

May 24, 2022May 24, 2022Kafka Connect / Connectorkafka, kafka connector, MongoDB

Reading Time: 3 minutes The MongoDB Kafka connector is a Confluent-verified connector that persists data from Kafka topics as a data sink into MongoDB as well as publishes changes from MongoDB into Kafka topics as a data source. Apache Kafka The Apache Kafka is an open-source publish/subscribe messaging system. Apache Kafka provides a flexible, fault tolerant, and horizontally scalable system to move data throughout datastores and applications. A system is fault tolerant if the Continue Reading

How to solve Producer and Consumer problems

May 20, 2022May 20, 2022Studio-DevOpsdocker, kafka, kafka consumer, kafka producer

Reading Time: 3 minutes In this blog, we will learn about how to solve producer and consumer problems using Kafka and Docker. It’s going to be interesting. So stay tuned Kafka is a high-performance, real-time, and also publish-subscribe messaging system. It is an open-source tool as well as a part of Apache Projects. Some Characteristics of Kafka are:- Firstly it is a distributed and partitioned messaging system. Secondly, Kafka Continue Reading

How to setup and launch Kafka Connect/Connector

May 17, 2022May 17, 2022Kafka Connect / ConnectorJava Studio, kafka, Kafka connect, Tech blogs

Reading Time: 3 minutes Let us discuss something about Kafka connector API and some basic fundamental of it and how to setup kafka connector in our system. Before start, we need to have basic knowledge of Kafka or we can go through this Document. We’ll have a look at: Kafka Connect API history Why kafka connect and Stream Setup and Launch kafka connector installation of Docker installation of Docker Continue Reading

An introduction to Kafka Connect

April 22, 2022May 23, 2022Kafka Connect / Connectorkafka

Reading Time: 4 minutes Hello! In this article we are going to talk about the Kafka Connect. This page includes why, what and features of Kafka connect. What is Kafka Connect? Kafka Connect, is the pluggable and declarative data integration framework for Kafka. It connects data source/destination to Kafka, letting the rest of the ecosystem do what it is expected from it. It is declarative and makes integration between Kafka Continue Reading