kafka

Visualizing data - blue matrix with data, electronic, digital, abstract, dark blue, data science

Sink Connector: The MarkLogic Kafka Connector

Reading Time: 2 minutes The MarkLogic Kafka connector is a sink connector for receiving messages from Kafka and writing them to a MarkLogic database. The sink pulls messages from the Kafka topics to store in MarkLogic as JSON documents. This acquires messages using Kafka from numerous brokers and then writes to marklogic with no coding required. The connector uses the MarkLogic Data Movement SDK (DMSDK) to store those messages in a Continue Reading

Getting started with Apache Kafka and Quarkus

Reading Time: 4 minutes Overview The ideal framework for continuous delivery and greater resilience is provided by micro-service-oriented architecture. In addition to boosting developer efficiency and improving real-time scalability, they encourage quicker innovation to respond to shifting market conditions. In order to build scalable architectures, we need event-driven and asynchronous integration between microservices. We have multiple options available for asynchronous integration. Kafka is the most popular platform offering characteristics Continue Reading

Axon vs Kafka

Reading Time: 4 minutes Introduction One of the most common discussion points that come up regularly in interactions with Customers/Prospects/Forums is how does Axon compare to  Apache Kafka? Can one do the job of the other? Are they complementary to each other? Can they work together? Does Axon provide any capabilities to work with Kafka? Apache Kafka is a very popular system for publishing and consuming events. Its architecture Continue Reading

How Kafka Relates to Axon Framework?

Reading Time: 3 minutes Axon and Kafka are used for different purposes, Axon is used for Event-Driven Architecture and provides the application-level support for domain modeling and Event Sourcing, as well as the routing of Commands and Queries, while Kafka serves as an Event Streaming system. The basic fundamental of Axon is to implement CQRS and Event Sourcing-based architecture.  With the help of this, we can design & develop Continue Reading

Kafka Connect example: Mysql to Elastic Search

Reading Time: 3 minutes Overview: Hello everyone, in this blog, we will see an example of Kafka connect in which we will take a MySQL table, stream it to a Kafka topic, and from there load it to Elasticsearch and index its content. Installation: Now first of all we will install MySQL and Elastic search to our local system. For installing simply run: The next step is to make Continue Reading

Deep dive into Kafka Connect

Reading Time: 6 minutes Hello! In this article we will continue our journey of understanding Kafka Connect. We will try to understand the architecture and internals of it. We’ve seen that Kafka Connect is a pluggable component that helps to fed data into or from Kafka and hence provides flexible integration pipelines. It is inherently fault tolerant and sacalable. To work with any software component and get the most Continue Reading

Streaming Kafka Messages to Google Cloud Pub/Sub

Reading Time: 3 minutes In this blog post i present an example that creates a pipeline to read data from a single topic/multiple topics from Apache Kafka and write data into a topic in Google Pub/Sub. The example provides code samples to implement simple yet powerful pipelines.also provides an out-of-the-box solution that you can just ” compatiable.This consicutive example is build in Apache Beam.And it can be downloaded here.So, we hope you will find this Continue Reading

Kafka connector with MongoDB

Reading Time: 3 minutes The MongoDB Kafka connector is a Confluent-verified connector that persists data from Kafka topics as a data sink into MongoDB as well as publishes changes from MongoDB into Kafka topics as a data source. Apache Kafka The Apache Kafka is an open-source publish/subscribe messaging system. Apache Kafka provides a flexible, fault tolerant, and horizontally scalable system to move data throughout datastores and applications. A system is fault tolerant if the Continue Reading

How to solve Producer and Consumer problems

Reading Time: 3 minutes In this blog, we will learn about how to solve producer and consumer problems using Kafka and Docker. It’s going to be interesting. So stay tuned Kafka is a high-performance, real-time, and also publish-subscribe messaging system. It is an open-source tool as well as a part of Apache Projects. Some Characteristics of Kafka are:- Firstly it is a distributed and partitioned messaging system. Secondly, Kafka Continue Reading

How to setup and launch Kafka Connect/Connector

Reading Time: 3 minutes Let us discuss something about Kafka connector API and some basic fundamental of it and how to setup kafka connector in our system. Before start, we need to have basic knowledge of Kafka or we can go through this Document. We’ll have a look at: Kafka Connect API history Why kafka connect and Stream Setup and Launch kafka connector installation of Docker installation of Docker Continue Reading

An introduction to Kafka Connect

Reading Time: 4 minutes Hello! In this article we are going to talk about the Kafka Connect. This page includes why, what and features of Kafka connect. What is Kafka Connect? Kafka Connect, is the pluggable and declarative data integration framework for Kafka. It connects data source/destination to Kafka, letting the rest of the ecosystem do what it is expected from it. It is declarative and makes integration between Kafka Continue Reading

Kafka Connect Fundamentals

Reading Time: 3 minutes Kafka connect is an framework to connect kafka with external ecosystem like file systems, databases using kafka connector. The Kafka Connect cluster supports running and scaling out connectors. Kafka Connectors are ready-to-use components, which can help us to import data from external systems into Kafka topics and export data from Kafka topics into external systems.  What is Kafka Connect? Kafka connect is use to perform stream integration Continue Reading