kafka

Kafka Connect example: Mysql to Elastic Search

Reading Time: 3 minutes Overview: Hello everyone, in this blog, we will see an example of Kafka connect in which we will take a MySQL table, stream it to a Kafka topic, and from there load it to Elasticsearch and index its content. Installation: Now first of all we will install MySQL and Elastic search to our local system. For installing simply run: The next step is to make Continue Reading

Deep dive into Kafka Connect

Reading Time: 6 minutes Hello! In this article we will continue our journey of understanding Kafka Connect. We will try to understand the architecture and internals of it. We’ve seen that Kafka Connect is a pluggable component that helps to fed data into or from Kafka and hence provides flexible integration pipelines. It is inherently fault tolerant and sacalable. To work with any software component and get the most Continue Reading

Streaming Kafka Messages to Google Cloud Pub/Sub

Reading Time: 3 minutes In this blog post i present an example that creates a pipeline to read data from a single topic/multiple topics from Apache Kafka and write data into a topic in Google Pub/Sub. The example provides code samples to implement simple yet powerful pipelines.also provides an out-of-the-box solution that you can just ” compatiable.This consicutive example is build in Apache Beam.And it can be downloaded here.So, we hope you will find this Continue Reading

Kafka connector with MongoDB

Reading Time: 3 minutes The MongoDB Kafka connector is a Confluent-verified connector that persists data from Kafka topics as a data sink into MongoDB as well as publishes changes from MongoDB into Kafka topics as a data source. Apache Kafka The Apache Kafka is an open-source publish/subscribe messaging system. Apache Kafka provides a flexible, fault tolerant, and horizontally scalable system to move data throughout datastores and applications. A system is fault tolerant if the Continue Reading

How to solve Producer and Consumer problems

Reading Time: 3 minutes In this blog, we will learn about how to solve producer and consumer problems using Kafka and Docker. It’s going to be interesting. So stay tuned Kafka is a high-performance, real-time, and also publish-subscribe messaging system. It is an open-source tool as well as a part of Apache Projects. Some Characteristics of Kafka are:- Firstly it is a distributed and partitioned messaging system. Secondly, Kafka Continue Reading

How to setup and launch Kafka Connect/Connector

Reading Time: 3 minutes Let us discuss something about Kafka connector API and some basic fundamental of it and how to setup kafka connector in our system. Before start, we need to have basic knowledge of Kafka or we can go through this Document. We’ll have a look at: Kafka Connect API history Why kafka connect and Stream Setup and Launch kafka connector installation of Docker installation of Docker Continue Reading

An introduction to Kafka Connect

Reading Time: 4 minutes Hello! In this article we are going to talk about the Kafka Connect. This page includes why, what and features of Kafka connect. What is Kafka Connect? Kafka Connect, is the pluggable and declarative data integration framework for Kafka. It connects data source/destination to Kafka, letting the rest of the ecosystem do what it is expected from it. It is declarative and makes integration between Kafka Continue Reading

Kafka Connect Fundamentals

Reading Time: 3 minutes Kafka connect is an framework to connect kafka with external ecosystem like file systems, databases using kafka connector. The Kafka Connect cluster supports running and scaling out connectors. Kafka Connectors are ready-to-use components, which can help us to import data from external systems into Kafka topics and export data from Kafka topics into external systems.  What is Kafka Connect? Kafka connect is use to perform stream integration Continue Reading

Security & SSL Setup in Confluent Kafka

Reading Time: 2 minutes What is SSL ? Secure Socket Layer (SSL) is a security protocol for the transport layer. In SSL Protocol data is divided into fragments. The fragments are compressed and encrypted Message Authentication Code (MAC) generated by algorithms like Secure Hash Protocol(SHA) and MD5(Message Digest) is appended. SSL is the predecessor of Transport Layer Security(TLS) . After encryption of data, finally, the SSL header is appended Continue Reading

Kafka Streams

Reading Time: 2 minutes What are Streams Streams are known as unbounded and continuous flow of data packets in real time. Data packets are generally generated in form of key value pair. Producer transfer these packets automatically, means there is no need to place a request. What are Kafka Streams Kafka Streams is one of the project of Apache Kafka community. It is a client library for building data Continue Reading

Introduction to Kafka

Reading Time: 5 minutes Apache Kafka is a software platform that is based on a distributed streaming process. It is a publish-subscribe messaging system that lets exchanging of data between applications, servers, and processors as well. Apache Kafka was originally developed by LinkedIn, and later it was donated to the Apache Software Foundation. Apache Kafka has resolved the lethargic trouble of data communication between a sender and a receiver. Architecture Continue Reading

Introduction To Apache Kafka

Reading Time: 6 minutes Introduction Apache Kafka is a framework implementation of a software bus using stream-processing . It is an open source platform, developed by the Apache Software Foundation. It is written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka can connect to external systems (for data import/export) via Kafka Connect and provides Kafka Streams, a Java stream processing library. Apache Continue Reading

Kafka Kerberos Authentication

Reading Time: 2 minutes In this article we will start looking into Kerberos authentication and will focus on the client-side configuration required to authenticate with clusters configured to use Kerberos. Kafka supports four different communication protocols between Consumers, Producers, and Brokers. Each protocol considers different security aspects, while PLAINTEXT is the old insecure communication protocol. PLAINTEXT (non-authenticated, non-encrypted) SSL (SSL authentication, encrypted) PLAINTEXT+SASL (authentication, non-encrypted) SSL+SASL (encrypted authentication, encrypted Continue Reading