How to setup and launch Kafka Connect/Connector

Table of contents

Reading Time: 3 minutes

Let us discuss something about Kafka connector API and some basic fundamental of it and how to setup kafka connector in our system. Before start, we need to have basic knowledge of Kafka or we can go through this Document.

We’ll have a look at:

Kafka Connect API history
Why kafka connect and Stream
Setup and Launch kafka connector
installation of Docker
installation of Docker Composer

1- Kafka Connect API history

In (2013) there was a Kafka 0.8.x:

Topic replication, Log compaction
Simplified producer client API

(Nov 2015) Kafka 0.9.x:

Simplified high level consumer APIS, without Zookeeper dependency
Added security (Encryption and Authentication)
Kafka Connect APIS

(May 2016): Kafka 0.10.0:

Kafka Streams APIS so finally when kafka 10 version is out then kafka streams APIs are come into the picture.

(end 2016–March 2017) Kafka 0.10.1,0.10.2:

Improved Connect API, Single Message Transforms API

So basically kafka connect been around 5 year now and every year continuously improving.

2- Why kafka connect and Stream ?

Basically when you have kafka you have four common use cases

Source => Kafka Producer API
Kafka => Kafka Consumer, Producer API
Kafka => Sink Consumer API
Kafka => App Consumer API

So in first case kafka connect allow you to easily have a source and put all the data into kafka. Then Kafka’s team served a meal purpose to do transformations on kafka topics .more importantly Kafka connect sync serves the third purpose into getting data out of kafka wherever you want and for the last use case the consumer API still great so we keep it this way .

3- Setup and Launch kafka connector

So setup kafka connector we need few things in our system.

Docker
Docker composer
Kafka

4- So let’s install Docker in our system

Set up the repository

Update the apt package index and install packages to allow apt to use a repository over HTTPS:

sudo apt-get update

sudo apt-get install 
    ca-certificates 
    curl 
    gnupg 
    lsb-release

Now we need to add docker official GPG key:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor 
-o /usr/share/keyrings/docker-archive-keyring.gpg

Now we need to varify the key :

echo 
  "deb [arch=$(dpkg --print-architecture) signed-
by=/usr/share/keyrings/docker-archive-keyring.gpg] 
https://download.docker.com/linux/ubuntu 
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

After that we need to update the apt package :

 sudo apt-get update

Now install the latest version of docker

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

To check whether docker is installed,just run the command :

 docker --version

If you saw the above image that’s means Docker is successfully installed in your system.

5- Now we are going to Installed Docker compose in our system

Update the apt package index

 sudo apt-get update

Now install the latest version of Docker Compose

sudo apt-get install docker-compose-plugin

Run this command to download the current stable release of Docker Compose

curl -SL https://github.com/docker/compose/releases/download/v2.5.0/docker-

compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose

Check the Docker compose

docker compose version

In this blog we had look at brief history of kafka connect and we also looked at steps how we can setup the kafka connect And finally, we learned installation process of the docker and docker compose in the next we will discuss about kafka.

That’s it, folks. I hope you liked the blogs. Thanks!