How to setup and launch Kafka Connect/Connector

Reading Time: 3 minutes

Let us discuss something about Kafka connector API and some basic fundamental of it and how to setup kafka connector in our system. Before start, we need to have basic knowledge of Kafka or we can go through this Document.

We’ll have a look at:

  • Kafka Connect API history
  • Why kafka connect and Stream
  • Setup and Launch kafka connector
  • installation of Docker
  • installation of Docker Composer

1- Kafka Connect API history

In (2013) there was a Kafka 0.8.x:

  • Topic replication, Log compaction
  • Simplified producer client API

(Nov 2015) Kafka 0.9.x:

  • Simplified high level consumer APIS, without Zookeeper dependency
  • Added security (Encryption and Authentication)
  • Kafka Connect APIS

(May 2016): Kafka 0.10.0:

  • Kafka Streams APIS so finally when kafka 10 version is out then kafka streams APIs are come into the picture.

(end 2016–March 2017) Kafka 0.10.1,0.10.2:

  • Improved Connect API, Single Message Transforms API

So basically kafka connect been around 5 year now and every year continuously improving.

2- Why kafka connect and Stream ?

Basically when you have kafka you have four common use cases

  • Source => Kafka Producer API
  • Kafka => Kafka Consumer, Producer API
  • Kafka => Sink Consumer API
  • Kafka => App Consumer API

So in first case kafka connect allow you to easily have a source and put all the data into kafka. Then Kafka’s team served a meal purpose to do transformations on kafka topics .more importantly Kafka connect sync serves the third purpose into getting data out of kafka wherever you want and for the last use case the consumer API still great so we keep it this way .

3- Setup and Launch kafka connector

So setup kafka connector we need few things in our system.

  1. Docker
  2. Docker composer
  3. Kafka

4- So let’s install Docker in our system

Set up the repository

  • Update the apt package index and install packages to allow apt to use a repository over HTTPS:
sudo apt-get update
sudo apt-get install 
  • Now we need to add docker official GPG key:
curl -fsSL | sudo gpg --dearmor 
-o /usr/share/keyrings/docker-archive-keyring.gpg
  • Now we need to varify the key :
  "deb [arch=$(dpkg --print-architecture) signed-
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  • After that we need to update the apt package :
 sudo apt-get update
  • Now install the latest version of docker
sudo apt-get install docker-ce docker-ce-cli docker-compose-plugin

To check whether docker is installed,just run the command :

 docker --version

If you saw the above image that’s means Docker is successfully installed in your system.

5- Now we are going to Installed Docker compose in our system

  • Update the apt package index
 sudo apt-get update
  • Now install the latest version of Docker Compose
sudo apt-get install docker-compose-plugin
  • Run this command to download the current stable release of Docker Compose
curl -SL

compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
  • Check the Docker compose
docker compose version

In this blog we had look at brief history of kafka connect and we also looked at steps how we can setup the kafka connect And finally, we learned installation process of the docker and docker compose in the next we will discuss about kafka.

That’s it, folks. I hope you liked the blogs. Thanks!


To read more tech blogs, visit Knoldus Blogs.

Written by 

Gaurav srivastav is a Software Consultant working in java domain.