Kafka-Using Kafka-Docker to solve producer and consumer problems

Reading Time: 3 minutes

In this blog, we will learn about how to solve producer and consumer problems using Kafka and Docker. It’s going to be interesting. So stay tuned

Kafka is a high-performance, real-time, and also publish-subscribe messaging system. It is an open-source tool as well as a part of Apache Projects.

Some Characteristics of Kafka are:-

Firstly it is a distributed and partitioned messaging system.
Secondly, Kafka is highly scalable.
Thirdly it can process and send millions of messages per second to several receivers.

Standard terms used in Kafka:-

Messages represent data, such as lines in a log file and system error messages.
A Topic is a category of messages in Kafka and the producer organizes the messages into topics. Consumers read the messages from the topics.
The producer is the creator of messages likewise the consumer reads messages in Kafka.
Partitions allow messages on a topic to be distributed to multiple servers.

Let’s talk about the producer’s and consumers’ problem

The producer and consumer problem is the classical synchronization problem, in which the producer writes messages to a shared buffer and the consumer consumes those messages.

The problem in this synchronization problem is that the producer continues to produce messages without knowing whether or not the consumer will be able to consume them. To solve this problem we use Kafka.

How is Kafka used to solve the problem?

Explanation of the above diagram

The Kafka cluster consists of Brokers that take messages from the producer to add them to the partitions.
The broker provides messages to consumers from the partitions and handles all the clients’ requests (producer and consumer), but keeps the data within the cluster.
Each partition at as a message queue in the cluster.
Additionally, the zookeeper manages the Broker.

Use of Docker

As containerized applications are running with the help of Docker similarly we will create multiple containers for these.

Kafka Server
Zookeeper Server
Producer Console
Consumer Console

First, we have to write Dockerfiles for each container

1. Kafka Server

FROM openjdk:8-jre-slim

WORKDIR /kafka-server

ADD kafka/ /kafka-server/

EXPOSE 9092

CMD [ "./bin/kafka-server-start.sh","config/server.properties" ]

FROM- act as a base image for the server.
WORKDIR- to set a working directory for all Dockerfile instructions.
ADD- to copy any folder or file inside the current directory.
EXPOSE- on which the server will run.
CMD- to start the server.

2. Zookeeper

FROM openjdk:8-jre-slim

WORKDIR /kafka-server

ADD kafka/ /kafka-server/

EXPOSE 2181

CMD ./bin/zookeeper-server-start.sh config/zookeeper.properties

3. Producer

FROM openjdk:8-jre-slim

WORKDIR /kafka-server

ADD kafka/ /kafka-server/

ENTRYPOINT ["./bin/kafka-console-producer.sh","--bootstrap-server","server:9092","--topic"]

CMD [ "test" ]

4. Consumer

FROM openjdk:8-jre-slim

WORKDIR /kafka-server

ADD kafka/ /kafka-server/

ENTRYPOINT [ "./bin/kafka-console-consumer.sh","--bootstrap-server","server:9092","--from-beginning","--topic" ]

CMD [ "test" ]

To start the services of these containers, we write a Docker-Compose file

version: '3.6'

services:
  zookeeper:
    restart: always
    ports:
      - 2181:2181
    container_name: zookeeper
    build: 
      context: .
      dockerfile: zookeeper/Dockerfile
    networks:
      - kafka
    volumes:
      - ~/Desktop/kafka-zoo:/tmp/zookeeper
  server:
    ports:
      - 9092:9092
    container_name: server
    build: 
      context: .
      dockerfile: server/Dockerfile
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 0
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_CREATE_TOPICS: 'test'
    networks:
      - kafka
    volumes:
      - ~/Desktop/kafka-data:/tmp/kafka-logs
  producer:
    container_name: producer
    build: 
      context: .
      dockerfile: producer/Dockerfile
    depends_on:
      - zookeeper
      - server
    networks:
      - kafka

  consumer:
    container_name: consumer
    build: 
      context: .
      dockerfile: consumer/Dockerfile
    depends_on:
      - zookeeper
      - server
    networks:
      - kafka
networks:
  kafka:

In the above file, we have four services servers, zookeeper, producer, and consumer.
Ports on which these services will run.
Location of Dockerfiles(like server/Dockerfile).
Networks are for communication between containers.
Volumes to store data.

docker-compose file is used for running multiple containers

OUTPUT:-

To run the docker-compose file

docker-compose up --build

Producer console

Open another terminal for the producer:-

docker-compose run producer

CONCLUSION:-

Always use Best Practices to write Dockerfiles.
Moreover instead of running a single container each time. We can create a docker-compose file to run multiple containers.