- consumer group
Producer:Producer is an application that sends the messages to the kafka broker. A message is something that has low to medium size.
Kafka Broker:Kafka server acts as kafka broker. It acts as a message broker and handles the request sent by the producer.
Consumer :Consumer is the application that receives the message through kafka broker.
It reads the messages send by the producer through kafka broker.
Cluster: The group of kafka brokers or servers when working together to tackle the single task or request is called kafka cluster. Usually more than one system are involved.
Consumer group: When there are multiple prodcuers are sending the messages, which in turn are handled by the cluster (group of brokers) and only one cosumer is receiving the messages, then it becomes very difficult for a single consumer to receive many messages. To avoid this issue consumer group comes into the picture, in which multiple consumers are handling the same messages fairly distributed. So it makes the process more efficient. Once can even distribute only one producer to only one consumer, but one can not have more than two consumers on each producer so as to avaoid name collision.
Topic: When working with multiple producers and multiple consumers , there are times when certain properties of messages or server are identical, or there may be multiple names that are same for the different servers or other data may be occuring at more than one place so to avoid these issues, we have a concept of topic. A topic can only have a single name. It means you can not have two topics with the same name.
Offsets:The sequence ID that has been assigned to the messages in a topic (maybe stored in multiple partitions) is called offset of that message.
Partition:There are times when data stream becomes so much heavy that it can not be stored on a single system or we want to process messages faster, so to acheive that we use a concept called partition, in which we divide the load and distribute it over to other systems. Kafka broker does not choose the number of partitions , it solely depends on the programmer how he wants to handle the load, and in turn creates the partions accordingly. Remember that the messages are stored in a sequence.