Way To Ship Kafka Logs To Kibana Dashboard

Table of contents

Reading Time: 2 minutes

What are Logs?

Logs are well-explanatory records of events regarding the application, its performance, and user activities. The events can include deleting, or modifying a file for the application. They also include system configuration changes.

Whereas, Log files stores records about these events occurring. Since they are used for tracking information about computing events. They come in handy when it comes to identifying the problems and correcting them.

How to Ship Kafka Logs to the Kibana Dashboard?

To ship the Kafka logs, we will be using the filebeat agent. A filebeat agent is a lightweight shipper whose purpose is to forward and centralize the log data.

For filebeat to work, you need to install it as an agent on the desired servers. Filebeat then monitors the log files, collects the log events, and forwards them to the ElasticSearch or LogStash for indexing.

To install the filebeat agent to servers, we will use ansible since it makes the deployment of the agent effortless.

How Filebeat agent Works

When you start the filebeat agent it starts its input and looks into the log location you have mentioned. For every log the filebeat locates, a harvester is started. Moreover every single harvester reads a single log for new content and sends the new log data to libbeat. Libbeat is a go framework used for data forwarding. Libbeat aggregates the events and sends the collected data to the output that you’ve configured for Filebeat. In our case it is Kibana.

Ansible playbook for deploying the Filebeat agent to Kafka servers

- name: Run kafka logging role to deploy filebeat to kafka servers 
  hosts: <kafka_server_hosts>
  gather_facts: False 
  roles:
    - deploy-filebeat
  become: True
  become_method: sudo

Role to run the necessary tasks for deploying Filebeat

Before you run the role make sure you have the filebeat.yml and kafka.yml files which consist of the configuration changes needed for the role and also the set-logstash-endpoint.sh script which sets the endpoints needed for the logstash. You can have both these files stored in the files directory of the role deploy-filebeat.

- name:  Install filebeat 6.2.4
  shell: |
    curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-amd64.deb
    sudo dpkg -i filebeat-6.2.4-amd64.deb


- name: Upload filebeat.yml
  copy: 
    src: filebeat.yml
    dest: /etc/filebeat 
    owner: root
    group: root 
    mode: 644
  
- name: Install filebeat Kafka module
  shell: |
    filebeat modules enable kafka

- name: Upload set-logstash-endpoint.sh script
  copy: 
    src: set-logstash-endpoint.sh
    dest: /usr/local/bin/
    owner: root
    group: root
    mode: 755
  
- name: Upload filebeat Kafka module configuration
  copy: 
    src: kafka.yml
    dest: /etc/filebeat/modules.d/
    owner=root
    group=root
    mode=644
  notify:
    - restart filebeat

- name:  Set the logstash endpoint for this environment
  shell: |
    /usr/local/bin/set-logstash-endpoint.sh

- name:  Start filebeat
  shell: |
    sudo /etc/init.d/filebeat start

After running this playbook you can go and check on your kibana dashboard endpoint as the kafka logs from the kafka servers will start popping up.

If running locally you can go to – http://localhost:5601/app/logs/ and check the Kafka logs that are shipped here.

logs