What is the ELK Stack?

Table of contents

Reading Time: 3 minutes

ELK Stack or more recently called Elastic Stack, is a combination of three open source projects – Elasticsearch, Logstash and Kibana – all developed by Elastic and used for storing and analyzing logs. Even though these are three separate products, they compliment each other to the extend that they have come to be recognised as one.

Why Log Analysis?

Before getting to know more about the ELK stack, we must have an idea about why we need to do log analysis.

In cloud-based environment infrastructures, performance, and isolation is very important. The performance of virtual machines in the cloud may vary based on the specific loads, environments, and number of active users in the system. Therefore, reliability and node failure can become a significant issue.

Log management platform can monitor all above-given issues as well as process operating system logs, NGINX, IIS server log for web traffic analysis, application logs, and logs on AWS (Amazon web services).

Log management helps DevOps engineers, system admin to make better business decisions. Hence, log analysis via Elastic Stack or similar tools is important.

ELK Stack Architecture

As we can see in this architecture, Logstash collects the logs. It even parses and transforms the data. The data that is transformed by Logstash is stored, searched, and indexed in Elasticsearch. Then we use Kibana to visualize and explore this data indexed in Elasticsearch.

However, there is one more component – Beats – which collects the data and sends it to Logstash. This led Elastic to rename ELK as the Elastic Stack.

Further, if we are dealing with very large data, we could provide buffering mechanism using Kafka, RabbitMQ etc to send data from Beats to Logstash.

Let us now get more acquainted with these open source products:

Elasticsearch

Elasticsearch is a NoSQL database. It is based on Lucene search engine, and it is built with RESTful APIS. Elasticsearch offers simple deployment, maximum reliability, and easy management. It also offers advanced queries to perform detail analysis and stores all the data centrally. It is helpful for executing a quick search of the documents.

Elasticsearch also allows you to store, search and analyze big volume of data. Modern web and mobile applications have adopted it in search engine platforms. Apart from a quick search, the tool also offers complex analytics and many advanced features.

Logstash

Logstash is the data collection pipeline tool. It collects data inputs and feeds into the Elasticsearch. It gathers all types of data from the different source and makes it available for further use.

Logstash can unify data from disparate sources and normalize the data into your desired destinations. It allows you to cleanse and democratize all your data for analytics and visualization of use cases.

It consists of three components:

Input: passing logs to process them into machine understandable format
Filters: It is a set of conditions to perform a particular action or event
Output: Decision maker for processed event or log

Kibana

Kibana is a data visualization which completes the ELK stack. This tool is used for visualizing the Elasticsearch documents and helps developers to have a quick insight into it. Kibana dashboard offers various interactive diagrams, geospatial data, and graphs to visualize complex quires.

It is used to search, view, and interact with data stored in Elasticsearch directories and helps you to perform advanced data analysis and visualize your data in a variety of tables, charts, and maps.

Case Made by ELK Stack

Advantages

ELK works best when logs from various Apps of an enterprise converge into a single ELK instance
It provides amazing insights for this single instance and also eliminates the need to log into hundred different log data sources
Rapid on-premise installation
Easy to deploy Scales vertically and horizontally
Elastic offers a host of language clients which includes Ruby. Python. PHP, Perl, .NET, Java, and JavaScript, and more
Availability of libraries for different programming and scripting languages

Disadvantages

Different components in the stack can become difficult to handle when you move on to complex setup
There’s nothing like trial and error. Thus, the more you do, the more you learn along the way

Conclusion

ELK seem to be a one stop solution for centralized logging issues. Being open-sourced it has great community support and its use in production by companies like Netflix, LinkdIn, Medium speaks volumes of its performance in production environments.