How to send logs to AWS OpenSearch using Logstash

aerial view of clouds
Reading Time: 3 minutes

Hello Readers!!! We are again back with a new blog. In this blog, we will see how we can send logs to AWS OpenSearch using Logstash. Basically, logstash is an ingestion tool that we can use for sending the data to our desired destination from any source that we want. And We will use AWS OpenSearch for analyzing the ingest data coming from logstash.

Let’s start!!

Step 1: Create an AWS OpenSearch service domain. Follow the following blog for creating a service domain according to your requirement:

This is my OpenSearch domain which I will use for visualizing my logs.

AWS OPENSEARCH

Here we will find OpenSearch dashboard URL and domain endpoint.

AWS OS

Step 2: I have created an EC2 Linux machine on which I will install the logstash.

ec2

Step 3: Install logstash by using the following commands.

$ wget https://artifacts.opensearch.org/logstash/logstash-oss-with-opensearch-output-plugin-8.4.0-linux-x64.tar.gz
$ tar -zxvf logstash-oss-with-opensearch-output-plugin-8.4.0-linux-x64.tar.gz
$ cd logstash-8.4.0

Step4: Install logstash output OpenSearch plugin by using the following command:

$ sudo bin/logstash install logstash-output-opensearch
plugin

My plugin is installed successfully.

Step 5: I will create a sample log file and name it sample.log. You can place any type of log inside this file.

log file

Step 6: Create a file inside it for connecting AWS OpenSearch to logstash. Place the following configuration inside this file.

input {

    file {

        path => "/root/logstash-8.4.0/sample.log"

        start_position => "beginning"

        sincedb_path => "/dev/null/"

    }

}

output {

    opensearch {

        ecs_compatibility => disabled

        index => "<index_name>"

        hosts => "https://domain-endpoint:443"

        auth_type => {

            type => 'basic'

            service_name => 'aoss'

            user => '<opensearch_username>'

            password => '<opensearch_password>'

        }

        legacy_template => false

        default_server_major_version => 2

    }

}
log conf

Step 7: Now run logstash using following command:

$ ./bin/logstash -f <path/to/your/config/file>

Step 8: Let’s open our AWS OpenSearch dashboard. I will use my username and password for logging into it.

OS dashboard

Here we will now get our log on the AWS OpenSearch dashboard as you can see below.

OS dashboard

We are all done now!!

Conclusion:

In this blog, we have learned how we can send logs to AWS OpenSearch using Logstash. This is really very quick and simple. Thank you for sticking to the end. If you like this blog, please share my blog and show your appreciation by giving thumbs-ups, and don’t forget to give me suggestions on how I can improve my future blogs that can suit your needs.

HAPPY LEARNING! 

Written by 

Naincy Kumari is a DevOps Consultant at Knoldus Inc. She is always ready to learn new technologies and tools. She loves painting and dancing.

Leave a Reply