How to send logs to AWS OpenSearch using Logstash

Table of contents

Reading Time: 3 minutes

Hello Readers!!! We are again back with a new blog. In this blog, we will see how we can send logs to AWS OpenSearch using Logstash. Basically, logstash is an ingestion tool that we can use for sending the data to our desired destination from any source that we want. And We will use AWS OpenSearch for analyzing the ingest data coming from logstash.

Let’s start!!

Step 1: Create an AWS OpenSearch service domain. Follow the following blog for creating a service domain according to your requirement:

This is my OpenSearch domain which I will use for visualizing my logs.

Here we will find OpenSearch dashboard URL and domain endpoint.

Step 2: I have created an EC2 Linux machine on which I will install the logstash.

Step 3: Install logstash by using the following commands.

$ wget https://artifacts.opensearch.org/logstash/logstash-oss-with-opensearch-output-plugin-8.4.0-linux-x64.tar.gz

$ tar -zxvf logstash-oss-with-opensearch-output-plugin-8.4.0-linux-x64.tar.gz

$ cd logstash-8.4.0

Step4: Install logstash output OpenSearch plugin by using the following command:

$ sudo bin/logstash install logstash-output-opensearch

My plugin is installed successfully.

Step 5: I will create a sample log file and name it sample.log. You can place any type of log inside this file.

Step 6: Create a file inside it for connecting AWS OpenSearch to logstash. Place the following configuration inside this file.

input {

    file {

        path => "/root/logstash-8.4.0/sample.log"

        start_position => "beginning"

        sincedb_path => "/dev/null/"

    }

}

output {

    opensearch {

        ecs_compatibility => disabled

        index => "<index_name>"

        hosts => "https://domain-endpoint:443"

        auth_type => {

            type => 'basic'

            service_name => 'aoss'

            user => '<opensearch_username>'

            password => '<opensearch_password>'

        }

        legacy_template => false

        default_server_major_version => 2

    }

}

Step 7: Now run logstash using following command:

$ ./bin/logstash -f <path/to/your/config/file>

Step 8: Let’s open our AWS OpenSearch dashboard. I will use my username and password for logging into it.

Here we will now get our log on the AWS OpenSearch dashboard as you can see below.

We are all done now!!

Conclusion:

In this blog, we have learned how we can send logs to AWS OpenSearch using Logstash. This is really very quick and simple. Thank you for sticking to the end. If you like this blog, please share my blog and show your appreciation by giving thumbs-ups, and don’t forget to give me suggestions on how I can improve my future blogs that can suit your needs.

HAPPY LEARNING!