elasticsearch

background

How to send Github commits and PR logs to ElasticSearch using a custom script

Reading Time: 6 minutes Hello Readers!! In this blog, we will see how we can send GitHub commits and PR logs to Elasticsearch using a custom script. Here we will use a bash script that will send GitHub logs to elasticsearch. It will create an index in elasticsearch and push there the logs. After sending logs to elasticsearch we can visualize the following github events in kibana:- 1. GitHub Continue Reading

protecting sensitive data in docker

How to run Filebeat in a Docker container

Reading Time: 4 minutes Introduction Hi everyone! Today in this blog we are going to learn how to run Filebeat in a container environment. For a quick understanding – Set-up In this setup, I have an ubuntu host machine running Elasticsearch and Kibana as docker containers. I will bind the Elasticsearch and Kibana ports to my host machine so that my Filebeat container can reach both Elasticsearch and Kibana. Continue Reading

How to Backup and Restore Elasticsearch using Snapshots

Reading Time: 3 minutes Introduction Hello everyone! Today in this blog, we will learn how to backup and restore Elasticsearch using snapshots. Before diving in, let’s first brush up on the basics of the topic. Elasticsearch at a glance It is a search and analytics engine It is based on NoSQL technology It exposes REST API instead of CLI to perform various operations It is a combination of different Continue Reading

Kafka Connect example: Mysql to Elastic Search

Reading Time: 3 minutes Overview: Hello everyone, in this blog, we will see an example of Kafka connect in which we will take a MySQL table, stream it to a Kafka topic, and from there load it to Elasticsearch and index its content. Installation: Now first of all we will install MySQL and Elastic search to our local system. For installing simply run: The next step is to make Continue Reading

How to set up a 2 node Elasticsearch cluster on Kubernetes.

Reading Time: 6 minutes In this blog, we will learn to set up Elasticsearch on the minikube cluster but before that let’s look at elasticsearch. Elasticsearch is a distributed, scalable, real-time search engine that supports full-text and structured searches, and analytics. It’s most typically used to index and search vast amounts of log data, but it can also be used to explore a variety of documents. Use Cases of Continue Reading

What is the ELK Stack?

Reading Time: 3 minutes ELK Stack or more recently called Elastic Stack, is a combination of three open source projects – Elasticsearch, Logstash and Kibana – all developed by Elastic and used for storing and analyzing logs. Even though these are three separate products, they compliment each other to the extend that they have come to be recognised as one. Why Log Analysis? Before getting to know more about Continue Reading

Knolx: Alpakka-Connecting Kafka & ElasticSearch to Akka Streams

Reading Time: < 1 minute Hi all, Knoldus has organized a 30 min session on 1st  March 2019 at 3:30 PM. The topic was Alpakka – Connecting Kafka and ElasticSearch to Akka Streams.  Many people have joined and enjoyed the session. I am going to share the slides here. Please let me know if you have any question related to linked slides or video. The slides of the KnolX are here: And Continue Reading

A Basic CRUD Application Using ElasticSearch

Reading Time: 2 minutes In our previous post Introduction To ElasticSearch, we talked about the basic terminology of elastic search and basic requests to create or delete an index, check the health status of the cluster, indices etc. In this post, we are going to talk about creating a basic CRUD application using DSL. Without waiting let’s get started. To create an application, first, create an SBT Project, add Continue Reading

Introduction to ElasticSearch

Reading Time: 3 minutes Hey Folks, Today, we are going to explore about basics of ElasticSearch. From the documentation,  its definition is: Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time.   It is a real-time distributed search and analytics engine based on top of Apache Lucene. It is Continue Reading

Alpakka – Connecting Kafka and ElasticSearch to Akka streams

Reading Time: 4 minutes In our previous blog, we had a look at what Akka streams are and how they are different from the other streaming mechanisms we have. In this blog, we will be taking a little step forward into the world of Akka Streams. In order to work with Akka streams, we need a mechanism to connect Akka Streams to the existing system components. That is where Alpakka Continue Reading

Spark Structured Streaming with Elasticsearch

Reading Time: 3 minutes There’s been a lot of time we have been working on streaming data. Using Apache Spark for that can be much convenient. Spark provides two APIs for streaming data one is Spark Streaming which is a separate library provided by Spark. Another one is Structured Streaming which is built upon the Spark-SQL library. We will discuss the trade-offs and differences between these two libraries in Continue Reading

Amazon ES – Secure your cluster from anonymous users! #2

Reading Time: 5 minutes In the previous blog, we have learned how to create a domain on Amazon ES and also how to create an index using Curl on the cluster. Now, let’s just look how we can control access to Amazon ES Domain. One of the key benefits of using Amazon ES is that you can utilize AWS Identity and Access Management (IAM) to control access to your Continue Reading