Big Data and Fast Data

Do you really need Spark? Think Again!

With the massive amount of increase in big data technologies today, it is becoming very important to use the right tool for every process. The process can be anything like Data ingestion, Data processing, Data retrieval, Data Storage, etc. Today we are going to focus on one of those popular big data technologies i.e., Apache Spark. Apache Spark is an open-source distributed general-purpose cluster-computing framework. Spark Continue Reading

Build your own Kafka Producer

“It’s Not Whether You Get Knocked Down, It’s Whether You Get Up.” – Inspirational Quote By Vince Lombardi Kafka Producer API allows applications to send streams of data to topics in the Kafka cluster. Looking for a way to implement Custom Kafka Producer in your project. This blog post gives you an end to end solution to implement this functionality using KAFKA API. Introduction There Continue Reading

Protein Structure determination aided by Stochastic Search (Replica Exchange Monte-Carlo Method)

Introduction Proteins are large molecules, which occur in abundance in every single living organism. They carry out vital functions such as transporting oxygen, converting the food you eat into energy your body can use, and many more. Proteins are long chains of linked units called amino acids. There are 20 types of amino acids. Proteins fold into different shapes depending upon their sequence of amino Continue Reading

Monitoring Kafka with Prometheus and Grafana

Kafka monitoring is an operation which is used for the optimization of the Kafka deployment. This process is easy and efficient, by applying one of the existing monitoring solutions instead of building your own. Let’s say, we use Apache Kafka for message transfer and processing and we want to monitor it.But, before learning the steps for monitoring, let’s first understand the prerequisites. Kafka It is Continue Reading

Getting Started with Akka Remoting

When we start with Akka we generally start with one actor system on our local machine but when we talk about the business application we can have multiple parts of an application and those part can run on different machine or node. Akka Remoting is a communication module to connect the actor system in peer to peer fashion. It also serves as the foundation for Continue Reading

Flinkathon: Guide to setting up a Local Flink Custer

In our previous blog post, Flinkathon: First Step towards Flink’s DataStream API, we created our first streaming application using Apache Flink. It was easy, clean, and concise. However, the real power of Apache Flink is seen on a cluster, where data is processed in a distributed manner, with the advantage of multi-core/multi-memory systems. So, in this blog post, we will see how to set up Continue Reading

Determine Kafka broker health using Kafka stream application’s JMX metrics and setup Grafana alert

As we all know, Kafka exposes the JMX metrics whether it is Kafka broker, connectors or Kafka applications. A few days ago, I got the scenario where I needed to determine Kafka broker health with the help of Kafka stream application’s JMX metrics. It looks bit awkward, right? I should use the broker’s JMX metrics to do this, why am I looking to application JMX Continue Reading

HAWK-Rust Series: Automate Infrastructure using Terraform

HAWK is a Rust based Image Recognition project, which implements a two-factor authentication by using the RFID card for user identification and Image for user validation. In this project, we have used AWS services and the whole AWS infrastructure required by this project is automated using Terraform (a tool for building, changing, and versioning infrastructure safely and efficiently).

Need the Grafana slack monitoring alert? Let’s create it now.

In my previous blog, we created the first Grafana dashboard. Now in this blog, we will create our first monitoring Grafana alert. Before creating monitoring alerts, we need to add the notification channel through which we will be notified. Here, we will be adding slack as the notification channel. Slack notification channel: First, open the URL http://localhost:3000 in the browser and log in to the Continue Reading

Let’s create your first Grafana dashboard

In my previous blog, we discussed the setup of Grafana-Graphite for JMX monitoring.  Now we will create a first Grafana dashboard where we will create Grafana queries to visualize JMX metrics stored in Graphite. As we know, Grafana UI runs on http://localhost:3000/ by default so let’s open the URL in the browser with the default username and password which is admin: admin After login either Continue Reading

Knoldus Pune Careers - Hiring Freshers

Get a head start on your career at Knoldus. Join us!