Author: Bhavya Garg

The architecture of Kafka Connect

Reading Time: 3 minutes Kafka Connect Kafka Connect is an open-source component and framework to get it connected with external systems, including the databases. There are connectors that help to move huge data sets into and out of the Kafka system. Kafka Connect is only used to copy the streamed data, thus its scope is not broad. It executes as an independent process for testing and a distributed, scalable Continue Reading

A Quick Insight of Kafka Connect/Connector

Reading Time: 3 minutes Overview In this blog, we are going to discuss Kafka connect/connectors in detail. If you want a basic introduction to Kafka Connect then you can refer to this blog. Now we will understand concepts such as Kafka Connect, Kafka connectors, and Kafka Convertors. Kafka Connect Kafka Connect is a framework to stream data into and out of Apache Kafka®. The Confluent Platform ships with several built-in Continue Reading

Dynamic DAGs in Apache Airflow

Reading Time: 4 minutes Airflow dynamic DAGs can save you a ton of time. As you know, Apache Airflow is written in Python, and DAGs are created via Python scripts. That makes it very flexible and powerful (even complex sometimes). By leveraging Python, you can create DAGs dynamically based on variables, connections, a typical pattern, etc. This very nice way of generating DAGs comes at the price of higher Continue Reading

Creating DAG in Apache Airflow

Reading Time: 5 minutes In my previous blog, I have discussed about the Introduction to the Apache Airflow. In this blog, we will learn how to create a DAG for Airflow that would define a workflow of tasks and their dependencies.  What is DAG? First of all the question that comes to our mind is that what is this DAG .So in Airflow, a DAG – or a Directed Acyclic Graph – Continue Reading

Introduction to Apache Airflow

Reading Time: 4 minutes What is Apache Airflow? Airflow is a platform to programmatically author, schedule and monitor workflows.These functions achieved with Directed Acyclic Graphs (DAG) of the tasks. It is an open-source and still in the incubator stage. It was initialized in 2014 under the umbrella of Airbnb since then it got an excellent reputation with approximately 800 contributors on GitHub and 13000 stars. The main functions of Apache Airflow is to schedule workflow, monitor Continue Reading

Deep Dive into Object Oriented Programming Concept

Reading Time: 4 minutes OOP Concept Object-Oriented Programming (OOP) concept is a programming methodology that uses objects and classes and is also used to implement real world scenarios. Most important concepts of OOP are as follows:- 1.Class :- A class is a user defined blueprint or prototype from which objects are created. It also represents the set of properties or methods that are common to all objects of one Continue Reading