data-pipelines

Introduction to Apache Airflow and its Components

January 28, 2022January 31, 2022Apache Airflowapache airflow, data-pipelines

Reading Time: 3 minutes What is Apache Airflow ? Apache Airflow is a free and open-source application for managing complicated workflows and data processing pipelines. It’s a platform for automating and monitoring workflows for scheduled jobs. It allows us to configure and schedule our processes according to our needs while simplifying and streamlining the process. Why do we need Apache Airflow ? Lets us assume a use case where Continue Reading

Apache Airflow – A Workflow Manager

May 16, 2021May 16, 2021Apache Airflow, Big Data and Fast Data, Studio-DevOpsairflow, airflow-setup, apache, data-pipelines

Reading Time: 4 minutes As the industry is becoming more data driven, we need to look for a couple of solutions that would be able to process a large amount of data that is required. A workflow management system provides an infrastructure for the set-up, performance and monitoring of a defined sequence of tasks, arranged as a workflow application. Workflow management has become such a common need that most Continue Reading