pipeline

Apache Airflow: Writing your first pipeline

Reading Time: 3 minutes Before jumping into the code, you need to get used to what Airflow DAG is all about. it is important so stay with me, Airflow DAG? DAG stands for Directed Acyclic Graph. In simple terms, it is a graph with nodes, directed edges and no cycles. Basically, this is a DAG: We will learn step by step how to write your first DAG. Steps to Continue Reading

Karate DSL and Concourse – How to integrate?

Reading Time: 3 minutes Hi folks, In this short blog, we will see how to integrate Karate DSL having API tests with the concourse. Since CI/CD is a major thing in the software industry now, we need to have our tests integrated with the CI/CD pipeline that we are using in our respective projects. This blog is all about integrating Karate with Concourse CI. So, let’s begin. Setting up Continue Reading

Apache Beam: Side input Pattern

Reading Time: 3 minutes Apache Beam is a unified programming model for defining both batch and streaming data-parallel processing pipelines. It is a modern way of defining data processing pipelines. It has rich sources of APIs and mechanisms to solve complex use cases. In some use cases, while we define our data pipelines the requirement is, the pipeline should use some additional inputs. For example, In streaming analytics applications, it Continue Reading

What is CI/CD ? A FreeStyle Project Demo with Jenkins

Reading Time: 3 minutes What is CI/CD? CI /CD gave a new vision to how developers and testers produce software. In other words, the adoption of CI/CD is basically of continuous methodologies. They showcase an environment and operates around constantly integrating new code. Similar to that of the software development lifecycle. Above all starting from the waterfall, the development process changed to Agile and then to DevOps.In conclusion, CI Continue Reading

An Introduction to bitbucket pipeline

Reading Time: 4 minutes Bitbucket Pipelines is an integrated CI/CD service built into Bitbucket Cloud that’s integrated into the UI and sits alongside your repositories. It allows you to automatically build, test, and even deploy your code based on a configuration file in your repository. CI/CD tools are an integral part of a software team’s development cycle. Whether you’re using it to automate tests, a release process, or deployments to customers, Continue Reading

Using CodeShip with CodeDeploy

Reading Time: 5 minutes What is CodeShip ? CodeShip is a hosted continuous delivery service that focuses on speed, simplicity and reliability. It is a fast and secure hosted CI service that scales as per user’s needs. CodeShip was founded in 2011, and pretty soon it became one amongst the Saas Continuous Integration and Continuous Delivery leader in the market. CodeShip is now backed by a larger company with Continue Reading

jenkins logo

Playing with Jenkinsfile

Reading Time: 7 minutes Jenkins – Introduction Jenkins is an open-source automation server that is used to automate all sorts of tasks related to building, testing, and delivering, or even deploying software. CD pipelines refers to the process of getting the application / software from the source control and delivering it to end-users.Few of the features of Jenkins that make it the first choice for a user are : Continue Reading

A Quick Demo: Kafka to Flink to Cassandra

Reading Time: 3 minutes Hi Folks!! In this blog, we are going to learn how we can integrate Flink with Kafka and Cassandra to build a simple streaming data pipeline. Apache Flink is a framework and distributed processing engine. it is used for stateful computations over unbounded and bounded data streams.Kafka is a scalable, high performance, low latency platform. It allows reading and writing streams of data like a messaging system.Cassandra: A distributed and wide-column Continue Reading

Running Apache Airflow DAG with Docker

Reading Time: 3 minutes In this blog, we are going to run the sample dynamic DAG using docker. Before that, let’s get a quick idea about the airflow and some of its terms. What is Airflow? Airflow is a workflow engine which is responsible for managing and scheduling running jobs and data pipelines. It ensures that the jobs are ordered correctly based on dependencies and also manages the allocation Continue Reading

Simple CI/CD Pipeline Using Jenkins

Reading Time: 5 minutes In this blog we gone a see creating a CI/CD Pipeline using Jenkins. Some pre-requisites to create a CI/CD pipeline using Jenkins are. Open-JDK-8 Jenkins installed on our system At the very first we see a little bit introduction of Jenkins and why we need Jenkins Pipeline. Jenkins Jenkins is an open-source automation tool to create a continuous integration pipeline. It is used to building Continue Reading

jenkins logo

Glimpse of Pipeline in Jenkins

Reading Time: 3 minutes Hi readers, In this blog, we will be creating the Sample Jenkins pipeline also we will be looking at basic syntax of the pipeline. Jenkins Pipeline is a suite of plugins that supports implementing and integrating continuous delivery pipelines into Jenkins. A continuous delivery (CD) pipeline is an automated expression of your process for getting software from version control right through to your users and customers. Let’s get started Continue Reading

Defining your workflow: Why Not Airflow?

Reading Time: 4 minutes What is Apache Airflow? Airflow is a platform to programmatically author, schedule & monitor workflows or data pipelines. These functions achieved with Directed Acyclic Graphs (DAG) of the tasks. It is an open-source and still in the incubator stage. It was initialized in 2014 under the umbrella of Airbnb since then it got an excellent reputation with approximately 800 contributors on GitHub and 13000 stars. Continue Reading