Search Results for: kubernetes

Introduction to Apache Airflow

Reading Time: 4 minutes What is Apache Airflow? Apache Airflow is a workflow management system which is used to programmatically author, schedule and monitor workflows. Airflow is also known as DAG. Airflow allows users to create workflows with high granularity and track the progress as they execute. They make it easy to do potentially large data operations. For Example: If you want to run an SQL query every day, Continue Reading

A Beginner’s Guide to Deploying a Lagom Microservice on Kubernetes

Reading Time: 4 minutes Both Lagom and Kubernetes are gaining popularity quite fast. Lagom is an open source framework for building reactive microservice systems in Java/Scala. And, Kubernetes (or K8s in short) is an open-source system for automating deployment, scaling, and management of containerized applications. Together they make an excellent stack for developing Reactive microservices of production grade. We have already seen a lot of blogs on Lagom on this Continue Reading

How to connect HashiCorp Consul Connect and Service Mesh

Reading Time: 4 minutes Hello Readers, In this blog, we’ll learn about Consul Service Mesh and why Consul Connect and how it works, and the features provided by HashiCorp. What is HashiCorp? HashiCorp is a software company with a freemium business model based in San Francisco, California. HashiCorp provides open-source tools and commercial products that enable developers, operators, and security professionals to provision, secure, run and connect cloud-computing infrastructure.  HashiCorp open-source Continue Reading

Introduction to Apache Airflow

Reading Time: 4 minutes What is Apache Airflow? Airflow is a platform to programmatically author, schedule and monitor workflows.These functions achieved with Directed Acyclic Graphs (DAG) of the tasks. It is an open-source and still in the incubator stage. It was initialized in 2014 under the umbrella of Airbnb since then it got an excellent reputation with approximately 800 contributors on GitHub and 13000 stars. The main functions of Apache Airflow is to schedule workflow, monitor Continue Reading

Apache Pulsar: A Quick Overview

Reading Time: 3 minutes What is Apache Pulsar? Yahoo developed Pulsar and it is now open source under the Apache License. Apache Pulsar is a distributed messaging system that is based on the publisher and subscriber model, and unlike other pub-sub models, apache pulsar decouples producers from consumers. Pulsar is the middleware that accepts information from producers and consumers then source that data from the pulsar. Why Apache Pulsar? Continue Reading

Akka Cluster in use (Part 7): Manually Healing an Akka Cluster

Reading Time: 4 minutes In our previous blog post, Understanding Failures in Akka Cluster, we came to know how an Akka Cluster reacts in case of a failure and what are the reasons behind the failures. Now, whenever a failure will occur in an Akka Cluster, we would need a way to heal the cluster, so that we can restore it back to its normal working condition. Hence, in Continue Reading

Linking Logs to Grafana Panels using Loki

Reading Time: 4 minutes Hi folks, in this blog, we will be discussing what challenges we face when we want to view logs with respect to metrics and how we can achieve this using Grafana, Loki, and Promtail. Problem Statement When using monitoring solutions like Grafana, Prometheus, Elatic Stack etc for our infrastructure, we have metrics and logs decoupled to an extent that it is very difficult to look Continue Reading

Airflow on Google Cloud Composer

Reading Time: 4 minutes If you are wondering how to start working with Apache Airflow for small development or academic purposes here you will learn how to. Well deploying Airflow on GCP Compute Engine (self-managed deployment) could cost less than you think with all the advantages of using its services like BigQuery or Dataflow. Table of Content What is apache airflow cloud composer overview Google cloud composer benefit Composer Continue Reading

Apache Beam Vs Apache Spark

Reading Time: 4 minutes Before going through the comparison of  Apache Beam and Apache Spark, we should have a glimpse of what these two exactly are. Apache Beam means a unified programming model. It implements batch and streaming data processing jobs that run on any execution engine. It executes pipelines in multiple execution environments. Apache Spark defines as a fast and general engine for large-scale data processing. Spark is a fast Continue Reading

How To Deploy Fargate Using AWS CDK

Reading Time: 5 minutes Introduction What is AWS Fargate ? In this blog we’ll see How To Deploy Fargate Using AWS CDK. As per the aws official documentation ” AWS Fargate is a serverless, pay-as-you-go compute engine that lets you focus on building applications without managing servers. “ In simple words Fargate allows you to run container on ECS In aws without managing the servers.Using this makes it easier to Continue Reading

Deploy a Microservice into Istio service mesh

Reading Time: 3 minutes Service mesh is a dedicated infrastructure layer for handling service to service communication.
Basically, it’s a way to control how different micro services deployed on kubernetes will manage secure communication and traffic between them with lot’s of cross-cutting concerns like logging, security, etc.

Apache Kafka for beginners

Reading Time: 4 minutes Introduction One of the biggest challenges associated with big data is, analyzing the data. But before we get to that part, the data has to be first collected, and also for a system to process impeccably it should be able to grasp and make the data available to users. This is where Apache Kafka comes in handy. Let’s briefly understand how Kafka came into existence? Continue Reading

Core Concepts of Apache Airflow

Reading Time: 4 minutes In this blog we will go over the core concepts basic you must understand if you want to use Apache airflow. In this article, you will learn: What is Airflow Architecture Overview Dag Task Operator Dag Run Execution Date Airflow Airflow was started in October 2014 and developed by Maxime Beauchemin at Airbnb. It is a platform for programmatically authoring, scheduling, and monitoring workflows. It Continue Reading