Reading Time: 4 minutes Codepipeline is one of the AWS’s developer tools that acts as the glue and processor for other AWS services. It uses code commit or another external source control to pull in source code and trigger a pipeline execution. CodeBuild can run code, build software, and do basically anything else you need to do in your pipeline. At any point in your pipeline, you can trigger Continue Reading
Reading Time: 5 minutes Amazon Elastic Cloud Compute is a service provided by amazon for performing computing operations by the users. We often create an ec2 instance for various requirements. Most of which are solved by accessing the instance via ssh. This enables you to get a CLI mode of interaction with your instance. It is suitable for most of your workloads, but sometimes it becomes necessary to have Continue Reading
Reading Time: 6 minutes Snowball Edge is a physical, shippable and self-contained data storage and compute appliance. It enables organisations to move large volumes of data in and out of AWS cloud. Snowball edge can be used as a standalone device or clustered together to provide data mobility at petabyte scale. Each device includes onboard compute capabilities that enable data collection, data manipulation and data preparation activities. Data is Continue Reading
Reading Time: 4 minutes In my previous blog, I have discussed Airflow – A workflow Manager. In this blog, we will write a DAG for Airflow that would define a workflow of tasks and their dependencies. Before writing a DAG file, we will first look into the operators that can be used while writing a DAG. Airflow Operators An operator represents a single, ideally idempotent, task. Operators determine what actually Continue Reading
Reading Time: 4 minutes As the industry is becoming more data driven, we need to look for a couple of solutions that would be able to process a large amount of data that is required. A workflow management system provides an infrastructure for the set-up, performance and monitoring of a defined sequence of tasks, arranged as a workflow application. Workflow management has become such a common need that most Continue Reading
Reading Time: 3 minutes Traefik is an open source API gateway, written in Golang and was developed in a Unix-centric way. It is designed to simplify the complexity regarding microservices operations. Traefik performs auto-configuration of services, which means that the developer only needs to worry about developing and deploying applications. Traefik auto-configures with sensible defaults and sends a request to the said service. With changing requirements and needs of Continue Reading
Reading Time: 4 minutes Microservices is a software architecture style that advocates for many granular services. Each of these services perform a single business function. Each microservice is a self-contained, independently deployable piece of a larger application. It interacts with other components, typically via well-defined REST APIs. Clusters is an important concept that has been introduced to implement microservices in a more efficient way. Adopting a microservices architecture consisting Continue Reading
Reading Time: 3 minutes Kubernetes is an open source platform for automating deployments, scaling, and operations of application containers across clusters of hosts. It provides a platform for container-centric infrastructure. It addresses all the challenges that were described above. Among its key features are: Automated deployment and replication of containers Online scale-in and scale-out of container clusters Load balancing over groups of containers Rolling upgrades of application containers Resiliency, Continue Reading
Reading Time: 4 minutes Starting with Kubernetes version 1.6.0, cluster security was ramped up considerably. In earlier versions, if you managed to acquire the authentication token from one of the pods, you could use it to do anything you want in the cluster. RBAC in kubernetes prevents unauthorized users from viewing or modifying the cluster state. The default ServiceAccount isn’t allowed to view cluster state, let alone modify it Continue Reading
