AWS

Stream your data using Amazon Kinesis Firehose

Reading Time: 4 minutes We live in a world driven by data and every second we are processing a large amount of data, using it, analyzing it, and transforming it. Data is very essential for businesses these days. Therefore need for handling the Dynamically generating data is important. As the number, variety, and velocity of data sources grow, new architectures and technologies are needed. Technologies like Amazon Kinesis are Continue Reading

How to generate AWS Signature with Postman

Reading Time: 4 minutes Hello folks, I hope you are having a productive day in this pandemic of COVID-19. Let’s move on to our next blog in the series of API automation. Most of us are doing automation using the tool Postman. So while performing automation with postman we have to integrate many other tools and APIs with Postman. Similarly to use the AWS APIs we need to create Continue Reading

KSnow: Know about Cloning in Snowflake

Reading Time: 2 minutes This blog pertains to Cloning feature in Snowflake, and I will explain you all the things you need to know about these features with practical example. So let’s get started. Zero Copy Clone Cloning also Snowflake as Zero Copy Clone in Snowflake. It used to create a copy of a Table or Schema or a Database. In most database, in order to make a copy Continue Reading

KSnow: Time Travel and Fail-safe in Snowflake

Reading Time: 5 minutes This blog pertains to Time Travel and Fail-safe in Snowflake, and I will explain you all the things you need to know about these features with practical example. So let’s get started. Introduction to Time Travel Snowflake allows accessing historical data of a point in the past that may have been modified or deleted at the current time. Using time travel functionality a number of Continue Reading

KSnow: Load continuous data into Snowflake using Snowpipe

Reading Time: 5 minutes In this blog, we will discuss loading streaming data into Snowflake table using Snowpipe. But before that, if you haven’t read the previous part of this blog i.e., Loading Bulk Data into Snowflake then I would suggest you go through it. As now we have been set so let’s get started and see what Snowpipe is all about. Introduction Snowpipe is a mechanism provided by Continue Reading

KSnow: Loading Data Into Snowflake

Reading Time: 5 minutes This blog pertains to Loading Data into Snowflake, and I will explain you about the various step involved in this process. So let’s get started. Before moving ahead, you can visit the blog on understanding the basic of Snowflake Data Warehouse in case you want to refresh your concepts. Now let’s talk about the actual topic for which you have click on this blog. To Continue Reading

Automate deployment using AWS CodeDeploy

Reading Time: 6 minutes In the CodeDeploy blog series, we are going to write two blogs the first blog covers the CodeDeploy theory-based and In the second blog, we will cover the full end-to-end automation practical of the application deployment using CodeDeploy and Jenkins. Let’s start aws CodeDeploy is basically a deployment service through which we can easily automate your deployment.

Apache Spark: Handle Corrupt/Bad Records

Reading Time: 3 minutes Most of the time writing ETL jobs becomes very expensive when it comes to handling corrupt records. And in such cases, ETL pipelines need a good solution to handle corrupted records. Because, larger the ETL pipeline is, the more complex it becomes to handle such bad records in between. Corrupt data includes: Missing information Incomplete information Schema mismatch Differing formats or data types Apache Spark: Continue Reading

Amazon EMR

Reading Time: 3 minutes Businesses worldwide are discovering the power of new big data processing and analytics frameworks like Apache Hadoop and Apache Spark, but they are also discovering some of the challenges of operating these technologies in on-premises data lake environments. They may also have concerns about the future of their current distribution vendor. Common problems of on-premises big data environments include a lack of agility, excessive costs, Continue Reading

Setting up the IAM permission for AWS Lambda

Reading Time: 3 minutes About AWS AWS Lambda is a compute service that lets you run code without managing servers. AWS Lambda executes your code only when needed and scales automatically. You pay only for the compute time you consume – there is no charge when your code is not running. Why IAM? Before we call Lambda function we need to get correct permission setup to access the function. Continue Reading

Getting started with Amazon SNS

Reading Time: 2 minutes Introduction The Simple Notification Service (SNS) is used as a publish and subscribe messaging service. But what does it mean? SNS is centered around topics and you can think of a topic as a group for collecting messages. Users or endpoints can then subscribe to this topic and messages or events are then published to that topic. When a message is published, all subscribers to Continue Reading