Reading Time: 3 minutes Introduction As we all know optimization plays an important role in the success of spark SQL. Therefore, a lot of work has been done in this direction. Before spark 3.0, cost-based optimization was a major hit in which different stages related to cost (based on time efficiency and estimated CPU and I/O usage) are compared and executes the strategy which minimizes the cost. But, because Continue Reading
Reading Time: 2 minutes In this post, we are going to look into how we can leverage apache spark’s memory streams for Unit testing What is it ? Apache spark’s memory streams is a concrete streaming source of memory data source that supports reading in Micro-Batch Stream Processing. Lets jump into it We will be using a memory stream writing some test data in memory as a stream. We Continue Reading
Reading Time: 4 minutes You must have heard that Apache Spark is a powerful distributed data processing engine. But do you know that Spark (with the help of Hive) can also act as a database? So, in this blog, we will learn how Apache Spark can be leveraged as a database by creating tables in it and querying upon them. Introduction Since Spark is a database in itself, we Continue Reading
Reading Time: 4 minutes In this blog, we will try to understand the concept of Persistence in Apache Spark in a very layman term with scenario-based examples. Note: The scenarios are only meant for your easy understanding. Spark Architecture Note: Cache memory can be shared between Executors. What does it mean by persisting/caching an RDD? Spark RDD persistence is an optimization technique which saves the result of RDD evaluation Continue Reading
Reading Time: 3 minutes Nowadays Delta lake is a buzz word in the Big Data world, especially among the spark developers because it relegates lots of issues found in the Big Data domain. Delta Lake is an open-source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. It is evolving day by day and adds cool features in its every release. Continue Reading
Reading Time: 4 minutes Joins in Apache Spark are fundamental transformations, but if you are not familiar with their internal algorithm, they can become too expensive.
Reading Time: 3 minutes Hi Folks!! In this blog, we are going to learn how we can integrate Spark Structured Streaming with Kafka and Cassandra to build a simple data pipeline. Spark Structured Streaming is a component of Apache Spark framework that enables scalable, high throughput, fault tolerant processing of data streams.Apache Kafka is a scalable, high performance, low latency platform that allows reading and writing streams of data Continue Reading
Reading Time: 3 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Understanding Stateful Streaming“. And this blog pertains to Handling Late Arriving Data in Spark Structured Streaming. So let’s get started. Handling Late Data With window aggregates (discussed in the previous blog) Spark automatically takes cares of late data. Every aggregate window is like a bucket Continue Reading
Reading Time: 3 minutes Spark providing us a high-level API – Dataset, which makes it easy to get type safety and securely perform manipulation in a distributed and a local environment without code changes. Also, spark structured streaming, a high-level API for stream processing allows us to stream a particular Dataset which is nothing but a type-safe structured streams. In this blog, we will see how we can create Continue Reading
Reading Time: 4 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Internals of Structured Streaming“. And this blog pertains to Stateful Streaming in Spark Structured Streaming. So let’s get started. Let’s start from the very basic understanding of what is Stateful Stream Processing. But to understand that, let’s first understand what Stateless Stream Processing is. In Continue Reading
Reading Time: 4 minutes Apache Spark is a fast and general-purpose cluster computing system. In Spark, we can do the batch processing and stream processing as well. It does near real-time processing. It means that it processes the data in micro-batches. I have discussed more Spark Streaming in my previous blog. Now in this blog, I’ll discuss Stateful Streaming in Spark. So let’s start !! What is Stateful Streaming? Continue Reading
Reading Time: 2 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Introduction to Structured Streaming“. So I’ll exactly start from the point where I left in the previous blog. Structure of Streaming Query When we call start() API, Spark internally translates this code into a Logical Plan (an abstract representation of what the code does), then Continue Reading
Reading Time: 5 minutes In this Spark Structured Streaming series of blogs, we will have a deep look into what structured streaming is in a very layman language. So let’s get started. Introduction Structured streaming is a stream processing engine built on top of the Spark SQL engine and uses the Spark SQL APIs. It is fast, scalable and fault-tolerant. It provides rich, unified and high-level APIs in the Continue Reading
Aanchal Agarwal
Agnibhas Chattopadhyay
Akash Kumar
Mohd Alimuddin
Alka Vats
Aman Verma
Ankit Mogha
Anuradha Kumari
Asbin Bhadra
Ashish Chaudhary
Atul Rawat
Ambika Bhardwaj
Bhavya Verma
DEVANSHU VERMA
Durgesh Gupta
Gaurav Kumar
Meenakshi Goyal
Shivam Gupta
Harshit Gupta
Jagdish Singh
Kiranjeet Kaur
Sakshi Mittal
Kuldeepak Gupta
Kuldeep
LOKESH KUMAR
Manas
Sonu Mehrotra
Nitin Mishra
Mohit Saxena
Naveen Singh
Nikhil Varghese
Nitin Mittal
Nitin Saxena
Piyush Agarwal
Praanav Bhowmik
Prajjawal Kansal
Prashant Trivedi
Prateek Gupta
Prince Agrahari
Rahul Soni
Rishabh Verma
Sanjay Rathore
Saumya
Shivani Sarthi
Sumit Raj
Shiv Oberoi
Utkarsh Upadhyay
Vikas Vashisth
Aamir Shahab
Abhishek Tiwari
Addi
Aditya Narayan
Akash Kale
Abhijna Saksena
Amarjeet Singh
Ankur Thakur
Anshika Agrawal
Ashutosh
Ayushi Dwivedi
Azmat
Bhavya Aggarwal
Syed Ali
Pooja Chawla
Harshit Daga
Diksha Yadav
Gaurav
Gaurav Dubey
Yash Gupta
Himanshu Upadhaya
Himanshu Chaudhary
Himanshu Gupta
Rohit Jagati
Jony Mandal
Rachel Jones
JustinB
Kartik Tiwari
Jasmine Kaur
Manish Mishra
Vikas Hazrati
Kritika Khandelwal
kumar Pratik
kundankumarr
Lokesh Aggarwal
Mansi
Mayura Zadane
Mithilesh
Mukesh Yadav
munandersingh
Muskan Gupta
Niraj
Narayan
Pawan Bisht
Peter Boyajian
Piyush
Prabhat Kashyap
Prachi singh
Praful Bangar
Pranay
Priyabrata Sahoo
Purva Agrawal
Rahul Khowal
Manyam Rajesh
Ram Indukuri
Ravinder
Rituraj Khare
Riya
Rohit Sharma
Joseph Ross
Ruchika Dubey
Sakshi Gawande
Santosh Pandre
Satyam
Shaurav Ranjan
Shradha Saxena
Sidharth Khattri
Sparsh Bhardwaj
GauravShukla
Shubhrank Rastogi
Sumit Agarwal
Suraj
Sushma
Swantika Gupta
Tanishka Garg
Vaibhav Sharma
Vidushi Bansal
Vijay
Vipul Kumar
Sam
Rahul
Yamini Bansal
yatharth45
