Reading Time: 4 minutes In this blog, we will try to understand the concept of Persistence in Apache Spark in a very layman term with scenario-based examples. Note: The scenarios are only meant for your easy understanding. Spark Architecture Note: Cache memory can be shared between Executors. What does it mean by persisting/caching an RDD? Spark RDD persistence is an optimization technique which saves the result of RDD evaluation Continue Reading
Reading Time: 3 minutes Nowadays Delta lake is a buzz word in the Big Data world, especially among the spark developers because it relegates lots of issues found in the Big Data domain. Delta Lake is an open-source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. It is evolving day by day and adds cool features in its every release. Continue Reading
Reading Time: 4 minutes Joins in Apache Spark are fundamental transformations, but if you are not familiar with their internal algorithm, they can become too expensive.
Reading Time: 3 minutes Hi Folks!! In this blog, we are going to learn how we can integrate Spark Structured Streaming with Kafka and Cassandra to build a simple data pipeline. Spark Structured Streaming is a component of Apache Spark framework that enables scalable, high throughput, fault tolerant processing of data streams.Apache Kafka is a scalable, high performance, low latency platform that allows reading and writing streams of data Continue Reading
Reading Time: 3 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Understanding Stateful Streaming“. And this blog pertains to Handling Late Arriving Data in Spark Structured Streaming. So let’s get started. Handling Late Data With window aggregates (discussed in the previous blog) Spark automatically takes cares of late data. Every aggregate window is like a bucket Continue Reading
Reading Time: 3 minutes Spark providing us a high-level API – Dataset, which makes it easy to get type safety and securely perform manipulation in a distributed and a local environment without code changes. Also, spark structured streaming, a high-level API for stream processing allows us to stream a particular Dataset which is nothing but a type-safe structured streams. In this blog, we will see how we can create Continue Reading
Reading Time: 4 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Internals of Structured Streaming“. And this blog pertains to Stateful Streaming in Spark Structured Streaming. So let’s get started. Let’s start from the very basic understanding of what is Stateful Stream Processing. But to understand that, let’s first understand what Stateless Stream Processing is. In Continue Reading
Reading Time: 4 minutes Apache Spark is a fast and general-purpose cluster computing system. In Spark, we can do the batch processing and stream processing as well. It does near real-time processing. It means that it processes the data in micro-batches. I have discussed more Spark Streaming in my previous blog. Now in this blog, I’ll discuss Stateful Streaming in Spark. So let’s start !! What is Stateful Streaming? Continue Reading
Reading Time: 2 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Introduction to Structured Streaming“. So I’ll exactly start from the point where I left in the previous blog. Structure of Streaming Query When we call start() API, Spark internally translates this code into a Logical Plan (an abstract representation of what the code does), then Continue Reading
Reading Time: 5 minutes In this Spark Structured Streaming series of blogs, we will have a deep look into what structured streaming is in a very layman language. So let’s get started. Introduction Structured streaming is a stream processing engine built on top of the Spark SQL engine and uses the Spark SQL APIs. It is fast, scalable and fault-tolerant. It provides rich, unified and high-level APIs in the Continue Reading
Reading Time: 4 minutes Spark provides two shared variables in distributed computing which are accessible to all the nodes in a spark cluster – broadcast variables & Accumulators.
Reading Time: 3 minutes Corona Virus Pandemic has brought the world to a standstill. Countries are on a major lockdown. Schools, colleges, theatres, gym, clubs, and all other public places are shut down, the country’s economy is suffering, human health is on stake, people are losing their jobs and nobody knows how worse it can get. Since most of the places are on lockdown, and you are working from Continue Reading
Reading Time: 5 minutes This blog focuses on some of the problems faced while working with the Spark SQL
Atul Rawat
Krishna Singh
Manas Kashyap
Sonu Mehrotra
Sarfaraz Hussain
Suraj Saini
Abhinav Sharma
Abhishek Baranwal
Addi
Aditi Varshney
Ahmed Moustafa
Anjali Sharma
Ankur Thakur
Anshika Agrawal
Anuj Saxena
Ayush Hooda
Ayush Mishra
Ayush_Singhal
Azmat Hasan
Bhavya Aggarwal
Bhawna Sharma
Piyush Rana
Pawan Bisht
Bobin Singla
Shubham Verma
Harshit Daga
Rudar
Deepak Mehra
Deevisha Verma
etash2901
Gautam Singal
girdharshubham
Girish Bharti
gopal
Himanshu Chaudhary
Himanshu Gupta
Sonu Mehrotra
Jashan Goyal
Jasmine Kaur
Jitendra Kumar Singh
Juoko Virtanen
Rachel Jones
JustinB
Jyoti Jali
Jyoti Sachdeva
Kartik Tiwari
Rishi Khandelwal
Manish Mishra
Vikas Hazrati
Komal Rajpal
Kritika Khandelwal
kundankumarr
Sahil Sawhney
Lokesh Aggarwal
Megha Jatana
Mahesh Chand
Mansi Babbar
Mayank Bairagi
Minh Nguyen
Miral Gandhi
Mukesh Yadav
munandersingh
Muskan Gupta
Narayan Kumar
Neelu Aggarwal
Neha Bhardwaj
Nitin Arora
Pankaj Chaudhary
Peter Boyajian
Pinku Swargiary
Prabhat Kashyap
Praful Bangar
Priyanka Thakur
Rachna Bulbule
Rahul Agarwal
Rahul Khanna
Ramandeep
Ram Indukuri
Randhir kumar
Joseph Ross
Ruchika Dubey
Sakshi Gawande
Sanjana Aggarwal
Sanjoli Varshney
Rajeev Thakur
Shivani Mehrotra
Shivraj Singh
Shubham Goyal
Shubham Dangare
Shubhangi Pandey
Sidharth Khattri
Sparsh Bhardwaj
GauravShukla
Shubhrank Rastogi
Swantika Gupta
Swapnil Gosavi
Umang Goyal
Upanshu Chaudhary
Vandana Yadav
Vidisha Gupta
Vinay Kumar
Vinisha Sharma
woodruffsw
Yamini Bansal
Yatharth Sharma
