Reading Time: 6 minutes There are various resources that give advice on how to [and how not to] partition your data, how to calculate the ideal file size, how to handle evolving schemas, how to build compaction routines, how to recover from failed ETL jobs, how to stream raw data into the data lake, etc. We have been working with customers throughout this time to encapsulate all of the Continue Reading
Reading Time: 3 minutes Nowadays Delta lake is a buzz word in the Big Data world, especially among the spark developers because it relegates lots of issues found in the Big Data domain. Delta Lake is an open-source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. It is evolving day by day and adds cool features in its every release. Continue Reading
Reading Time: 4 minutes Nowadays data is constantly evolving and changing. As well as the business problems and requirements are evolving, the shape or the structure of the data is also changing. When that happens, we want to be in control of how the data or schema changes. But how we can achieve this? Delta Lake has good ways to control how schema changes. With Delta Lake, users have Continue Reading
Reading Time: 3 minutes Well, we have already covered the missing features in Apache Spark & also the causes of the issue in executing Delta Lake in Part1. However, today we will be talking about What Delta Lake is & how it provides the solution to all those problems discussed herein Delta Lake as a Solution: Part1.As we all know that Spark is just a processing engine, it doesn’t Continue Reading
Reading Time: 3 minutes Today, everyone is talking about Delta Lake. Why? Ever tried to find the answer to this question? Yes or No doesn’t matter, don’t worry here in Part1 we will be discussing the same & also will be targetting the following questions: What are the features missing from Apache Spark? What kind of issues it causes in executing Data Lake? Answering the above questions will definitely Continue Reading
Reading Time: 3 minutes Spark doesn’t provide some of the most essential features of a reliable data processing system such as Atomic APIs and ACID transactions as discussed in the blog Spark: ACID compliant or not. Spark welcomes a solution to the problem by working with Delta Lake. Delta Lake plays an intermediary service between Apache Spark and the storage system. Instead of directly interacting with the storage layer, Continue Reading
Reading Time: 3 minutes In today’s Big Data world, we process large amounts of data continuously and store the resulting data into data lake. This keeps changing the state of the data lake. But, sometimes we would like to access a historical version of our data. This requires versioning of data. Such kinds of data management simplifies our data pipeline by making it easy for professionals or organizations to Continue Reading
Reading Time: 6 minutes Delta Lake is an open-source storage layer that brings reliability to data lakes. It has numerous reliability features including ACID transactions, scalable metadata handling, and unified streaming and batch data processing.
Reading Time: 4 minutes Welcome Back. In our previous blogs, we tried to get some insights about Spark RDDs and also tried to explore some new things in Spark 2.4. You can go through those blogs here: RDDs – The backbone of Apache Spark Spark 2.4: Adding a little more Spark to your code In this blog, we will be discussing something called a Delta Lake. But first, let’s Continue Reading
Agnibhas Chattopadhyay
Asbin Bhadra
Atul Rawat
Durgesh Gupta
Meenakshi Goyal
Kamlesh Mishra
Krishna Singh
Kuldeepak Gupta
Manas Kashyap
Megha Tyagi
Sonu Mehrotra
Nikhil Varghese
Praanav Bhowmik
Prajjawal Kansal
Prateek Gupta
Suraj Saini
Abhinav Sharma
Addi
Aditi Varshney
Anjali Sharma
Ankur Thakur
Anshika Agrawal
Anuj Saxena
Ayush Hooda
Ayush Mishra
Azmat Hasan
Bhavya Aggarwal
Piyush Rana
Pawan Bisht
Chirag Agarwal
Harshit Daga
Deevisha Verma
Divyanshu Srivastava
etash2901
girdharshubham
Girish Bharti
gopal
Yash Gupta
Himanshu Upadhaya
Himanshu Chaudhary
Himanshu Gupta
Sonu Mehrotra
Jashan Goyal
Jasmine Kaur
Jitendra Kumar Singh
Juoko Virtanen
Rachel Jones
JustinB
Jyoti Jali
Kartik Tiwari
Manish Mishra
Vikas Hazrati
Komal Rajpal
Kritika Khandelwal
Pratik Kumar
kundankumarr
Lokesh Aggarwal
Megha Jatana
Mahesh Chand
Mansi Babbar
Mukesh Yadav
munandersingh
Muskan Gupta
Narayan Kumar
Neelu Aggarwal
Neha Bhardwaj
Pankaj Chaudhary
Peter Boyajian
Pinku Swargiary
Prabhat Kashyap
Praful Bangar
Priyanka Thakur
Purva Agrawal
Rachna Bulbule
Rahul Agarwal
Rahul Khanna
Ram Indukuri
Joseph Ross
Ruchika Dubey
Sakshi Gawande
Sanjana Aggarwal
Sanjoli Varshney
Rajeev Thakur
Shivani Mehrotra
Shivani Verma
Shivraj Singh
Shradha Saxena
Shubham Dangare
Sidharth Khattri
Sparsh Bhardwaj
GauravShukla
Shubhrank Rastogi
Swantika Gupta
Swapnil Gosavi
Tanishka Garg
Umang Goyal
Upanshu Chaudhary
Vandana Yadav
Vidushi Bansal
Vinay Kumar
Vinisha Sharma
woodruffsw
Yamini Bansal
Yatharth Sharma
