Apache Spark’s Join Algorithms
Reading Time: 4 minutes Joins in Apache Spark are fundamental transformations, but if you are not familiar with their internal algorithm, they can become too expensive.
Creating Data Pipeline with Spark streaming, Kafka and Cassandra
Reading Time: 3 minutes Hi Folks!! In this blog, we are going to learn how we can integrate Spark Structured Streaming with Kafka and Cassandra to build a simple data pipeline. Spark Structured Streaming is a component of Apache Spark framework that enables scalable, high throughput, fault tolerant processing of data streams.Apache Kafka is a scalable, high performance, low latency platform that allows reading and writing streams of data Continue Reading
Spark: Streaming Datasets
Reading Time: 3 minutes Spark providing us a high-level API – Dataset, which makes it easy to get type safety and securely perform manipulation in a distributed and a local environment without code changes. Also, spark structured streaming, a high-level API for stream processing allows us to stream a particular Dataset which is nothing but a type-safe structured streams. In this blog, we will see how we can create Continue Reading
Optimizations In Spark: For BETTER OR For WORSE
Reading Time: 5 minutes This blog focuses on some of the problems faced while working with the Spark SQL
Delta Lake: Schema Enforcement & Evolution
Reading Time: 4 minutes Nowadays data is constantly evolving and changing. As well as the business problems and requirements are evolving, the shape or the structure of the data is also changing. When that happens, we want to be in control of how the data or schema changes. But how we can achieve this? Delta Lake has good ways to control how schema changes. With Delta Lake, users have Continue Reading
Spark: createDataFrame() vs toDF()
Reading Time: 2 minutes There are two different ways to create a Dataframe in Spark. First, using toDF() and second is using createDataFrame(). In this blog we will see how we can create Dataframe using these two methods and what’s the exact difference between them. toDF() toDF() method provides a very concise way to create a Dataframe. This method can be applied to a sequence of objects. To access Continue Reading
Streaming from Kafka to PostgreSQL through Spark Structured Streaming
Reading Time: 3 minutes Hello everyone, in this blog we are going to learn how to do a structured streaming in spark with kafka and postgresql in our local system. We will be doing all this using scala so without any furthur pause, lets begin. Setting up the necessities first: Dependencies Set up the required dependencies for scala, spark, kafka and postgresql. 2. PostgreSQL setup Lets start fresh by Continue Reading
Apache Spark: Handle Corrupt/Bad Records
Reading Time: 3 minutes Most of the time writing ETL jobs becomes very expensive when it comes to handling corrupt records. And in such cases, ETL pipelines need a good solution to handle corrupted records. Because, larger the ETL pipeline is, the more complex it becomes to handle such bad records in between. Corrupt data includes: Missing information Incomplete information Schema mismatch Differing formats or data types Apache Spark: Continue Reading
Apache Spark: Tricks to Increase Job Performance
Reading Time: 2 minutes Apache Spark is quickly adopting the Real-world and most of the companies like Uber are using it in their production. Spark is gaining its popularity in the market as it also provides you with the feature of developing Streaming Applications and doing Machine Learning, which helps companies get better results in their production along with proper analysis using Spark. Although companies are using Spark in Continue Reading
Spark: ACID Transaction with Delta Lake
Reading Time: 3 minutes Spark doesn’t provide some of the most essential features of a reliable data processing system such as Atomic APIs and ACID transactions as discussed in the blog Spark: ACID compliant or not. Spark welcomes a solution to the problem by working with Delta Lake. Delta Lake plays an intermediary service between Apache Spark and the storage system. Instead of directly interacting with the storage layer, Continue Reading
Time Travel: Data versioning in Delta Lake
Reading Time: 3 minutes In today’s Big Data world, we process large amounts of data continuously and store the resulting data into data lake. This keeps changing the state of the data lake. But, sometimes we would like to access a historical version of our data. This requires versioning of data. Such kinds of data management simplifies our data pipeline by making it easy for professionals or organizations to Continue Reading
Spark: ACID compliant or not
Reading Time: 3 minutes Spark is not ACID compliant
Analytics on the edge – How Apache Mesos enabled ships to crunch data
Reading Time: 5 minutes Introduction & the Problem One of our key customers, a large cruise line has ships sail with capacity running into few thousands of people on board. They are going through a successful digital transformation which includes managing full life cycle of a guest on mobile, data science-driven personalization, etc and we are fortunate to be part of the whole journey. These ships generate varieties of Continue Reading
Agnibhas Chattopadhyay
Atul Rawat
Kamlesh Mishra
Krishna Singh
Kuldeepak Gupta
Manas Kashyap
Megha Tyagi
Sonu Mehrotra
Suraj Saini
Abhinav Sharma
Addi
Aditi Varshney
Anjali Sharma
Ankur Thakur
Anshika Agrawal
Anuj Saxena
Ayush Hooda
Ayush Mishra
Azmat Hasan
Bhavya Aggarwal
Piyush Rana
Pawan Bisht
Chirag Agarwal
Harshit Daga
Deevisha Verma
Divyanshu Srivastava
etash2901
girdharshubham
Girish Bharti
gopal
Himanshu Upadhaya
Himanshu Chaudhary
Himanshu Gupta
Sonu Mehrotra
Jashan Goyal
Jasmine Kaur
Jitendra Kumar Singh
Juoko Virtanen
Rachel Jones
JustinB
Jyoti Jali
Kartik Tiwari
Manish Mishra
Vikas Hazrati
Komal Rajpal
Kritika Khandelwal
Pratik Kumar
kundankumarr
Sahil Sawhney
Lokesh Aggarwal
Megha Jatana
Mahesh Chand
Mansi Babbar
Mayank Bairagi
Mukesh Yadav
munandersingh
Muskan Gupta
Narayan Kumar
Neelu Aggarwal
Neha Bhardwaj
Pankaj Chaudhary
Peter Boyajian
Pinku Swargiary
Prabhat Kashyap
Praful Bangar
Priyanka Thakur
Purva Agrawal
Rachna Bulbule
Rahul Agarwal
Rahul Khanna
Ram Indukuri
Randhir kumar
Joseph Ross
Ruchika Dubey
Sakshi Gawande
Sanjana Aggarwal
Sanjoli Varshney
Rajeev Thakur
Shivani Mehrotra
Shivraj Singh
Shradha Saxena
Shubham Dangare
Sidharth Khattri
Sparsh Bhardwaj
GauravShukla
Shubhrank Rastogi
Swantika Gupta
Swapnil Gosavi
Umang Goyal
Upanshu Chaudhary
Vandana Yadav
Vidushi Bansal
Vinay Kumar
Vinisha Sharma
woodruffsw
Yamini Bansal
Yatharth Sharma
