Data Lake – Build it in Phases
Reading Time: 3 minutes Data Lake – How to build a data lake and what are the phases involved in the same.
Apache Spark: Read Data from S3 Bucket
Reading Time: 2 minutes Well, a one working with spark is very much familiar with the ways of reading the file from local either from a Table or HDFS or from any file. But do you know how tricky it is to read data into spark from an S3 bucket? So, this blog makes you give a stepwise follow up to how to read data from an S3 bucket. Continue Reading
Apache Spark: Repartitioning v/s Coalesce
Reading Time: 3 minutes Does partitioning help you increase/decrease the Job Performance? Spark splits data into partitions and computation is done in parallel for each partition. It is very important to understand how data is partitioned and when you need to manually modify the partitioning to run spark applications efficiently. Now, diving into our main topic i.e Repartitioning v/s Coalesce What is Coalesce? The coalesce method reduces the number Continue Reading
Big Data Landscape explained
Reading Time: 5 minutes Big Data has now evolved into a buzz word and it seems everyone is either working on it or want to work on it. However, most of the people associate Big Data with some of the popular tool sets like Hadoop, Spark, NoSql databases like Hive, Cassandra , HBase etc. HDFS made Big Data popular as it gave us an option to distribute the data Continue Reading
Understanding the working of Spark Driver and Executor
Reading Time: 4 minutes This blog pertains to Apache SPARK, where we will understand how Spark’s Driver and Executors communicate with each other to process a given job. So let’s get started. First, let’s see what Apache Spark is. The official definition of Apache Spark says that “Apache Spark™ is a unified analytics engine for large-scale data processing.” It is an in-memory computation processing engine where the data is Continue Reading
Understanding how Spark runs on YARN with HDFS
Reading Time: 6 minutes This blog pertains to Apache SPARK and YARN (Yet Another Resource Negotiator), where we will understand how Spark runs on YARN with HDFS. So let’s get started. First, let’s see what Apache Spark is. The official definition of Apache Spark says that “Apache Spark™ is a unified analytics engine for large-scale data processing.” It is an in-memory computation processing engine where the data is kept Continue Reading
Kryo Serialization in Spark
Reading Time: 4 minutes Spark provides two types of serialization libraries: Java serialization and (default) Kryo serialization. For faster serialization and deserialization spark itself recommends to use Kryo serialization in any network-intensive application. Then why is it not set to default : Why Kryo is not set to default in Spark? The only reason Kryo is not set to default is because it requires custom registration. Although, Kryo is Continue Reading
Understanding Spark’s Logical and Physical Plan in layman’s term
Reading Time: 5 minutes This blog pertains to Apache SPARK 2.x, where we will find out how Spark SQL works internally in layman’s terms and try to understand what is Logical and Physical Plan. Also we will be looking into Catalyst Optimizer. So let’s get started. First let’s see what Apache Spark is. The official definition of Apache Spark says that “Apache Spark™ is a unified analytics engine for large-scale Continue Reading
Deep Dive into Apache Spark Transformations and Action
Reading Time: 4 minutes In our previous blog of Apache Spark, we discussed a little about what Transformations & Actions are? Now we will get deeper into the topic and will understand what actually they are & how they play a vital role to work with Apache Spark? What is Spark RDD? Spark introduces the concept of an RDD (Resilient Distributed Dataset), an immutable fault-tolerant, distributed collection of objects Continue Reading
Diving deeper into Delta Lake
Reading Time: 6 minutes Delta Lake is an open-source storage layer that brings reliability to data lakes. It has numerous reliability features including ACID transactions, scalable metadata handling, and unified streaming and batch data processing.
Delta Lake To the Rescue
Reading Time: 4 minutes Welcome Back. In our previous blogs, we tried to get some insights about Spark RDDs and also tried to explore some new things in Spark 2.4. You can go through those blogs here: RDDs – The backbone of Apache Spark Spark 2.4: Adding a little more Spark to your code In this blog, we will be discussing something called a Delta Lake. But first, let’s Continue Reading
Spark – Actions and Transformations
Reading Time: 4 minutes Hey guys, welcome to series of spark blogs, this blog being the first blog in this series we would try to keep things as crisp as possible, so let’s get started. So I recently get to start learning spark about believe me and now it has made me inquisitive about it, for a brief introduction of spark, I would say that it is a pretty Continue Reading
Tale of Apache Spark
Reading Time: 6 minutes Data is being produced extensively in today’s world and it is going to be generated more rapidly in future. 90% of total data that is produced in the world is produced in last two years only and it is estimated that in 2020 world’s total data would reach 45 ZB and data generated each day would be enough that if we try to store it Continue Reading
Sarfaraz Hussain
Aakash Jain
Alok Jha
Anjali Sharma
Ankur Thakur
Anmol Sarna
Anuj Saxena
Ayush Hooda
Ayush Mishra
Ayush Prashar
Ayush_Singhal
Azmat Hasan
Bhavya Aggarwal
Bhawna Sharma
Piyush Rana
Pawan Bisht
Shubham Verma
Charmy Garg
Harshit Daga
Rudar
Deepak Mehra
Deepankar
Divyansh Jain
Ayush Tiwari
etash2901
Gautam Singal
girdharshubham
Girish Bharti
Himani Arora
Himanshu Gupta
Sonu Mehrotra
Nancy Jain
Jashan Goyal
Jasmine Kaur
Jatin Demla
Juoko Virtanen
Rachel Jones
JustinB
Jyoti Sachdeva
Rishi Khandelwal
Manish Mishra
Vikas Hazrati
Komal Rajpal
Kritika Khandelwal
kundankumarr
Sahil Sawhney
Lokesh Aggarwal
Megha Jatana
Mahesh Chand
Mayank Bairagi
Miral Gandhi
Mukesh Kumar
munandersingh
Kaushik Nath
Narayan Kumar
Neelam Lakra
Neelu Aggarwal
Neeraj Chugh
Neha Bhardwaj
Niharika Datta
Nitin Arora
Pankaj Chaudhary
Pinku Swargiary
Prabhat Kashyap
Praful Bangar
Pranjut Gogoi
Prashant Sharma
Priyanka Thakur
Rachna Bulbule
Rahul Khanna
Ramandeep
Ram Indukuri
Randhir kumar
Joseph Ross
Ruchika Dubey
Sakshi Gawande
Rajeev Thakur
Shivangi Gupta
Shivraj Singh
Shubham Goyal
Shubham Dangare
Sidharth Khattri
Sachin Slathia
GauravShukla
Shubhrank Rastogi
Sudeep James Tirkey
Swantika Gupta
Swapnil Gosavi
Nitin Aggarwal
Utkarsha Musmade
Vandana Yadav
Vidisha Gupta
Vinay Kumar
Vinisha Sharma
Yatharth Sharma
+(1) 647-467-4396
hello@knoldus.com
