Insights and Perspectives to keep you updated
Reading Time: 4 minutes Apache Spark is a fast and general-purpose cluster computing system. In Spark, we can do the batch processing and stream processing as well. It does near real-time processing. It means that it processes the data in micro-batches. I have discussed more Spark Streaming in my previous blog. Now in this blog, I’ll discuss Stateful Streaming in Spark. So let’s start !! What is Stateful Streaming? Continue Reading
Reading Time: 2 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Introduction to Structured Streaming“. So I’ll exactly start from the point where I left in the previous blog. Structure of Streaming Query When we call start() API, Spark internally translates this code into a Logical Plan (an abstract representation of what the code does), then Continue Reading
Reading Time: 5 minutes In this Spark Structured Streaming series of blogs, we will have a deep look into what structured streaming is in a very layman language. So let’s get started. Introduction Structured streaming is a stream processing engine built on top of the Spark SQL engine and uses the Spark SQL APIs. It is fast, scalable and fault-tolerant. It provides rich, unified and high-level APIs in the Continue Reading
Reading Time: 4 minutes Spark provides two shared variables in distributed computing which are accessible to all the nodes in a spark cluster – broadcast variables & Accumulators.
Reading Time: 3 minutes Corona Virus Pandemic has brought the world to a standstill. Countries are on a major lockdown. Schools, colleges, theatres, gym, clubs, and all other public places are shut down, the country’s economy is suffering, human health is on stake, people are losing their jobs and nobody knows how worse it can get. Since most of the places are on lockdown, and you are working from Continue Reading
Reading Time: 5 minutes This blog focuses on some of the problems faced while working with the Spark SQL
Reading Time: 4 minutes A Spark app that takes data from HDFS and persists in Vertica.
Reading Time: 4 minutes This blog pertains to Apache SPARK 3.x, where we will find out how Spark’s new feature named Adaptive Query Execution (AQE) works internally. So let’s get started. One of the major feature introduced in Apache Spark 3.0 is the new Adaptive Query Execution (AQE) over the Spark SQL engine. So, in this feature, the Spark SQL engine can keep updating the execution plan per computation Continue Reading
Reading Time: 5 minutes Distributed stream processing engines have been on the rise in the last few years, first Hadoop became popular as a batch processing engine, then focus shifted towards stream processing engines. Stream processing engines can make the job of processing data that comes in via a stream easier than ever before and by using clustering can enable processing data in larger sets in a timely manner. Continue Reading
Reading Time: 4 minutes Nowadays data is constantly evolving and changing. As well as the business problems and requirements are evolving, the shape or the structure of the data is also changing. When that happens, we want to be in control of how the data or schema changes. But how we can achieve this? Delta Lake has good ways to control how schema changes. With Delta Lake, users have Continue Reading
Reading Time: 2 minutes There are two different ways to create a Dataframe in Spark. First, using toDF() and second is using createDataFrame(). In this blog we will see how we can create Dataframe using these two methods and what’s the exact difference between them. toDF() toDF() method provides a very concise way to create a Dataframe. This method can be applied to a sequence of objects. To access Continue Reading
Reading Time: 3 minutes Whenever we submit a Spark application to the cluster, the Driver or the Spark App Master should get started. And the Driver will be starting N number of workers. Spark driver will be managing spark context object to share the data and coordinates with the workers and cluster manager across the cluster. Cluster Manager can be Spark Standalone or Hadoop YARN or Mesos. Workers will Continue Reading
Reading Time: 4 minutes Morpheus brings the world of Graph Processing to Spark. It’s Property Graph Model and Cypher language allows easy inclusion of Graph into Spark pipelines
Atul Rawat
Krishna Singh
Manas Kashyap
Sonu Mehrotra
Sarfaraz Hussain
Suraj Saini
Abhinav Sharma
Abhishek Baranwal
Addi
Aditi Varshney
Anjali Sharma
Ankur Thakur
Anuj Saxena
Ayush Hooda
Ayush Mishra
Ayush_Singhal
Azmat Hasan
Bhavya Aggarwal
Bhawna Sharma
Piyush Rana
Pawan Bisht
Bobin Singla
Shubham Verma
Harshit Daga
Rudar
Deepak Mehra
etash2901
Gautam Singal
girdharshubham
Girish Bharti
gopal
Himanshu Gupta
Sonu Mehrotra
Jashan Goyal
Jasmine Kaur
Jitendra Kumar Singh
Juoko Virtanen
Rachel Jones
JustinB
Jyoti Jali
Jyoti Sachdeva
Kartik Tiwari
Rishi Khandelwal
Manish Mishra
Sarvesh Tripathi
Vikas Hazrati
Komal Rajpal
Kritika Khandelwal
kundankumarr
Sahil Sawhney
Lokesh Aggarwal
Megha Jatana
Mahesh Chand
Mansi Babbar
Mayank Bairagi
Miral Gandhi
Mukesh Yadav
munandersingh
Muskan Gupta
Narayan Kumar
Neelu Aggarwal
Neeraj Kumar
Neha Bhardwaj
Nishchal Vashisht
Nitin Arora
Pankaj Chaudhary
Peter Boyajian
Pinku Swargiary
Prabhat Kashyap
Praful Bangar
Pranjut Gogoi
Priyanka Thakur
Rachna Bulbule
Rahul Agarwal
Rahul Khanna
Ramandeep
Ram Indukuri
Randhir kumar
Joseph Ross
Ruchika Dubey
Sakshi Gawande
Sanjana Aggarwal
Sanjoli Varshney
Rajeev Thakur
Shivani Mehrotra
Shivraj Singh
Shubham Goyal
Shubham Dangare
Sidharth Khattri
Sparsh Bhardwaj
GauravShukla
Shubhrank Rastogi
Swantika Gupta
Swapnil Gosavi
Umang Goyal
Upanshu Chaudhary
Vandana Yadav
Vidisha Gupta
Vinay Kumar
Vinisha Sharma
woodruffsw
Yamini Bansal
Yatharth Sharma
Knoldus is the world's largest pure-play Scala and Spark company. We modernize enterprise through cutting-edge digital engineering by leveraging Scala, Functional Java and Spark ecosystem. Our mission is to provide reactive and streaming fast data solutions that are message-driven, elastic, resilient, and responsive.
+(1) 647-467-4396
hello@knoldus.com
