With the massive amount of increase in big data technologies today, it is becoming very important to use the right tool for every process. The process can be anything like Data ingestion, Data processing, Data retrieval, Data Storage, etc. Today we are going to focus on one of those popular big data technologies i.e., Apache Spark. Apache Spark is an open-source distributed general-purpose cluster-computing framework. Spark Continue Reading
As I have already discussed in my previous blog Spark: RDD vs DataFrames about the shortcomings of RDDs and how DataFrames overcome them. Now we’ll try to have a look at the shortcomings of DataFrames and how Dataset APIs can overcome them. DataFrames:- A DataFrame is a distributed collection of data, which is organized into named columns. Conceptually, it is equivalent to the relational tables with Continue Reading
Fan of Apache Spark? I am too. The reason is simple. Interesting APIs to work with, fast and distributed processing, unlike map-reduce no I/O overhead, fault tolerance and many more. With this much, you can do a lot in this world of Big data and Fast data. From “processing huge chunks of data” to “working on streaming data”, Spark works flawlessly in all. In this Continue Reading
Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. Internally, Spark SQL uses this extra information to perform extra optimizations.One use of Spark SQL is to execute SQL queries. When running SQL from within another Continue Reading
optimize indexing
bulk indexing elasticsearch
Knoldus has organized a 30 min session on Oct 12 at 3:30 PM. The topic was How Spark does it internally? Many people have joined and enjoyed the session. I am going to share the slides and the video here. Please let me know if you have any question related to linked slides. How Spark Does It Internally? from Knoldus Inc. Here’s the video of the Continue Reading
Continuing with the objectives to make Spark faster, easier, and smarter, Apache Spark recently released its fifth release in the 2.x version line i.e Spark 2.4. We were lucky enough to experiment with it so soon in one of our projects. Today we will try to highlight the major changes in this version that we explored as well as experienced in our project. In our Continue Reading
Having trouble optimizing your Spark application? If yes, then this blog will surely guide you on how you can optimize it and what parameters should be tuned so that our spark application gives the best performance. Spark applications can cause a bottleneck due to resources such as CPU, memory, network etc. We need to tune our memory usage, data structures tuning, how RDDs need to Continue Reading
There has been a significant boom in distributed computing over the past few years. Various components communicate with each other over network inspite of being deployed on different physical machines. A distributed file system (DFS) is a file system with data stored on a server. The data is accessed and processed as if it was stored on the local client machine. The DFS makes it convenient to share information Continue Reading
Spark 2.0 is the next major release of Apache Spark. This brings major change for the level of abstraction for the spark API and libraries. The release has the major change for the ones who want to make use of all the advancement in this release, So in this blog post, I’ll be discussing Spark-Session. Need Of Spark-Session
Both the technologies are equipped with amazing features, however with the increased need for real-time analytics, these two giving tough competition to each other What are MapReduce and Spark? MapReduce:- MapReduce is a programming model for processing huge amounts of data in a parallel and distributed. In this model, there are two tasks that are undertaken Map and Reduce and there is a map function Continue Reading
There’s been a lot of time we have been working on streaming data. Using Apache Spark for that can be much convenient. Spark provides two APIs for streaming data one is Spark Streaming which is a separate library provided by Spark. Another one is Structured Streaming which is built upon the Spark-SQL library. We will discuss the trade-offs and differences between these two libraries in Continue Reading
A fault-tolerant collection of elements that can be operated on in parallel: “Resilient Distributed Dataset” a.k.a. RDD RDD (Resilient Distributed Dataset) is the fundamental data structure of Apache Spark which are an immutable collection of objects which computes on the different node of the cluster. Each and every dataset in Spark RDD is logically partitioned across many servers so that they can be computed on Continue Reading
Paramjeet Yadav
Aakash Jain
Abhishek Baranwal
Akhil Vijayan
Akshansh Jain
Alok Jha
Anmol Sarna
Anuj Saxena
Anurag Srivastava
Ayush Hooda
Ayushi Hasija
Ayush Mishra
Ayush Prashar
Ayush Sharma
Ayush_Singhal
Bhavya Aggarwal
Bhawna Sharma
Piyush Rana
Pawan Bisht
Shubham Verma
Charmy Garg
Harshit Daga
Rudar
Deepak Mehra
Deepankar
Ayush Tiwari
girdharshubham
Girish Bharti
Harmeet Singh(Taara)
Himani Arora
Himanshu Gupta
Himanshu Rajput
Sonu Mehrotra
Nancy Jain
Jashan Goyal
Jasmine Kaur
Jatin Demla
joukovirtanen
Rachel Jones
JustinB
Jyoti Sachdeva
Kamlesh prajapati
Rishi Khandelwal
Manish Mishra
Vikas Hazrati
Kritika Khandelwal
Sahil Sawhney
Megha Jatana
Mahesh Chand
Manjot Kaur
Mayank Bairagi
Mehak Juneja
Miral Gandhi
muditchhabra6125
Mukesh Kumar
Narayan Kumar
Neelam Lakra
Neelu Aggarwal
Neha Bhardwaj
Neelaksh
Nitin Arora
Pallavi Singh
Prabhat Kashyap
Pranjut Gogoi
Prashant Sharma
Priyanka Thakur
Rahul Khanna
Ramandeep
Ram Indukuri
Randhir kumar
Joseph Ross
Ruchika Dubey
Sajit Gupta
sakshigupta2312
Rajeev Thakur
Shivangi Gupta
Shubham Dangare
shubhamgoyal01
Sidharth Khattri
Sachin Slathia
Sourabh Verma
GauravShukla
Shubhrank Rastogi
Sudeep James Tirkey
Nitin Aggarwal
kunal sethi
Vandana Yadav
Vidisha Gupta
Vinay Kumar
Vinisha Sharma
Yatharth Sharma
Knoldus is the world's largest pure-play Scala and Spark company. We modernize enterprise through cutting-edge digital engineering by leveraging Scala, Functional Java and Spark ecosystem. Our mission is to provide reactive and streaming fast data solutions that are message-driven, elastic, resilient, and responsive.
+(1) 647-467-4396
hello@knoldus.com
