Insights and Perspectives to keep you updated
Reading Time: 6 minutes Delta Lake is an open-source storage layer that brings reliability to data lakes. It has numerous reliability features including ACID transactions, scalable metadata handling, and unified streaming and batch data processing.
Reading Time: 4 minutes Welcome Back. In our previous blogs, we tried to get some insights about Spark RDDs and also tried to explore some new things in Spark 2.4. You can go through those blogs here: RDDs – The backbone of Apache Spark Spark 2.4: Adding a little more Spark to your code In this blog, we will be discussing something called a Delta Lake. But first, let’s Continue Reading
Reading Time: 4 minutes Hey guys, welcome to series of spark blogs, this blog being the first blog in this series we would try to keep things as crisp as possible, so let’s get started. So I recently get to start learning spark about believe me and now it has made me inquisitive about it, for a brief introduction of spark, I would say that it is a pretty Continue Reading
Reading Time: 4 minutes We are now generating massive volumes of data at an accelerated rate. To meet business needs, address changing market dynamics as well as improve decision-making, sophisticated analysis of this data from disparate sources is required. The challenge is how to capture, store and model these massive pools of data effectively in relational databases. Big data is not a fad. We are just at the beginning Continue Reading
Reading Time: 3 minutes In two previous blogs, we explored about Vertica and how it can be connected to Apache Spark. The first blog in this mini series was about reading data from Vertica using Spark and saving that data into Kafka. The next blog explained the reverse flow i.e. reading data from Kafka and writing data to Vertica but in a batch mode. i.e reading data from Kafka Continue Reading
Reading Time: 4 minutes In previous blog of this series, we took a glance over the basic definition of Spark and Vertica. We also did a code overview for reading data from Vertica using Spark as DataFrame and saving the data into Kafka. In this blog we will be doing the reverse flow i.e. working on reading the data from Kafka as a DataFrame and writing that DataFrame into Continue Reading
Reading Time: 4 minutes We live in a world of Big data where the size of data is so big even for small results. This is the result of an increase in data collection on a rapid scale in the modern world. This massiveness of data brings the requirements of such tools which can work upon such a big chunk of data. I am pretty sure that you guys Continue Reading
Reading Time: 5 minutes With the massive amount of increase in big data technologies today, it is becoming very important to use the right tool for every process. The process can be anything like Data ingestion, Data processing, Data retrieval, Data Storage, etc. Today we are going to focus on one of those popular big data technologies i.e., Apache Spark. Apache Spark is an open-source distributed general-purpose cluster-computing framework. Spark Continue Reading
Reading Time: 3 minutes As I have already discussed in my previous blog Spark: RDD vs DataFrames about the shortcomings of RDDs and how DataFrames overcome them. Now we’ll try to have a look at the shortcomings of DataFrames and how Dataset APIs can overcome them. DataFrames:- A DataFrame is a distributed collection of data, which is organized into named columns. Conceptually, it is equivalent to the relational tables with Continue Reading
Reading Time: 6 minutes Fan of Apache Spark? I am too. The reason is simple. Interesting APIs to work with, fast and distributed processing, unlike map-reduce no I/O overhead, fault tolerance and many more. With this much, you can do a lot in this world of Big data and Fast data. From “processing huge chunks of data” to “working on streaming data”, Spark works flawlessly in all. In this Continue Reading
Reading Time: 3 minutes Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. Internally, Spark SQL uses this extra information to perform extra optimizations.One use of Spark SQL is to execute SQL queries. When running SQL from within another Continue Reading
Reading Time: 4 minutes optimize indexing
bulk indexing elasticsearch
Reading Time: 1 minute Knoldus has organized a 30 min session on Oct 12 at 3:30 PM. The topic was How Spark does it internally? Many people have joined and enjoyed the session. I am going to share the slides and the video here. Please let me know if you have any question related to linked slides. How Spark Does It Internally? from Knoldus Inc. Here’s the video of the Continue Reading
Aakash Jain
Abhishek Baranwal
Akshansh Jain
Alok Jha
Anmol Sarna
Anuj Saxena
Ayush Hooda
Ayushi Hasija
Ayush Mishra
Ayush Prashar
Ayush_Singhal
Azmat Hasan
Bhavya Aggarwal
Bhawna Sharma
Piyush Rana
Pawan Bisht
Shubham Verma
Charmy Garg
Harshit Daga
Rudar
Deepak Mehra
Deepankar
Divyansh Jain
Ayush Tiwari
etash2901
girdharshubham
Girish Bharti
Harmeet Singh(Taara)
Himani Arora
Himanshu Gupta
Himanshu Rajput
Sonu Mehrotra
Nancy Jain
Jashan Goyal
Jasmine Kaur
Jatin Demla
Juoko Virtanen
Rachel Jones
JustinB
Jyoti Sachdeva
Kamlesh prajapati
Rishi Khandelwal
Manish Mishra
Vikas Hazrati
Kritika Khandelwal
kundankumarr
Sahil Sawhney
Megha Jatana
Mahesh Chand
Mayank Bairagi
Miral Gandhi
Mukesh Kumar
munandersingh
Kaushik Nath
Narayan Kumar
Neelam Lakra
Neelu Aggarwal
neerajchugh
Neha Bhardwaj
Niharika Datta
Nitin Arora
Pallavi Singh
Pankaj Chaudhary
Prabhat Kashyap
Pranjut Gogoi
Prashant Sharma
Priyanka Thakur
Rahul Khanna
Ramandeep
Ram Indukuri
Randhir kumar
Joseph Ross
Ruchika Dubey
Rajeev Thakur
Shivangi Gupta
Shubham Dangare
Shubham Goyal
Sidharth Khattri
Sachin Slathia
Sourabh Verma
GauravShukla
Shubhrank Rastogi
Sudeep James Tirkey
Nitin Aggarwal
kunal sethi
Vandana Yadav
Vidisha Gupta
Vinay Kumar
Vinisha Sharma
Yatharth Sharma
Knoldus is the world's largest pure-play Scala and Spark company. We modernize enterprise through cutting-edge digital engineering by leveraging Scala, Functional Java and Spark ecosystem. Our mission is to provide reactive and streaming fast data solutions that are message-driven, elastic, resilient, and responsive.
+(1) 647-467-4396
hello@knoldus.com
