Fan of Apache Spark? I am too. The reason is simple. Interesting APIs to work with, fast and distributed processing, unlike map-reduce no I/O overhead, fault tolerance and many more. With this much, you can do a lot in this world of Big data and Fast data. From “processing huge chunks of data” to “working on streaming data”, Spark works flawlessly in all. In this Continue Reading
optimize indexing
bulk indexing elasticsearch
Continuing with the objectives to make Spark faster, easier, and smarter, Apache Spark recently released its fifth release in the 2.x version line i.e Spark 2.4. We were lucky enough to experiment with it so soon in one of our projects. Today we will try to highlight the major changes in this version that we explored as well as experienced in our project. In our Continue Reading
There’s been a lot of time we have been working on streaming data. Using Apache Spark for that can be much convenient. Spark provides two APIs for streaming data one is Spark Streaming which is a separate library provided by Spark. Another one is Structured Streaming which is built upon the Spark-SQL library. We will discuss the trade-offs and differences between these two libraries in Continue Reading
In our last blog, we tried to understand about using the spark streaming to transform and transport data between Kafka topics. After reading that many of the readers asked us to give a brief description of RDDs in Spark which we used. So, this blog is totally dedicated to the RDDs in Spark. So let’s start with the very basic question that comes to our mind Continue Reading
I am working in a company (Knoldus Software LLP) where Apache Spark is literally running into people’s blood means there are certain people who are really good at it. If you ever visit our blogging page and search for stuff related to spark, you will find enough content which is capable of solving your most of spark related queries, starting form introductions to solutions for Continue Reading
The term Big Data has created a lot of hype already in the business world. Hadoop and Spark are both Big Data frameworks – they provide some of the most popular tools used to carry out common Big Data-related tasks. In this blog, we will cover what is the difference between Apache Hadoop and Apache Spark MapReduce. Introduction Spark – It is an open source Continue Reading
As we know Spark is used at a wide range of organizations to process large datasets. It seems like spark becoming main stream. In this blog we will talk about Assimilation of Spark Streaming With Kafka. So, lets get started. How Kafka can be integrated with Spark? Kafka provides a messaging and integration platform for Spark streaming. Kafka act as the central hub for real-time streams of Continue Reading
Apache recently released a newer version of Spark i.e Apache Spark 2.2. The new version comes with new improvements as well as the addition of new functionalities. The major addition to this release is Structured Streaming. It has been marked as production ready and its experimental tag has been removed. Some of the high-level changes and improvements : Production ready Structured Streaming Expanding SQL functionalities New Continue Reading
“Structured Streaming”, nowadays we are hearing this term in Apache Spark ecosystem quite a lot, as it is being preached as next big thing in scalable big data world. Although, we all know that Structured Streaming means a stream having structured data in it, but very few of us knows what exactly it is and where we can use it. So, in this blog post Continue Reading
Apache Spark is an open source big data processing framework built around speed, ease of use, and sophisticated analytics. It was originally developed in 2009 in UC Berkeley’s AMPLab, and open sourced in 2010 as an Apache project. Spark has several advantages compared to other big data and Map Reduce technologies like Hadoop and Storm. Apache Spark is an improvement on the original Hadoop MapReduce Continue Reading
While working with Spark often we come across the three APIs: DataFrames, Datasets and RDDs. In this blog I will discuss the three in terms of use case, performance and optimization. It is essential to keep in mind that there is seamless transformation available between the three DataFrames, Datasets and RDDs. Implicitly the RDD forms the apex of both DataFrame and Datasets. The inception of Continue Reading
Data loading, in Spark SQL, means loading data in memory/cache of Spark worker nodes. For which we use to write following code: val connectionProperties = new Properties() connectionProperties.put(“user”, “username”) connectionProperties.put(“password”, “password”) val jdbcDF = spark.read .jdbc(“jdbc:postgresql:dbserver”, “schema.table”, connectionProperties) In here we are using jdbc function of DataFrameReader API of Spark SQL to load the data from table into Spark Executor’s memory, no matter how many rows are Continue Reading
Paramjeet Yadav
Aakash Jain
Abhishek Baranwal
Akhil Vijayan
Akshansh Jain
Alok Jha
Anmol Sarna
Anuj Saxena
Anurag Srivastava
Ayush Aggarwal
Ayush Hooda
Ayushi Hasija
Ayush Mishra
Ayush Prashar
Ayush Sharma
Ayush_Singhal
Bhavya Aggarwal
Bhawna Sharma
Piyush Rana
Pawan Bisht
Shubham Verma
Charmy Garg
Harshit Daga
Rudar
Deepak Mehra
Deepankar
Divya Dua
Ayush Tiwari
girdharshubham
Girish Bharti
Harmeet Singh(Taara)
Himani Arora
Himanshu Gupta
Himanshu Rajput
Sonu Mehrotra
Nancy Jain
Jasmine Kaur
Jatin Demla
Rachel Jones
JustinB
Jyoti Sachdeva
Rishi Khandelwal
Manish Mishra
Vikas Hazrati
Kritika Khandelwal
Sahil Sawhney
Megha Jatana
Mahesh Chand
Manjot Kaur
Mayank Bairagi
Mehak Juneja
Miral Gandhi
muditchhabra6125
Mukesh Kumar
Narayan Kumar
Neelam Lakra
Neelu Aggarwal
Neha Bhardwaj
Neelaksh
Nitin Arora
Pallavi Singh
Prabhat Kashyap
Pranjut Gogoi
Prashant Sharma
Anubhav Tarar
Priyanka Thakur
Rahul Kumar
Ramandeep
Randhir kumar
Joseph Ross
Ruchika Dubey
Sajit Gupta
sakshigupta2312
Rajeev Thakur
Shivangi Gupta
Shubham Dangare
shubhamgoyal01
Sidharth Khattri
Sachin Slathia
Sourabh Verma
GauravShukla
Shubhrank Rastogi
Sudeep James Tirkey
Nitin Aggarwal
kunal sethi
Vandana Yadav
Vidisha Gupta
Vinay Kumar
Vinisha Sharma
Yatharth Sharma
Knoldus is the world's largest pure-play Scala and Spark company. We modernize enterprise through cutting-edge digital engineering by leveraging Scala, Functional Java and Spark ecosystem. Our mission is to provide reactive and streaming fast data solutions that are message-driven, elastic, resilient, and responsive.
+(1) 647-467-4396
hello@knoldus.com
