Advertisements

Apache Spark

Kryo Serialization in Spark

Pinku SwargiaryApache Spark, Scala, ,

Reading Time: 4 minutes Spark provides two types of serialization libraries: Java serialization (default) Kryo serialization For faster serialization and deserialization spark itself recommends to use Kryo serialization in any network-intensive application. Then why is it not set to default : Why Kryo is not set to default in Spark? The only reason Kryo is not set to default is because it requires custom registration. Although, Kryo is supported Continue Reading

Advertisements

Understanding Spark’s Logical and Physical Plan in layman’s term

Sarfaraz HussainApache Spark, Big Data and Fast Data, Scala, Spark, SQL, , , , ,

Reading Time: 5 minutes This blog pertains to Apache SPARK 2.x, where we will find out how Spark SQL works internally in layman’s terms and try to understand what is Logical and Physical Plan. Also we will be looking into Catalyst Optimizer. So let’s get started. First let’s see what Apache Spark is. The official definition of Apache Spark says that “Apache Spark™ is a unified analytics engine for large-scale Continue Reading

Apache Spark

Deep Dive into Apache Spark Transformations and Action

Divyansh JainApache Spark, Big Data and Fast Data, Cloud, Database, HDFS, linux, NoSql, Scala, Spark, SQL, Streaming, Streaming Solutions, , , , , , , , , ,

Reading Time: 4 minutes In our previous blog of Apache Spark, we discussed a little about what Transformations & Actions are? Now we will get deeper into the topic and will understand what actually they are & how they play a vital role to work with Apache Spark? What is Spark RDD? Spark introduces the concept of an RDD (Resilient Distributed Dataset), an immutable fault-tolerant, distributed collection of objects Continue Reading

Diving deeper into Delta Lake

Kaushik NathApache Kafka, Apache Spark, Big Data and Fast Data, github, Scala, Spark, Streaming, Tech Blogs, , , , , ,

Reading Time: 6 minutes Delta Lake is an open-source storage layer that brings reliability to data lakes. It has numerous reliability features including ACID transactions, scalable metadata handling, and unified streaming and batch data processing.

Delta Lake To the Rescue

Anmol SarnaApache Spark, Big Data and Fast Data, Java, python, Scala, Spark, Streaming, , , ,

Reading Time: 4 minutes Welcome Back. In our previous blogs, we tried to get some insights about Spark RDDs and also tried to explore some new things in Spark 2.4. You can go through those blogs here: RDDs – The backbone of Apache Spark Spark 2.4: Adding a little more Spark to your code In this blog, we will be discussing something called a Delta Lake. But first, let’s Continue Reading

Spark – Actions and Transformations

Shubham VermaApache Spark, Big Data and Fast Data, cluster, HDFS, Scala, Spark, Streaming, Streaming Solutions

Reading Time: 4 minutes Hey guys, welcome to series of spark blogs, this blog being the first blog in this series we would try to keep things as crisp as possible, so let’s get started. So I recently get to start learning spark about believe me and now it has made me inquisitive about it, for a brief introduction of spark, I would say that it is a pretty Continue Reading

Tale of Apache Spark

Divyansh JainApache Spark, Big Data and Fast Data, cluster, Database, ML, AI and Data Engineering, NoSql, Scala, Spark, SQL, Streaming, Streaming Solutions, Tutorial, , , , , , , , , ,

Reading Time: 6 minutes Data is being produced extensively in today’s world and it is going to be generated more rapidly in future. 90% of total data that is produced in the world is produced in last two years only and it is estimated that in 2020 world’s total data would reach 45 ZB and data generated each day would be enough that if we try to store it Continue Reading

Big Data Evolution: Migrating on-premise database to Hadoop

Divyansh JainApache Spark, Big Data and Fast Data, HDFS, Scala, Tableau, , , , , , , , , , , , , , ,

Reading Time: 4 minutes We are now generating massive volumes of data at an accelerated rate. To meet business needs, address changing market dynamics as well as improve decision-making, sophisticated analysis of this data from disparate sources is required. The challenge is how to capture, store and model these massive pools of data effectively in relational databases. Big data is not a fad. We are just at the beginning Continue Reading

Using Vertica with Spark-Kafka: Write using Structured Streaming

Anuj SaxenaApache Kafka, Apache Spark, Big Data and Fast Data, Functional Programming, HDFS, Scala, Spark, Streaming, Streaming Solutions, , , , , , , , , , , ,

Reading Time: 3 minutes In two previous blogs, we explored about Vertica and how it can be connected to Apache Spark. The first blog in this mini series was about reading data from Vertica using Spark and saving that data into Kafka. The next blog explained the reverse flow i.e. reading data from Kafka and writing data to Vertica but in a batch mode. i.e reading data from Kafka Continue Reading

Using Vertica with Spark-Kafka: Writing

Anuj SaxenaApache Kafka, Apache Spark, Database, HDFS, Scala, Spark, , , , , , , , ,

Reading Time: 4 minutes In previous blog of this series, we took a glance over the basic definition of Spark and Vertica. We also did a code overview for reading data from Vertica using Spark as DataFrame and saving the data into Kafka. In this blog we will be doing the reverse flow i.e. working on reading the data from Kafka as a DataFrame and writing that DataFrame into Continue Reading

Using Vertica with Spark-Kafka: Reading

Anuj SaxenaApache Kafka, Apache Spark, Big Data and Fast Data, Database, Scala, SQL, , , , , , ,

Reading Time: 4 minutes We live in a world of Big data where the size of data is so big even for small results. This is the result of an increase in data collection on a rapid scale in the modern world. This massiveness of data brings the requirements of such tools which can work upon such a big chunk of data. I am pretty sure that you guys Continue Reading

Do you really need Spark? Think Again!

RamandeepApache Spark, Big Data and Fast Data, Functional Programming, ML, AI and Data Engineering, Scala, Spark, Tech Blogs, , , , , ,

Reading Time: 5 minutes With the massive amount of increase in big data technologies today, it is becoming very important to use the right tool for every process. The process can be anything like Data ingestion, Data processing, Data retrieval, Data Storage, etc. Today we are going to focus on one of those popular big data technologies i.e., Apache Spark. Apache Spark is an open-source distributed general-purpose cluster-computing framework. Spark Continue Reading

Spark: Introduction to Datasets

Ayush HoodaApache Spark, Big Data and Fast Data, Scala, Spark, , , , ,

Reading Time: 3 minutes As I have already discussed in my previous blog Spark: RDD vs DataFrames about the shortcomings of RDDs and how DataFrames overcome them. Now we’ll try to have a look at the shortcomings of DataFrames and how Dataset APIs can overcome them. DataFrames:- A DataFrame is a distributed collection of data, which is organized into named columns. Conceptually, it is equivalent to the relational tables with Continue Reading

Knoldus Pune Careers - Hiring Freshers

Get a head start on your career at Knoldus. Join us!

Knoldus logo

Knoldus is the world's largest pure-play Scala and Spark company. We modernize enterprise through cutting-edge digital engineering by leveraging Scala, Functional Java and Spark ecosystem. Our mission is to provide reactive and streaming fast data solutions that are message-driven, elastic, resilient, and responsive.

About Knoldus

Recent Posts

Subscribe to our newsletter

Contact us

+(1) 647-467-4396
hello@knoldus.com

Partners

  • Knoldus Lightbend
  • Knoldus Databricks
  • Knoldus IBM
  • Knoldus Datastax
  • Knoldus Confluent
Copyright 2017-19 © Knoldus
Upcoming Webinar :

How recommendation systems drive business success in Retail? Register Now!