Knoldus Blogs

Dataframe and Datasets: Apache Spark’s Developers Friendly Structured APIs

June 13, 2022June 15, 2022Studio-Scalaapache, Apache Spark, API, Big Data, CSV, Database, DataFrame, dataframes, scala, Spark, technology

Reading Time: 4 minutes This is a two-part blogs in which first we’ll be covering Dataframe API and in the second part Datasets. Spark 2.x introduced the concept of structuring the spark by introducing two concepts: – to express some computation by using common patterns found in data analysis, such as filtering, selecting, counting, aggregating, and grouping. And the second one of order and structure your data in a Continue Reading

Knolx: Structured Streaming in Spark

April 15, 2019Functional Programming, Studio-Scaladataframes, datasets, knolx, RDD, Structured Streaming

Reading Time: < 1 minute Knoldus has organized a session on 08th February 2019. The topic was “Understanding Spark Structured Streaming”. Many people attended and enjoyed the session. In this blog post, I am going to share the slides & video of the session. Slides: Video: If you have any query, then please feel free to comment below.

Spark: Introduction to Datasets

March 4, 2019Apache Spark, Big Data and Fast Data, Spark, Studio-ScalaBig Data, dataframes, datasets, RDDs, Spark, Structured Streaming

Reading Time: 3 minutes As I have already discussed in my previous blog Spark: RDD vs DataFrames about the shortcomings of RDDs and how DataFrames overcome them. Now we’ll try to have a look at the shortcomings of DataFrames and how Dataset APIs can overcome them. DataFrames:- A DataFrame is a distributed collection of data, which is organized into named columns. Conceptually, it is equivalent to the relational tables with Continue Reading

The Dominant APIs of Spark: Datasets, DataFrames and RDDs

April 16, 2017April 17, 2017SparkApache Spark, dataframes, datasets, performance optimization, RDD, space optimization, spark apis

Reading Time: 4 minutes While working with Spark often we come across the three APIs: DataFrames, Datasets and RDDs. In this blog I will discuss the three in terms of use case, performance and optimization. It is essential to keep in mind that there is seamless transformation available between the three DataFrames, Datasets and RDDs. Implicitly the RDD forms the apex of both DataFrame and Datasets. The inception of Continue Reading