Structured Streaming Archives

Spark Structured Streaming (Part 4) – Handling Late Data

August 18, 2020August 19, 2020Analytics, Apache Spark, Big Data and Fast Data, ML, AI and Data Engineering, Spark, Streaming, Streaming Solutions, Studio-Scala, Tech Blogsstateful streaming, Structured Streaming, watermark

Reading Time: 3 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Understanding Stateful Streaming“. And this blog pertains to Handling Late Arriving Data in Spark Structured Streaming. So let’s get started. Handling Late Data With window aggregates (discussed in the previous blog) Spark automatically takes cares of late data. Every aggregate window is like a bucket Continue Reading

Spark Structured Streaming (Part 3) – Stateful Streaming

August 14, 2020August 21, 2020Analytics, Apache Spark, Big Data and Fast Data, ML, AI and Data Engineering, Spark, Streaming, Streaming Solutions, Studio-Scala, Tech Blogsstateful streaming, stateful streaming scala, Structured Streaming

Reading Time: 4 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Internals of Structured Streaming“. And this blog pertains to Stateful Streaming in Spark Structured Streaming. So let’s get started. Let’s start from the very basic understanding of what is Stateful Stream Processing. But to understand that, let’s first understand what Stateless Stream Processing is. In Continue Reading

Spark Structured Streaming (Part 2) – The Internals

August 9, 2020August 14, 2020Analytics, Apache Spark, Big Data and Fast Data, ML, AI and Data Engineering, Spark, Streaming, Streaming Solutions, Studio-Scala, Tech BlogsStructured Streaming

Reading Time: 2 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Introduction to Structured Streaming“. So I’ll exactly start from the point where I left in the previous blog. Structure of Streaming Query When we call start() API, Spark internally translates this code into a Logical Plan (an abstract representation of what the code does), then Continue Reading

Spark Structured Streaming (Part 1) – Introduction

August 6, 2020October 12, 2020Analytics, Apache Spark, Big Data and Fast Data, ML, AI and Data Engineering, Spark, Streaming, Streaming Solutions, Studio-Scala, Tech BlogsStructured Streaming

Reading Time: 5 minutes In this Spark Structured Streaming series of blogs, we will have a deep look into what structured streaming is in a very layman language. So let’s get started. Introduction Structured streaming is a stream processing engine built on top of the Spark SQL engine and uses the Spark SQL APIs. It is fast, scalable and fault-tolerant. It provides rich, unified and high-level APIs in the Continue Reading

Using Vertica with Spark-Kafka: Write using Structured Streaming

July 3, 2019July 16, 2019Apache Kafka, Apache Spark, Big Data and Fast Data, Functional Programming, HDFS, Spark, Streaming, Streaming Solutions, Studio-ScalaApache Kafka, Apache Spark, DataFrame, Kafka Spark, Spark, Spark SQL, spark sql kafka, Spark Structured Streaming, Spark to Vertica, Streaming, Structured Streaming, Vertica, Write to vertica

Reading Time: 3 minutes In two previous blogs, we explored about Vertica and how it can be connected to Apache Spark. The first blog in this mini series was about reading data from Vertica using Spark and saving that data into Kafka. The next blog explained the reverse flow i.e. reading data from Kafka and writing data to Vertica but in a batch mode. i.e reading data from Kafka Continue Reading

Knolx: Structured Streaming in Spark

April 15, 2019Functional Programming, Studio-Scaladataframes, datasets, knolx, RDD, Structured Streaming

Reading Time: < 1 minute Knoldus has organized a session on 08th February 2019. The topic was “Understanding Spark Structured Streaming”. Many people attended and enjoyed the session. In this blog post, I am going to share the slides & video of the session. Slides: Video: If you have any query, then please feel free to comment below.

Spark: Introduction to Datasets

March 4, 2019Apache Spark, Big Data and Fast Data, Spark, Studio-ScalaBig Data, dataframes, datasets, RDDs, Spark, Structured Streaming

Reading Time: 3 minutes As I have already discussed in my previous blog Spark: RDD vs DataFrames about the shortcomings of RDDs and how DataFrames overcome them. Now we’ll try to have a look at the shortcomings of DataFrames and how Dataset APIs can overcome them. DataFrames:- A DataFrame is a distributed collection of data, which is organized into named columns. Conceptually, it is equivalent to the relational tables with Continue Reading

Spark Streaming vs. Structured Streaming

February 28, 2019Apache Spark, Big Data and Fast Data, Spark, Streaming, Streaming Solutions, Studio-ScalaApache Spark, Spark Streaming, Spark Structured Streaming, Streaming, Streaming Spark, Structured Streaming

Reading Time: 6 minutes Fan of Apache Spark? I am too. The reason is simple. Interesting APIs to work with, fast and distributed processing, unlike map-reduce no I/O overhead, fault tolerance and many more. With this much, you can do a lot in this world of Big data and Fast data. From “processing huge chunks of data” to “working on streaming data”, Spark works flawlessly in all. In this Continue Reading

Spark Structured Streaming with Elasticsearch

September 25, 2018Apache Spark, Big Data and Fast Data, Elasticsearch, Streaming, Streaming Solutions, Studio-ScalaApache Spark, elasticsearch, Elasticsearch with structured streaming, Streaming, Structured Streaming

Reading Time: 3 minutes There’s been a lot of time we have been working on streaming data. Using Apache Spark for that can be much convenient. Spark provides two APIs for streaming data one is Spark Streaming which is a separate library provided by Spark. Another one is Structured Streaming which is built upon the Spark-SQL library. We will discuss the trade-offs and differences between these two libraries in Continue Reading

Structured Streaming: Philosophy behind it

June 16, 2018July 20, 2018Apache Spark, Big Data and Fast Data, Spark, Streaming, Studio-ScalaSpark Structured Streaming, Structured Streaming

Reading Time: 3 minutes In our previous blogs: Structured Streaming: What is it? & Structured Streaming: How it works? We got to know 2 major points about Structured Streaming – It is a fast, scalable, fault-tolerant, end-to-end, exactly-once stream processing API that helps users in building streaming applications. It treats the live data stream as a table that is being continuously appended/updated which allows us to express our streaming computation as Continue Reading

Spark Structured Streaming: A Simple Definition

June 18, 2017July 20, 2018Spark, Streaming, Studio-ScalaApache Spark, Big Data, continuous application, spark 2.0, Structured Streaming

Reading Time: 3 minutes “Structured Streaming”, nowadays we are hearing this term in Apache Spark ecosystem quite a lot, as it is being preached as next big thing in scalable big data world. Although, we all know that Structured Streaming means a stream having structured data in it, but very few of us knows what exactly it is and where we can use it. So, in this blog post Continue Reading