watermark

Spark Structured Streaming (Part 4) – Handling Late Data

August 18, 2020August 19, 2020Analytics, Apache Spark, Big Data and Fast Data, ML, AI and Data Engineering, Spark, Streaming, Streaming Solutions, Studio-Scala, Tech Blogsstateful streaming, Structured Streaming, watermark

Reading Time: 3 minutes Welcome back folks to this blog series of Spark Structured Streaming. This blog is the continuation of the earlier blog “Understanding Stateful Streaming“. And this blog pertains to Handling Late Arriving Data in Spark Structured Streaming. So let’s get started. Handling Late Data With window aggregates (discussed in the previous blog) Spark automatically takes cares of late data. Every aggregate window is like a bucket Continue Reading