Stream Processing Archives

Flink: Time Windows based on Processing Time

September 9, 2020October 22, 2020Apache Flink, Big Data and Fast Data, Flink, Java, ML, AI and Data Engineering#apache flink, #apache-flink, #flink, DataStream API, Flink, Flink Streaming, real time streaming data, Stream Processing, Window Functions

Reading Time: 4 minutes In the previous blog, we talked about Flink’s windows operator, a heart of processing infinite streams. Generally in Flink, after specifying that the stream is keyed or non keyed, the next step is to define a window assigner. The window assigner defines how elements are assigned to windows. Flink provides some useful predefined window assigners like Tumbling windows, Sliding windows, Session windows, Count windows, and Continue Reading

Basic Anatomy of a Flink Program

September 2, 2020October 22, 2020Apache Flink, Big Data and Fast Data, Flink, Java#apache flink, #flink, Big Data, Big Data Analytics, fast data, Flink, Stream Processing, Streaming

Reading Time: 3 minutes Hi Folks! Hope you all are safe in the COVID-19 pandemic and learning new tools and tech while staying at home. I also have just started learning a very prominent Big Data framework for stream processing which is Flink. Flink is a distributed framework and based on the streaming first principle, means it is a real streaming processing engine and implements batch processing as a special case. In Continue Reading

Windows operator: Heart of processing infinite streams in Flink

August 24, 2020August 24, 2020Apache Flink, Big Data and Fast Data, Flink, Scala, Streaming, Streaming Solutions#apache flink, Big Data, Big Data Analytics, Flink, Flink Streaming, Stream Processing, Streaming

Reading Time: 3 minutes Apache Flink is an open-source, distributed, Big Data framework for stream and batch data processing. Flink is based on the streaming first principle which means it is a real streaming processing engine and implements batching as a special case. Flink is considered to have a heart and it is the “Windows” operator. It makes Flink capable of processing infinite streams quickly and efficiently. Windows split Continue Reading

Streaming from Kafka to PostgreSQL through Spark Structured Streaming

May 4, 2020May 4, 2020Apache Kafka, Apache Spark, ScalaApache Kafka, Apache Spark, postgreSQL, scala, Spark Structured Streaming, Stream Processing

Reading Time: 3 minutes Hello everyone, in this blog we are going to learn how to do a structured streaming in spark with kafka and postgresql in our local system. We will be doing all this using scala so without any furthur pause, lets begin. Setting up the necessities first: Dependencies Set up the required dependencies for scala, spark, kafka and postgresql. 2. PostgreSQL setup Lets start fresh by Continue Reading

Delta Lake To the Rescue

October 7, 2019October 7, 2019Apache Spark, Big Data and Fast Data, Java, python, Scala, Spark, StreamingApache Spark, batch processing, Big Data, delta lake, Stream Processing

Reading Time: 4 minutes Welcome Back. In our previous blogs, we tried to get some insights about Spark RDDs and also tried to explore some new things in Spark 2.4. You can go through those blogs here: RDDs – The backbone of Apache Spark Spark 2.4: Adding a little more Spark to your code In this blog, we will be discussing something called a Delta Lake. But first, let’s Continue Reading

Flinkathon: What makes Flink better than Kafka Streams?

April 16, 2019April 16, 2019Apache Flink, Apache Kafka, Big Data and Fast Data, cluster, Flink, Scala, Streaming, Streaming SolutionsApache Kafka, Flink, Flink Streaming, kafka, Kafka Streaming, Kafka Streams, Stream Processing, Streaming, streaming data

Reading Time: 2 minutes Initially, I would like you all to focus on a few questions before comparing the frameworks:1. Is there any comparison or similarity between Flink and the Kafka?2. What could be better in Flink over the Kafka?3. Is it the problem or system requirement to use one over the other? Before talking about the Flink betterment and use cases over the Kafka, let’s first understand their Continue Reading

Kafka Streams : More Than Just a Dumb Storage

July 21, 2017Apache Kafka, ScalaKafka Streams, Real Time Processing, Stream Processing, Streams

Reading Time: 5 minutes Whenever we hear the word Kafka, all we think about it as a messaging system with a publisher-subscriber model that we use for our streaming applications as a source and a sink. So we can say that Kafka is just a dumb storage system that stores the data provided by a producer for a long time (configurable) and it can provide it to some consumer whenever Continue Reading