Knoldus Blogs

Flink: Time Windows based on Processing Time

September 9, 2020October 22, 2020Apache Flink, Big Data and Fast Data, Flink, Java, ML, AI and Data Engineering#apache flink, #apache-flink, #flink, DataStream API, Flink, Flink Streaming, real time streaming data, Stream Processing, Window Functions

Reading Time: 4 minutes In the previous blog, we talked about Flink’s windows operator, a heart of processing infinite streams. Generally in Flink, after specifying that the stream is keyed or non keyed, the next step is to define a window assigner. The window assigner defines how elements are assigned to windows. Flink provides some useful predefined window assigners like Tumbling windows, Sliding windows, Session windows, Count windows, and Continue Reading

Reading Avro files using Apache Flink

June 24, 2020June 24, 2020Apache Flink, Flink, Streaming, Streaming Solutions, Studio-Scala#apache-flink, #avro files, apache, avro, Flink, programming, scala

Reading Time: 2 minutes In this blog, we will see how to read the Avro files using Flink. Before reading the files, let’s get an overview of Flink. There are two types of processing – batch and real-time. Batch Processing: Processing based on the data collected over time. Real-time Processing: Processing based on immediate data for an instant result. Real-time processing is in demand and Apache Flink is the Continue Reading

Using Apache Flink for Kinesis to Kafka Connect

June 24, 2020June 24, 2020Apache Flink, Flink, Studio-Scala#apache-flink, #kinesis, apache, Flink Streaming, kafka, scala

Reading Time: 3 minutes In this blog, we are going to use kinesis as a source and kafka as a consumer. Let’s get started. Step 1: Apache Flink provides the kinesis and kafka connector dependencies. Let’s add them in our build.sbt: Step 2: The next step is to create a pointer to the environment on which this program runs. Step 3: Setting parallelism of x here will cause all Continue Reading