Knoldus Blogs

Kafka Connect Fundamentals

April 18, 2022April 18, 2022Apache KafkaDataStream API, kafka, Kafka connect

Reading Time: 3 minutes Kafka connect is an framework to connect kafka with external ecosystem like file systems, databases using kafka connector. The Kafka Connect cluster supports running and scaling out connectors. Kafka Connectors are ready-to-use components, which can help us to import data from external systems into Kafka topics and export data from Kafka topics into external systems. What is Kafka Connect? Kafka connect is use to perform stream integration Continue Reading

Introduction to Java Stream API

June 25, 2021June 25, 2021JavaDataStream API, Functional java, Functional Programming, java, Streams

Reading Time: 3 minutes Hi friends in this blog we are going to cover streams, parallel streams and some of the popular function associated with them with example. Introduction Streams are wrappers around data source, which make data processing in bulk easier and fast. A stream is not a data structure i.e. it does not store data. Stream is defined in java.util.stream package. Different ways to create Streams 1) Continue Reading

A Quick Demo: Kafka to Flink to Cassandra

December 19, 2020December 21, 2020Apache Flink, Apache Kafka, Big Data and Fast Data, Cassandra, Database, Flink, Functional Programming, NoSql, Studio-Scala#apache flink, Apache Kafka, Cassandra, data analysis, DataStream API, Flink, Flink Streaming, pipeline, Streaming, streaming analytics, streaming data

Reading Time: 3 minutes Hi Folks!! In this blog, we are going to learn how we can integrate Flink with Kafka and Cassandra to build a simple streaming data pipeline. Apache Flink is a framework and distributed processing engine. it is used for stateful computations over unbounded and bounded data streams.Kafka is a scalable, high performance, low latency platform. It allows reading and writing streams of data like a messaging system.Cassandra: A distributed and wide-column Continue Reading

Flink: Implementing the Count Window

September 10, 2020September 10, 2020Apache Flink, Big Data and Fast Data, Flink, Functional Programming, Java#apache flink, Big Data, Big Data Analytics, DataStream API, Flink Streaming, Window Functions

Reading Time: 3 minutes In the blog, we learned about Tumbling and Sliding windows which is based on time. In this blog, we are going to learn to define Flink’s windows on other properties i.e Count window. As the name suggests, count window is evaluated when the number of records received, hits the threshold. Count window set the window size based on how many entities exist within that window. For example, if we fixed the count Continue Reading

Flink: Time Windows based on Processing Time

September 9, 2020October 22, 2020Apache Flink, Big Data and Fast Data, Flink, Java, ML, AI and Data Engineering#apache flink, #apache-flink, #flink, DataStream API, Flink, Flink Streaming, real time streaming data, Stream Processing, Window Functions

Reading Time: 4 minutes In the previous blog, we talked about Flink’s windows operator, a heart of processing infinite streams. Generally in Flink, after specifying that the stream is keyed or non keyed, the next step is to define a window assigner. The window assigner defines how elements are assigned to windows. Flink provides some useful predefined window assigners like Tumbling windows, Sliding windows, Session windows, Count windows, and Continue Reading

Creating Data Pipeline with Spark streaming, Kafka and Cassandra

August 24, 2020October 22, 2020Apache Kafka, Apache Spark, Big Data and Fast Data, Cassandra, MessagesAPI, Spark, Streaming, Studio-ScalaApache Kafka, Apache Spark, Cassandra, data analysis, DataStream API

Reading Time: 3 minutes Hi Folks!! In this blog, we are going to learn how we can integrate Spark Structured Streaming with Kafka and Cassandra to build a simple data pipeline. Spark Structured Streaming is a component of Apache Spark framework that enables scalable, high throughput, fault tolerant processing of data streams.Apache Kafka is a scalable, high performance, low latency platform that allows reading and writing streams of data Continue Reading

Spark: Streaming Datasets

August 18, 2020August 18, 2020Apache Spark, Big Data and Fast Data, Spark, Streaming, Studio-ScalaApache Spark, Big Data Analytics, DataFrame, DataStream API, Spark Streaming, Spark Structured Streaming, Streaming Spark

Reading Time: 3 minutes Spark providing us a high-level API – Dataset, which makes it easy to get type safety and securely perform manipulation in a distributed and a local environment without code changes. Also, spark structured streaming, a high-level API for stream processing allows us to stream a particular Dataset which is nothing but a type-safe structured streams. In this blog, we will see how we can create Continue Reading

Flinkathon: First Step towards Flink’s DataStream API

April 20, 2019April 20, 2019Apache Flink, Apache Kafka, Big Data and Fast Data, Flink, Streaming, Streaming Solutions, Studio-ScalaDataStream API

Reading Time: 3 minutes In our previous blog posts: Flinkathon: Why Flink is better for Stateful Streaming applications? Flinkathon: What makes Flink better than Kafka Streams? We saw why Apache Flink is a better choice for streaming applications. In this blog post, we will explore how easy it is to express a streaming application using Apache Flink’s DataStream API. DataStream API DataStream API is used to develop regular programs Continue Reading