Flinkathon: Why Flink is better for Stateful Streaming applications?

Stream processing is a way to query a continuous stream of data and draw conclusions from it within the boundaries of a real-time scenario. For example, receiving an alert as soon as a fraudulent transaction is done via a credit/debit card.

The 2 main types of stream processing done are:

1. Stateless: Where every event is handled completely independent from the preceding events.
2. Stateful: Where a “state” is shared between events and therefore past events can influence the way current events are processed.

Stateless stream processing is easy to scale up because events are processed independently. But Stateful stream processing is difficult to scale up because the “state” needs to be shared across the events.

In order to do stateful stream processing, there are a number of open source stream processing frameworks available like Apache’s Spark Streaming, Storm, Kafka Streams, Samza, Flink, and Apex.

In this post, we will try to answer the question: Why Flink is better for Stateful Streaming applications? We will not be presenting any comparison between the different stream processing frameworks, instead, we will just explain the features of Apache Flink, which makes it suitable for stateful stream processing applications.

So let’s first begin by answering the question: What is Flink? Apache Flink is an open source platform for distributed stream and batch data processing. Flink’s core is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams.

Now, let’s answer the question: Why Flink is better suited for Streaming/Stateful Streaming applications?

The answer is simple: Because the “state” is a first-class citizen in Flink. Now the reasons which make the “state” a first-class citizen in Flink are as follows:

1. Multiple State Primitives: Flink provides the “state” primitives for different data structures, such as atomic values, lists, or maps. This helps developers to choose the “state” primitive that is most efficient based on their use case.
2. Pluggable State Backends: Flink allows us to manage the “state” via any pluggable “state” backend like a Database or a File System.
3. Exactly-once state consistency: Flink’s checkpointing and recovery algorithms guarantee the consistency of the “state” in case of a failure. Hence, failures are transparently handled and do not affect the correctness of an application.
4. Very Large State: Flink can maintain a “state” of several terabytes in size with the help of its asynchronous and incremental checkpointing algorithm. So, the developers need not worry about the size of the “state” generated or maintained by the application.
5. Scalable Applications: Flink supports scaling of stateful applications by redistributing the “state” to its worker machines. Hence, developers need not worry about scaling stateful streaming applications too.

Now, What’s the conclusion? Simple, all the above reasons make Flink a suitable candidate for being the stream processing engine for stateful streaming applications. So, if you are looking for a stream processing engine for your application, then give Flink a try.