Flinkathon: What makes Flink better than Kafka Streams?

Initially, I would like you all to focus on a few questions before comparing the frameworks:
1. Is there any comparison or similarity between Flink and the Kafka?
2. What could be better in Flink over the Kafka?
3. Is it the problem or system requirement to use one over the other?

Before talking about the Flink betterment and use cases over the Kafka, let’s first understand their similarities:
1. Both guarantee exactly once semantics.
2. Both provide stateful operations.
3. Both provide High Availablity (Flink provides through zookeeper).
4. Both have SQL support and functionality.

Seems like both the frameworks are well capable of achieving or solving the stateful and streaming problems, but there is a huge difference in respect of following areas:

1. Deployment – while Kafka provides Stream APIs (a library) which can be integrated and deployed with the existing application (over cluster tools or standalone), whereas Flink is a cluster framework, i.e. it takes care of deploying the application, either in standalone Flink clusters, or using YARN, Mesos, or containers (Docker, Kubernetes).

2. Bounded and unbounded Streams – as we all know Kafka only support unbounded streams while Flink has provided the support for processing bounded streams as well by integrating streaming with micro batch processing,

3. Fault tolerance – Flink provides robust fault-tolerance using checkpointing (periodically saving internal state to external sources such as HDFS), while for Stream API it is managed and configured along with Kafka, not with Stream application.

4. Latency – No doubt Flink is much faster due to it’s architecture and cluster deployment mechanism, Flink throughput in the order of tens of millions of events per second in moderate clusters, sub-second latency that can be as low as few 10s of milliseconds.

5. Maintained By – as Flink application is deployed on the cluster, hence it is owned and maintained by data infrastructure or BI team while the Kafka Streams are integrated within the application hence it is done by the business team that manages the respective application.

6. Data Source & Sink – Flink can have kafka, external files, other messages queue as source of data stream, while Kafka Streams are bounded with Kafka topics for source, while for sink or output of the result both can have kafka, external files, DBs, but Flink can push to other Message queues as well.

Conclusion – As we have seen both have different ways of deployment, differences in their architecture and use cases while solving the business problems, but in term of throughput, latency, fault tolerance, integrating with other frameworks, Flink rules over the Kafka Streams, while with later one we don’t need to worry about the configurations and fault tolerance which is handled by the Kafka cluster itself, and makes it easy to integrate stream processing within the application through the APIs.