Axon vs Kafka

Table of contents

Reading Time: 4 minutes

Introduction

One of the most common discussion points that come up regularly in interactions with Customers/Prospects/Forums is how does Axon compare to Apache Kafka? Can one do the job of the other? Are they complementary to each other? Can they work together? Does Axon provide any capabilities to work with Kafka?

Apache Kafka is a very popular system for publishing and consuming events. Its architecture is fundamentally different from most messaging systems and combines speed with reliability.

Axon provides an extension dedicated to publishing and receiving event messages from Kafka. The Kafka Extension should be regarded as an alternative approach to distributing events, besides (the default) Axon Server.

Implementing Event Sourcing

Kafka – Event Streaming, not Event Sourcing

Initially developed at LinkedIn and currently under the Apache foundation, Kafka pioneered the adoption of the concept of Event Streaming which essentially centers around 3 main capabilities:

Implementing Event Sourcing with Kafka requires an optimal design of Kafka topics that serve as the Event Store for appending and reading Aggregate events. We could start with a basic design of a single partitioned topic per Aggregate instance. This would guarantee ordered appends for the events and reading the state could be as simple as reading the topic from offset 0. It becomes quite obvious that this would hit problems of scale once we start having millions of Aggregate instances as there will be just too many topics. Another design could be to have a single partitioned topic for all Aggregate types. This causes the reading of events to be extremely slow as the entire data set of Aggregate types and instances need to scan.

We could add a database to the mix to provide some of these capabilities but then we would run into the distributed transactions problem to guarantee writes across resources. To solve this issue, we could use another tool from the platform – Kafka Streams. Streams provide the concept of state stores that could store the Aggregate’s event stream as snapshots. While this solves the problem of distributed stores as well as the read operations for constructing an Aggregate state. Since it only stores the current snapshot, it cannot be used to build your read models.

The API provided by Kafka is limited only to a certain subset of operations required for Event Sourcing. It becomes fairly and quickly obvious that Kafka does not provide us the full range of capabilities. Which is to implement an efficient, robust, and scalable Event Sourcing Infrastructure.

Axon – Event Sourcing, and more

Axon is the leading platform built to cater exclusively to implementing Event-Driven Architecture, realizing that such architecture requires more than just Events.

In addition to providing a highly scalable Event Store, Axon elevates the concept of traditional CQRS/Event Sourcing architectures by treating every operation within an application (Commands/Queries and Events) as messages. Each of these message types requires a different routing strategy which Axon provides support for.

Axon provides two main components

Axon Server – A highly scalable, distributed, and purpose-built Event Store and zero-configuration Message Router. It routes messages based on the capabilities each application. It provided when connecting, taking the specific routing requirements of each type of message into account. Events, which carry value for an extended period of time, are stored for the purpose of Event Sourcing. And immediately made available for event streaming.
Axon Framework – Implements the full range of API capabilities required for Event Sourcing / Message Routing operations. It provides the building blocks required to deal with all the non-functional requirements, allowing developers to focus on the functional aspects of their application instead.

Axon has customers across a wide range of industries who use it to roll out an Enterprise-Grade Event-Driven Microservices infrastructure.

Combining Axon and Kafka

As we have seen above, implementing Event Sourcing from scratch is complex and it does require a purpose-built enterprise-grade platform. At first glance, the Kafka Platform does look like an obvious choice for Event Sourcing, and customers currently utilizing it as their Event Processing Infrastructure see it as a natural extension to support and roll out an Event Sourcing Infrastructure too. But as the various Event Sourcing concerns start getting implement. It becomes clear that Kafka does not provide it out of the box or simply might not support it. Is this a limitation of Kafka? Absolutely not, Kafka was built with the purpose of being an Event Streaming platform and it shines in that role.

However, we do see enterprises going down the path of extending Kafka to serve the purpose of an Event Sourcing Infrastructure. What it generally leads to is that these non-supported concerns are built around Kafka which might not be a very good idea. These concerns are quite complex and require significant effort, time, and knowledge. Enterprises are better off utilizing a purpose-built platform like Axon for Event Sourcing. It is similar to the choice of utilizing Kafka as your Event Infrastructure, and you are not going to roll out your own!

Strength :

Axon’s strength is in the application-level messaging, where services need to coordinate their activities on a more detailed level – for example, services that work together in the Order Fulfillment Domain. These activities lead to events, some of which are more important than others. These so-called Milestone Events are typically worth publishing beyond the scope of these cooperating services. That’s where Kafka’s strength is.

To help integrate with Kafka, Axon provides a Kafka connector as part of the Axon Framework. The purpose of the connector is to help utilize the best of both worlds. Which use Axon for its purpose-built Event Sourcing Capabilities and Command/Query Message processing while delegating the responsibility of event delivery to downstream systems utilizing Kafka.

The depiction of the integration below with multiple Axon applications relying on Kafka for event delivery to downstream systems

Axon and Kafka – two different purposes

In summary, Axon and Kafka serve two different purposes within the Event-Driven Architecture. Space – Axon provides the application-level support for domain modeling and Event Sourcing. As well as the routing of Commands, Events, and Queries, while Kafka shines as an Event Streaming platform. The combination of both acts as a compelling proposition as it helps customers retain their existing investment in Kafka while at the same time utilizing Axon to roll out a robust, Event-Driven Microservices infrastructure.