Security & SSL Setup in Confluent Kafka

Reading Time: 2 minutes

What is SSL ?

Secure Socket Layer (SSL) is a security protocol for the transport layer. In SSL Protocol data is divided into fragments. The fragments are compressed and encrypted Message Authentication Code (MAC) generated by algorithms like Secure Hash Protocol(SHA) and MD5(Message Digest) is appended. SSL is the predecessor of Transport Layer Security(TLS) . After encryption of data, finally, the SSL header is appended to the data.

By default, Confluent Kafka communicates with an unsecured plaintext protocol over 9092 ports.

Confluent Kafka security supports SSL security protocol in intra broker and client communications. Each broker authenticates other brokers and the clients. Brokers and the clients both authenticate each other (2-way authentication). One can also use SSL with SASL security, hit the reference section for Confluent Kafka sasl.

Note: Enabling SSL (TLS) in Confluent Kafka security would override the zero-copy optimization in Kafka consumers. In the Zero-copy mechanism, data is copied into page cache only once and reused on each consumption instead of being stored in memory and copied out to user-space every time it is read. This allows messages to be consumed at a rate that approaches the limit of the network connection.

For a successful handshake:

Each broker should have its own private-key/certificate pair, and the client uses the certificate to authenticate the broker.
Each client should have a private-key/certificate pair if client authentication is enabled*, and the broker uses the certificate to authenticate the client.

*Client authentication is optional but recommended. The tradeoff of having it is performance implications but not having it can allow non-authenticated brokers (impersonation) to communicate to clients.

Steps to enable SSL protocol in Kafka Brokers to enable Confluent Kafka Security:

Note: Below steps do not include steps for creating SSL certificates & keys, refer to this link.

Update the server.properties with below corresponding configurations.

ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.truststore.password=secret123
ssl.keystore.password=secret123
ssl.key.password=secret123

Set the below config in server.properties for enabling inter-broker SSL communication.

security.inter.broker.protocol=SSL

Now update the listeners and advertised listeners (If different) in the server.properties. Generally we use 9093 port for SSL as 9092 is default for plaintext.

listeners=SSL://kafkabroker1:9093
advertised.listeners=SSL://localhost:9093

For continuing plaintext protocol as well along with SSL use below.

listeners=PLAINTEXT://kafkabroker1:9092,SSL://kafkabroker1:9093
advertised.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093

Once the SSL port works fine, we can remove/disable the 9092 plaintext port.

For enabling client authentication (2 way authentication) the below configuration must be enabled.

ssl.client.auth=required

On the client side, we are required to have both certificates and keys. Below configuration is to be mentioned in all clients communicating with SSL enabled Kafka cluster.

These configs can be put in a file and reused with regular client application files.

bootstrap.servers=kafkabroker1:9093
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password=secret123
ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
ssl.keystore.password=secret123
ssl.key.password=secret123

Note: ssl.truststore.password value is optional in both broker and client configuration but is strongly recommended for integrity checking.

Examples of Kafka clients with SSL meta configuration in Confluent Kafka Securit y are below.

${home}/bin/kafka-console-producer –broker-list kafkabroker1:9093 –topic test_topic –producer.config client-ssl.properties
${home}/bin/kafka-console-consumer –bootstrap-server kafkabroker1:9093 –topic test_topic –consumer.config client-ssl.properties –from-beginning
For JVM based producers & consumers below export can be included in start scripts, in command line or set as environment variable for dedicated machines.

export KAFKA_OPTS=
-Djavax.net.ssl.trustStore=/path/to/truststore.jks
-Djavax.net.ssl.trustStoreType=jks
-Djavax.net.ssl.trustStorePassword=&lt;password
-Djavax.net.ssl.keyStore=/path/to/keystore.jks
-Djavax.net.ssl.keyStoreType=jks
-Djavax.net.ssl.keyStorePassword=;password>