Setup Kafka cluster with Kraft

We provided a set of instructions for setting up Apache Kafka version 3.6.1 along with necessary configurations and system tuning on CentOS/RHEL-based systems. These instructions guide you through setting up Kafka, configuring log rolling policies, configuring Kafka service, setting up cluster-ID, managing permissions, enabling and starting Kafka service, and performing OS-level tuning.

Apache Kafka’s KRaft mode, which is a replicated mode of Kafka controllers that relies on the Raft consensus algorithm for leader election and replication.

Here’s a breakdown of the steps:

Requests:

Install Utilities and Dependencies:

yum install curl 
wget java-1.8.0-openjdk-devel -y

Setting up a kafka cluster without zookeeper

Download Kafka Binaries

Extract kafka

wget https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz

tar -xzf kafka_2.13-3.6.1.tgz

Create Kafka User and Group

useradd -r -s /sbin/nologin kafka
groupadd kafka

Folder Creation

mkdir -p /etc/kafka /data/log/kafka /data/kafka

Move the Binaries to the Respective Directory

cp -rp ~/kafka_2.13–3.6.1/* /etc/kafka

Download the JMX Exporter Jar and Jolokia Jar

wget https://repo.maven.apache.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.20.0/jmx_prometheus_javaagent-0.20.0.jar #JMX Exporter Jar
wget https://repo1.maven.org/maven2/org/jolokia/jolokia-jvm/1.5.0/jolokia-jvm-1.5.0-agent.jar #jolokia agent jar

Move the downloaded jar to a specific folder

cp ~/jmx_prometheus_javaagent-0.20.0.jar /etc/kafka/libs/
cp ~/jolokia-jvm-1.5.0-agent.jar /etc/kafka/libs/

Create Profile for JMX exporter

echo -e "

rules:
- pattern: ".*"
" > /etc/kafka/config/jmx_exporter_kraft.yml

update log4j property file to roll daily basis and delete after 3 days

Execute the following command to configure log4j to delete logs older than 3 days and roll the logs daily

Create a service file for the Kafka

vi /etc/systemd/system/kafka.service

Kafka cluster configuration

If you go to config/kraft folder inside the kafka home directory, you will see a file called server.properties. This is a sample file which is provided by kafka, to show how kafka can be started without zookeeper

Create 3 new files from server.properties. This is because we will be creating a 3 node cluster

cd config/kraft
cp server.properties server1.properties
cp server.properties server2.properties
cp server.properties server3.properties

In server1.properties, modify the following properties. Please keep the other properties as is.

node.id=1
process.roles=broker,controller
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://:9092,CONTROLLER://:19092
log.dirs=/tmp/server1/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094

Let me explain what these properties do:

node.id: This will act as the node Id in the cluster. This will help us identify which broker this is. It will also help us identify which kraft controller node this is.

process.roles: A node can act as a broker or controller or both. Here we are indicating that this node can be both a kafka broker and a kraft controller node.

inter.broker.listener.name: Here the broker listener name is set to PLAINTEXT

controller.listener.names: Here the controller listener name is set to CONTROLLER

listeners: Here we indicate that the broker will use port 9092 and the kraft controller will use port 19092

log.dirs: This is the log directory where kafka will store the data

listener.security.protocol.map: Here the connection security details are added

controller.quorum.voters: This is used to indicate all the kraft controllers which are available. Here we are indicating that we will have 3 kraft controller nodes running on ports 19092, 19093 and 19094

For server2.properties modify the following properties. Please keep the other properties as is.

node.id=2
process.roles=broker,controller
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094
listeners=PLAINTEXT://:9093,CONTROLLER://:19093
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
log.dirs=/tmp/server2/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

In Server 2, the broker port is 9093 and controller port is 19093. Also the log.dirs is different

For server3.properties modify the following properties. Please keep the other properties as is.

node.id=3
process.roles=broker,controller
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094
listeners=PLAINTEXT://:9094,CONTROLLER://:19094
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
log.dirs=/tmp/server3/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

In Server 3 the broker port is 9094 and the controller port is 19094. Also the log.dirs is different

Configuration of broker/Controller node

• if we are configuring controller node, then we need to select /etc/kafka/config/kraft/controller.properties
• if we are configuring broker node, then we need to select /etc/kafka/config/kraft/broker.properties
• if we are configuring both broker and controller node, then we need to select /etc/kafka/config/kraft/server.properties
• Whatever we are configuring, we need to give the same path in the above kafka.service file in the ExecStart parameter

In the following config

• process.roles need to be mentioned for Kraft otherwise it will look for zookeeper mode.
• process.roles value can be a broker for broker node, controller for the controller node and broker, controller and for both.
• node.id is unique id for each node and for kraft mode, it is required.
• controller.quorum.voters contain all controller nodes to from Quorom
• Mode of communication will be PLAINTEXT for broker and CONTROLLER for controller node, mentioned in controller.listener.names.
• The rest of the config is the same as the old Kafka with the zookeeper setup.

Setup Cluster ID

we need to form a cluster id for the cluster for the new Kafka versions like the following and the cluster ID should be the same among all nodes in the cluster

/etc/kafka/bin/kafka-storage.sh format -c /etc/kafka/config/kraft/broker.properties --cluster-id MwFA64stS0-rQS5k9vNhbQ

chown -R kafka:kafka /data/kafka #data directory path to give permission of files generated by above command

We can generate random id from

/etc/kafka/bin/kafka-storage.sh random-uuid

Set Folder Permission

chown -R kafka:kafka /data /etc/kafka #data directory path to give permission of files generated by above command

Service Enable and Start

systemctl daemon-reload
systemctl enable kafka
service kafka restart
service kafka status

OS Tuning

In file/etc/sysctl.conf

For /etc/security/limits.conf

kafka soft nofile unlimited
kafka hard nofile unlimited
kafka soft nproc unlimited
kafka hard nproc unlimited
root soft nofile 95000
root hard nofile 95000
root soft nproc 95000
root hard nproc 95000

Disable Transparent Huge Pages

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

These steps will guide you through setting up an Apache Kafka cluster with Kraft mode on CentOS/RHEL-based systems. Make sure to follow each step carefully to ensure a successful setup.

Congratulations 😊

You have now learnt how to setup kafka without zookeeper

Understanding KRaft and its Benefits

Migrating from Zookeeper to Kraft Mode in Apache Kafka: Understanding KRaft and its Benefits

References

https://kafka.apache.org/

https://cwiki.apache.org/confluence/display/KAFKA/KIP-500