Setup Kafka cluster with Kraft
We provided a set of instructions for setting up Apache Kafka version 3.6.1 along with necessary configurations and system tuning on CentOS/RHEL-based systems. These instructions guide you through setting up Kafka, configuring log rolling policies, configuring Kafka service, setting up cluster-ID, managing permissions, enabling and starting Kafka service, and performing OS-level tuning.
Apache Kafkaโs KRaft mode, which is a replicated mode of Kafka controllers that relies on the Raft consensus algorithm for leader election and replication.
Hereโs a breakdown of the steps:
Requests:
Install Utilities and Dependencies:
yum install curl
wget java-1.8.0-openjdk-devel -y
Setting up a kafka cluster without zookeeper
Download Kafka Binaries
Extract kafka
wget https://downloads.apache.org/kafka/3.6.1/kafka_2.13-3.6.1.tgz
tar -xzf kafka_2.13-3.6.1.tgz
Create Kafka User and Group
useradd -r -s /sbin/nologin kafka
groupadd kafka
Folder Creation
mkdir -p /etc/kafka /data/log/kafka /data/kafka
Move the Binaries to the Respective Directory
cp -rp ~/kafka_2.13โ3.6.1/* /etc/kafka
Download the JMX Exporter Jar and Jolokia Jar
wget https://repo.maven.apache.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.20.0/jmx_prometheus_javaagent-0.20.0.jar #JMX Exporter Jar
wget https://repo1.maven.org/maven2/org/jolokia/jolokia-jvm/1.5.0/jolokia-jvm-1.5.0-agent.jar #jolokia agent jar
Move the downloaded jar to a specific folder
cp ~/jmx_prometheus_javaagent-0.20.0.jar /etc/kafka/libs/
cp ~/jolokia-jvm-1.5.0-agent.jar /etc/kafka/libs/
Create Profile for JMX exporter
echo -e "
rules:
- pattern: ".*"
" > /etc/kafka/config/jmx_exporter_kraft.yml
update log4j property file to roll daily basis and delete after 3 days
Execute the following command to configure log4j to delete logs older than 3 days and roll the logs daily
Create a service file for the Kafka
vi /etc/systemd/system/kafka.service
Kafka cluster configuration
If you go to config/kraft
folder inside the kafka home directory, you will see a file called server.properties
. This is a sample file which is provided by kafka, to show how kafka can be started without zookeeper
Create 3 new files from server.properties. This is because we will be creating a 3 node cluster
cd config/kraft
cp server.properties server1.properties
cp server.properties server2.properties
cp server.properties server3.properties
In server1.properties, modify the following properties. Please keep the other properties as is.
node.id=1
process.roles=broker,controller
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://:9092,CONTROLLER://:19092
log.dirs=/tmp/server1/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094
Let me explain what these properties do:
node.id: This will act as the node Id in the cluster. This will help us identify which broker this is. It will also help us identify which kraft controller node this is.
process.roles: A node can act as a broker or controller or both. Here we are indicating that this node can be both a kafka broker and a kraft controller node.
inter.broker.listener.name: Here the broker listener name is set to PLAINTEXT
controller.listener.names: Here the controller listener name is set to CONTROLLER
listeners: Here we indicate that the broker will use port 9092 and the kraft controller will use port 19092
log.dirs: This is the log directory where kafka will store the data
listener.security.protocol.map: Here the connection security details are added
controller.quorum.voters: This is used to indicate all the kraft controllers which are available. Here we are indicating that we will have 3 kraft controller nodes running on ports 19092, 19093 and 19094
For server2.properties modify the following properties. Please keep the other properties as is.
node.id=2
process.roles=broker,controller
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094
listeners=PLAINTEXT://:9093,CONTROLLER://:19093
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
log.dirs=/tmp/server2/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
In Server 2, the broker port is 9093 and controller port is 19093. Also the log.dirs is different
For server3.properties modify the following properties. Please keep the other properties as is.
node.id=3
process.roles=broker,controller
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094
listeners=PLAINTEXT://:9094,CONTROLLER://:19094
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
log.dirs=/tmp/server3/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
In Server 3 the broker port is 9094 and the controller port is 19094. Also the log.dirs is different
Configuration of broker/Controller node
- if we are configuring controller node, then we need to select /etc/kafka/config/kraft/controller.properties
- if we are configuring broker node, then we need to select /etc/kafka/config/kraft/broker.properties
- if we are configuring both broker and controller node, then we need to select /etc/kafka/config/kraft/server.properties
- Whatever we are configuring, we need to give the same path in the above kafka.service file in the ExecStart parameter
In the following config
- process.roles need to be mentioned for Kraft otherwise it will look for zookeeper mode.
- process.roles value can be a broker for broker node, controller for the controller node and broker, controller and for both.
- node.id is unique id for each node and for kraft mode, it is required.
- controller.quorum.voters contain all controller nodes to from Quorom
- Mode of communication will be PLAINTEXT for broker and CONTROLLER for controller node, mentioned in controller.listener.names.
- The rest of the config is the same as the old Kafka with the zookeeper setup.
Setup Cluster ID
we need to form a cluster id for the cluster for the new Kafka versions like the following and the cluster ID should be the same among all nodes in the cluster
/etc/kafka/bin/kafka-storage.sh format -c /etc/kafka/config/kraft/broker.properties --cluster-id MwFA64stS0-rQS5k9vNhbQ
chown -R kafka:kafka /data/kafka #data directory path to give permission of files generated by above command
We can generate random id from
/etc/kafka/bin/kafka-storage.sh random-uuid
Set Folder Permission
chown -R kafka:kafka /data /etc/kafka #data directory path to give permission of files generated by above command
Service Enable and Start
systemctl daemon-reload
systemctl enable kafka
service kafka restart
service kafka status
OS Tuning
In file/etc/sysctl.conf
For /etc/security/limits.conf
kafka soft nofile unlimited
kafka hard nofile unlimited
kafka soft nproc unlimited
kafka hard nproc unlimited
root soft nofile 95000
root hard nofile 95000
root soft nproc 95000
root hard nproc 95000
Disable Transparent Huge Pages
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
These steps will guide you through setting up an Apache Kafka cluster with Kraft mode on CentOS/RHEL-based systems. Make sure to follow each step carefully to ensure a successful setup.
Congratulations ๐
You have now learnt how to setup kafka without zookeeper