Kafka Single Node Multiple Brokers


Kafka Single Node Multiple Brokers
In this configuration you have a single ZooKeeper and multiple broker id instances. Following are the steps to configure it.
 
Steps to follow
Step 1 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka/config
 
$ cd /usr/local/kafka/config
Step 2 - Edit server-one.properties file. Automatically it will create a new server-one.properties file if it doesn't exists and opens for editing.
 
$ gedit server-one.properties
Step 3 - Add these below lines to server-one.properties file. Save and close.
 
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
# The port the socket server listens on
port=9093
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-1
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=1000000
delete.topic.enable = true
Step 4 - Edit server-two.properties file. Automatically it will create a new server-two.properties file if it doesn't exists and opens for editing.
 
$ gedit server-two.properties
Step 5 - Add these below lines to server-two.properties file. Save and close.
 
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2
# The port the socket server listens on
port=9094
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-2
zookeeper.connect=localhost:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=1000000
delete.topic.enable = true
Step 6 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka
 
$ cd /usr/local/kafka
Step 7 - Start the zookeeper
 
$ ./bin/zookeeper-server-start.sh config/zookeeper.properties
Step 8 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka
 
$ cd /usr/local/kafka
Step 9 - Start the kafka server
 
$ ./bin/kafka-server-start.sh config/server.properties
Step 10 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka
 
$ cd /usr/local/kafka
Step 11 - Start the kafka server-one
 
$ ./bin/kafka-server-start.sh config/server-one.properties
Step 12 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka
 
$ cd /usr/local/kafka
Step 13 - Start the kafka server-two
 
$ ./bin/kafka-server-start.sh config/server-two.properties
Step 14 - Open a new terminal (CTRL + ALT + T) and just type 'jps' (without quotes). It shows all the java processes that are running.
 
Here Kafka is the kafka server and QuorumPeerMain is zookeeper process.
Step 15 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka
 
$ cd /usr/local/kafka
Step 16 - Create a topic.
 
$ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic Multibrokerapplication
 
Step 17 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka
 
$ cd /usr/local/kafka
Step 18 - Producer
 
$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello-Kafka
Step 19 - Open a new terminal (CTRL + ALT + T) and change the directory to /usr/local/kafka
 
$ cd /usr/local/kafka
Step 20 - Consumer
 
$ ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic Hello-Kafka --from-beginning
Step 21 - Open a new terminal (CTRL + ALT + T) and just type 'jps' (without quotes). It shows all the java processes that are running.
 
Step 22 - Stop zookeeper, kafka server, producer and consumer. Just press 'CTRL + D' in the terminals of zookeeper, kafka server, producer and consumer.
 


Have any Question or Comment?

Leave a Reply

Your email address will not be published. Required fields are marked *