Content

Saturday, May 21, 2016

How to setup standalone instance of Apache Kafka 0.9.0.1 on localhost for Mac OS X?


Apache Kafka is distributed Message Broker which would also for reading messages in a sequential manner maintaining the order in which a message has arrived.
If also allows multiple read of message by way of partitions. So if we have a 4 partition topic it would allow 4 threads to read the messages in parallel. However its the job of the developer to make sure that the messages for the same entity goes in a sequential manner to the same thread instead of a different thread. Kafka does this by having a message key. So if we send an entity E1 with key Key1  with time T1, if another message of E1 comes at Time T2 its the developer responsibility to give the same Key 1 so that the messages are read in the order by a thread namely E1 T1 first and then E1 T2 next and so on.
1. You have admin privileges for your development box
2. Make sure Java 7 or 8 is installed and configured as default as discussed in How to install Java 7 and Java 8 in Mac OS X
3. Make sure Apache Zookeeper standalone is installed as specified in How to setup standalone instance of Apache Zookeeper 3.4.6 on localhost for Mac OS X?
//create a folder for kafka under the /usr/local directory cd /usr/local sudo mkdir kafka //create the data cum log directory for kafka under the var/lib cd /var/lib sudo mkdir kafka //download kafka wget http://apache.claz.com/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgz //unpack the file tar xzf kafka_2.11-0.9.0.1.tgz //move the kafka installation to the usr/local/kafka from the download directory mv kafka_2.11-0.9.0.1 /usr/local/kafka/ //switch to the kafka directory cd /usr/local/kafka/kafka_2.11-0.9.0.1/ //switch to the config directory cd config edit the config file and change the following vi server.properties #The broker Id should be unique broker.id=1 #change data cum log directory to log.dirs=/var/lib/kafka #include the zookeeper servers zookeeper.connect=YOURMACHOSTNAME.local:2181 #Since this is a dev machine allow a topic to be deleted delete.topic.enable=true //save the file :wq
//move to the kafka root
cd /usr/local/kafka/kafka_2.11-0.9.0.1

//start kafka broker
bin/kafka-server-start.sh config/server.properties >/dev/null &

//if you need to stop
kill processid

//check if the process is running
ps -a | grep kafka

//or use jps
jps


No comments:

Post a Comment