How to setup standalone instance of Apache Kafka 0.9.0.1 on localhost for Mac OS X?

Apache Kafka is distributed Message Broker which would also for reading messages in a sequential manner maintaining the order in which a message has arrived.

If also allows multiple read of message by way of partitions. So if we have a 4 partition topic it would allow 4 threads to read the messages in parallel. However its the job of the developer to make sure that the messages for the same entity goes in a sequential manner to the same thread instead of a different thread. Kafka does this by having a message key. So if we send an entity E1 with key Key1  with time T1, if another message of E1 comes at Time T2 its the developer responsibility to give the same Key 1 so that the messages are read in the order by a thread namely E1 T1 first and then E1 T2 next and so on.



1. You have admin privileges for your development box

2. Make sure Java 7 or 8 is installed and configured as default as discussed in How to install Java 7 and Java 8 in Mac OS X

3. Make sure Apache Zookeeper standalone is installed as specified in How to setup standalone instance of Apache Zookeeper 3.4.6 on localhost for Mac OS X?
//create a folder for kafka under the /usr/local directory
cd /usr/local
sudo mkdir kafka

//create the data cum log directory for kafka under the var/lib
cd /var/lib
sudo mkdir kafka

//download kafka
wget http://apache.claz.com/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgz

//unpack the file
tar xzf kafka_2.11-0.9.0.1.tgz


//move the kafka installation to the usr/local/kafka from the download directory
mv kafka_2.11-0.9.0.1 /usr/local/kafka/

//switch to the kafka directory
cd /usr/local/kafka/kafka_2.11-0.9.0.1/


//switch to the config directory
cd config

edit the config file and change the following

vi server.properties

#The broker Id should be unique 
broker.id=1

#change data cum log directory to
log.dirs=/var/lib/kafka

#include the zookeeper servers
zookeeper.connect=YOURMACHOSTNAME.local:2181

#Since this is a dev machine allow a topic to be deleted
delete.topic.enable=true

//save the file
:wq

//move to the kafka root
cd /usr/local/kafka/kafka_2.11-0.9.0.1

//start kafka broker
bin/kafka-server-start.sh config/server.properties >/dev/null &

//if you need to stop
kill processid

//check if the process is running
ps -a | grep kafka

//or use jps
jps