How to setup standalone instance of Apache Kafka 0.9.0.1 on localhost for Mac OS X?

Apache Kafka is distributed Message Broker which would also for reading messages in a sequential manner maintaining the order in which a message has arrived.

If also allows multiple read of message by way of partitions. So if we have a 4 partition topic it would allow 4 threads to read the messages in parallel. However its the job of the developer to make sure that the messages for the same entity goes in a sequential manner to the same thread instead of a different thread. Kafka does this by having a message key. So if we send an entity E1 with key Key1  with time T1, if another message of E1 comes at Time T2 its the developer responsibility to give the same Key 1 so that the messages are read in the order by a thread namely E1 T1 first and then E1 T2 next and so on.



1. You have admin privileges for your development box

2. Make sure Java 7 or 8 is installed and configured as default as discussed in How to install Java 7 and Java 8 in Mac OS X

3. Make sure Apache Zookeeper standalone is installed as specified in How to setup standalone instance of Apache Zookeeper 3.4.6 on localhost for Mac OS X?
//create a folder for kafka under the /usr/local directory
cd /usr/local
sudo mkdir kafka

//create the data cum log directory for kafka under the var/lib
cd /var/lib
sudo mkdir kafka

//download kafka
wget http://apache.claz.com/kafka/0.9.0.1/kafka_2.11-0.9.0.1.tgz

//unpack the file
tar xzf kafka_2.11-0.9.0.1.tgz


//move the kafka installation to the usr/local/kafka from the download directory
mv kafka_2.11-0.9.0.1 /usr/local/kafka/

//switch to the kafka directory
cd /usr/local/kafka/kafka_2.11-0.9.0.1/


//switch to the config directory
cd config

edit the config file and change the following

vi server.properties

#The broker Id should be unique 
broker.id=1

#change data cum log directory to
log.dirs=/var/lib/kafka

#include the zookeeper servers
zookeeper.connect=YOURMACHOSTNAME.local:2181

#Since this is a dev machine allow a topic to be deleted
delete.topic.enable=true

//save the file
:wq

//move to the kafka root
cd /usr/local/kafka/kafka_2.11-0.9.0.1

//start kafka broker
bin/kafka-server-start.sh config/server.properties >/dev/null &

//if you need to stop
kill processid

//check if the process is running
ps -a | grep kafka

//or use jps
jps

How to setup standalone instance of Apache Zookeeper 3.4.6 on localhost for Mac OS X?

Apache Zookeeper is a distributed state manager that other systems use for state management. You could also setup a standalone zookeeper instead of a built in one to share this zookeeper instance across multiple technologies like Kafka, Storm, Hbase etc so that each instance does not start its own instance. These instruction let you setup Zookeeper as a standalone instance

1. You have admin privileges for your development box
2. Make sure Java 7 or 8 is installed and configured as default as discussed in How to install Java 7 and Java 8 in Mac OS X

//create a folder for zookeeper under the /usr/local directory
cd /usr/local
sudo mkdir zookeeper

//create the data directory for Zookeeper under the var/lib
cd /var/lib
sudo mkdir zookeeper

//create a file named myid under the data directory
cd /var/lib/zookeeper
vi myid

//Put only the number 1.
1

//save the file
:wq

if you do a cat myid it should just display 1

//download zookeeper on any local directory
wget http://apache.arvixe.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

//unpack the file
tar xzf zookeeper-3.4.6.tar.gz


//move the zookeeper installation to the usr/local/zookeeper 

//from the download directory
mv zookeeper-3.4.6 /usr/local/zookeeper/

//switch to the /usr/local/zookeeper directory
cd /usr/local/zookeeper/zookeeper-3.4.6

//move to the conf folder for the version of zookeeper like
cd conf

//copy the sample config to zoo.cfg
cp zoo_sample.cfg zoo.cfg

//switch to the conf directory
cd /usr/local/zookeeper/zookeeper-3.4.6/conf

edit the zoo.cfg file and change the data directory to

vi zoo.cfg

//change data directory to
dataDir=/var/lib/zookeeper

#include the cluster servers

server.1=YOURMACHOSTNAME.local:2888:3888


//move to the root of zookeeper

cd /usr/local/zookeeper/zookeeper-3.4.6

//start zookeeper
bin/zkServer.sh start

//if you need to stop
bin/zkServer.sh stop

//check if the process is running
jps

//check for QuorumPeerMain

//check the status of zookeeper

bin/zkServer.sh status

//This should display
Mode: standalone