How to setup a 3 Node Apache Zookeeper 3.4.6 cluster in CentOS 7?

Zookeeper is short is a distributed state manager which can be used by many clusters to maintain state across its own clusters. Like HBase can use Zookeeper to maintain state across its own set of clusters without having to have cluster state within it.

The following needs to be done before beginning  the Zookeeper cluster Setup.

1. Create 3 CentOS 7 Servers ZKNODE1, ZKNODE2, and ZKNOD3 as discussed in How to install CentOS 7 on Virtual Machine using VMWare vSphere 6 client?

2. Make sure Java 7 is installed and configured as default as discussed in How to install Java 7 and Java 8 in CentOS 7?

3. Create the bigdatauser, bigdataadmin and the bigdatagroup as discussed in How to create a user, group and enable him to do what a super user can in CentOS7?

4. Make sure the firewall is disabled and stopped as discussed in How to turn off firewall on CentOS 7? 

5. Change etc/hosts file so that all the IPs and the names of the servers are resolved as discussed in
How to setup DNS entries for big data servers in the cloud or not on a domain in /etc/hosts file?

6. Using the bigdatauser setup password less ssh across the 3 clusters namely ZKNODE1, ZKNODE2 and ZKNODE3 as discussed in How to setup password less ssh between CentOS 7 cluster servers?


For each of the Server ZKNODE1, ZKNODE2 and ZKNODE3 do the following

Login using the bigdataadmin

//create a folder for zookeeper under the /usr/local directory
cd /usr/local
sudo mkdir zookeeper

//change ownership to bigdatauser
sudo chown -R bigdatauser:bigdatagroup zookeeper

//create the data directory for Zookeeper under the var/lib
cd /var/lib
sudo mkdir zookeeper

//change ownership to bigdatauser
sudo chown -R bigdatauser:bigdatagroup  zookeeper


Switch to bigdataauser

//create a file named myid under the data directory
cd /var/lib/zookeeper
vi myid

put only the number for the corresponding servers. DO NOT put all the 3 numbers in each server.
on ZKNODE1
1
on ZKNODE2
2
on ZKNODE3
3

if you do a cat myid it should just display 1 for ZKNODE1 and so on.


//download zookeeper
wget http://apache.arvixe.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

//unpack the file
tar xzf zookeeper-3.4.6.tar.gz


//move the zookeeper installation to the usr/local/zookeeper from the download directory
mv zookeeper-3.4.6 /usr/local/zookeeper/

//switch to the /usr/local/zookeeper directory
cd /usr/local/zookeeper/zookeeper-3.4.6

//move to the conf folder for the version of zookeeper like
cd conf

//copy the sample config to zoo.cfg
cp zoo_sample.cfg zoo.cfg

//switch to the conf directory
cd /usr/local/zookeeper/zookeeper-3.4.6/conf

edit the zoo.cfg file and change the data directory to

vi zoo.cfg

//change data directory to
dataDir=/var/lib/zookeeper

#include the cluster servers
server.1=ZKNODE1:2888:3888
server.2=ZKNODE2:2888:3888
server.3=ZKNODE3:2888:3888


//move to the root of the cluster server on any of the cluster. Start the zookeeper in all of the 3 servers.
cd /usr/local/zookeeper/zookeeper-3.4.6

//start zookeeper
bin/zkServer.sh start

//if you need to stop
bin/zkServer.sh stop

//check if the process is running
ps -aux | grep java

//check for QuorumPeerMain

//check the status of each server to see if they are in a cluster. Only one among the 3 should be master and the others are followers
bin/zkServer.sh status
to find if its running as follower or leader similar to master, slave.
Mode: follower
Mode: leader


Troubleshooting Errors
Error:
Using config: /usr/local/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
mkdir: cannot create directory ‚/var/bin‚: Permission denied
Starting zookeeper ... bin/zkServer.sh: line 113: /var/bin/zookeeper/zookeeper_server.pid: No such file or directory
Solution:
Make sure that the data directory is correct and you are running as bigdatauser and not bigdataadmin