How to submit an Apache Storm 0.10 topology job?

The following is the process of submitting an Apache Storm 0.10 Job.

1. Build your project or projects to a single jar in your local development environment. Refer How to build an Apache Storm 0.10 topology job in Eclipse using Maven?


2. Make sure that you have absolute paths for the configuration files. The preference would be to use Hadoop dfs so that the configuration file can be read by any worker "Supervisor" node. If you can't build with a path is common to all the users i.e under /var/local/storm/ . Do not put this under the user directory like /home/username or /Users/username

3. Make sure the file spout uses absolute path that is common to all users as discussed in step 2.

4. Copy the jar along with the supporting files for config and data files to the Storm master "Nimbus" node.

5. If you do not have Hadoop dfs, then you need to copy the config files and supporting data files to the common location, a common location would be

/var/local/storm/config - for config files
/var/local/storm/topology/topologyname/files and jars

IMPORTANT: you need to copy the supporting files not the jars to all the worker (Supervisor) nodes in the same path. The master (Nimbus) would not copy the supporting files to the worker nodes.

6. Make sure that storm is configured in your path so that you can execute storm command.

7. You can submit a storm job for any other user than the storm cluster is currently running. The only requirement is that the files that you specify should be readable from all the worker nodes in relative paths as well as the storm while submitting the job.

8. Execute the following command from a terminal by first changing directory to the folder that has the jar.


storm jar yourproject-1.0.0-jar-with-dependencies.jar com.yourcompany.yourproduct.yourproject.App "server" "/var/local/storm/config/config.json" "/var/local/storm/topology/yourtopology/topologydatafile.json"


IMPORTANT:
The best practice is to have the first parameter as either local or server to switch execution from a local storm cluster or a production storm cluster.

Do not put a quote or a dash (-) in the command for the jar or the main class name. This is not the same as the java -jar "jarpath" command.

You have to put quotes for all the parameters (arguments) that go into the main class.


9. Go to the Storm UI and check if your job has been submitted and track its status.
http://yourmaster(nimbus)node:8080/

How to install CentOS 7 on Virtual Machine using VMWare vSphere 6 client?

Now that we have created a Virtual Maching containing the CentOS installation media in the previous class. We can go ahead an now install teh CentOS 7 in the virtual machine (VM).

1. Go to the VMWare vSphere 6 client and select the VM , do a right click and start the VM.


2. Now we need to launch the VM so that we can see it in console. On the vSphere client you would see a terminal with an arrow icon. Click on it to launch the VM console.

3. Inside the console you can use "Alt" + Enter to toggle the mouse between VM and your desktop.
After clicking on the VM Console press the 'I' key to start the installation or use the arrow keys and press enter.


4. After a few minutes you would be presented the CentOS 7 installation start page.


5. We can set these properties in any order. The important thing to note is that the Software selection is a minimal install so you won't get a UI, you need to change it to Server with UI to be able to use the UI.



6. Setup Network by just sliding the bar to the on position. Make sure that you give a name to the machine in host name instead of the default. Make a note of the IP.


7. Setup the Date and Time. If you have setup Network first you can enable the network time else we could also do it later.


8. Setup Disk by just clicking on the disk icon


9. After all this is setup, click on the "Begin Installation" on step 4.
This would show the progress as well as ask you to create the root user password and other users.


10. Do create another user who is an admin/root and do not use the root user for any of the interaction.


11. Create another Admin user, make sure you check the "Make this user administrator"


12. After installation is done it would ask you to reboot and you are can now either login through the console or use any SSH compliant client like putty or console in other Unix based OS and login to the VM using the IP on setup 6.

OS X ability to open multiple Eclipse Mars workspace instances on Mac

By default only one instance of Eclipse can be opened in Mac.

In order to open multiple instance do the following.

1. On Eclipse Mars click on Help -> Eclipse Marketplace
2. On the file text box enter "OS X eclipse launcher" and click go
3. Install the plugin
more info from  
http://marketplace.eclipse.org/content/os-x-eclipse-launcher?mpc=true&mpc_state=
As of today the version is OS X eclipse launcher 3.0

After installing restart eclipse.
1. Click on File Menu
2. You should now have an "Open Workspace" menu link which would allow you to launch multiple Eclipse instances

Linux Class Overview

Unlike the Microsoft world where there is only one OS untill a new version comes and replaces it; in the open source world there are many OSes. Though this topic can span multiple pages, I would like to keep it short.. The mother of all OS is Unix, which according to many is only for geeks or scientist so a simpler flavor with UI was created and named as Linux. As with all open source there is no one owner of this OS, each vendor or community would create its own version of Linux, these are callled flavors. Few of the flavors are Ubuntu, CentOS, Suse, Redhat, Debian, OS X and so on. These in-turn have its own versions.

Few of the Linux commands are common across all these OS while some are not. In the Big Data School anything that is common is put in the OS-Linux Class while the specific OS lesson are put under OS Version Class. 

Using putty to connect to a remote linux box CentOS7 or Ubuntu 14.04.3 LTS

As explained in the previous post, the center of the open source server interaction client would revolve around the console.

The popular console that existing for Windows users is putty. Its currently available only on 32 bit mode but this is good enough.

1. Download putty.
2. Double click to open it.
3. On the putty configuration enter the IP address for the host name, leave the default SSH port 22.
You should also give a name for the Saved Sessions and click on the "Save" button so that you don't have to type this information again and again. Next it you can select from the saved list and then click on Load.

4.Click on the open to enter the console

5. Enter the username and password to connect to the server

Where is my remote desktop in open source world?

Well, in the Microsoft world we are spoiled, the server is not the place to have a UI, say goodbye to your UI and mouse skills, learn to live with the command line world and master this skill as fast as you can. No more UI or mouse when interacting with servers, everything should be done using the console. The most popular console in Windows world is putty and the most popular text editor is the vi editor. These are simple by powerful so learn these as fast as you can.

How to setup a CentOS 7 Virtual Machine using VMWare vSphere 6 client?

1. Download the vSphere client as discussed in the Big Data Lab Setup.
2. Login to the vSphere Hypervisor using the client using the root username and password which you used to install the vSphere Hypervisor.
   

3.Right click on the server and select create new virtual machine

4. Select the Typical configuration on the Create New Virtual Machine Popup

5. Give a name for the virtual machine

6. Select storage and size.

7. Select Guest Operating System. From the Guest Operating System select Linux and from the versions select the Cent OS 4/5/6/7 (64 bit)

8. Select Network
This would be the default. We would look at how to connect this network card to the external network when we setup the network settings for the ESXi Server.

9. Create a Virtual Disk. Select a disk size depending on the storage amount. We can always come back and add more  disk space if we run out of it

10. Select Ready to complete. Make sure that you check box the "edit the virtual machine before completion" so that we can change some default settings.

11. Change the Memory to 4 GB

12. Change the CPU Cores

13. Attached a CentOS7 iso for installation to the CD ROM drive, enable the "connect on power on" and select the CentOS7 iso on the radio button "Datastore ISO File".

14. Start the VM. Go back to the vSphere client and start the VM

15. Launch the Virtual Machine Console to install the CentOS which would be done in the next blog. Select the VM that we created and then click on the computer icon with the arrow.