Content

Wednesday, January 27, 2016

How to build an Apache Storm 0.10 topology job in Eclipse using Maven?

The following is the process of building an Apache Storm 0.10 Topology Job.

Technologies used:
a. Apache Storm 0.10
b. Eclipse Mars
c. Apache Maven 3.3.9
d. Java 7 or Java 8

1.You need to have two environments to build Storm Topology.
Environment 1: Developer Box
Environment 2: Development / Build Box

IMPORTANT: The changes you are going to make if done in Developer Box would render the running the topologies on the local Storm cluster broken.

2.Remove the jars that Storm already provides.
Remove Storm jar and Logging Jar like Log4j2
Go to the project pom.xml that has the Storm jar reference and add the following so that this jar is not including when we compile to a single jar with dependencies.


    <dependency>
      <groupid>org.apache.storm</groupid>
      <artifactid>storm-core>/artifactid>
      <version>0.10.0</version>
      <!--  This need to be enabled for storm submit job --> 
      <scope>provided</scope>   
    </dependency>

    <dependency>
      <groupid>org.apache.logging.log4j</groupid>
      <artifactid>log4j-api>/artifactid>
      <version>2.5</version>
      <!--  This need to be enabled for storm submit job --> 
      <scope>provided</scope>   
    </dependency>

    <dependency>
      <groupid>org.apache.logging.log4j</groupid>
      <artifactid>log4j-core>/artifactid>
      <version>2.5</version>
      <!--  This need to be enabled for storm submit job --> 
      <scope>provided</scope>   
    </dependency>   

3. Packing the project into a single jar with dependencies.
Go to the maven project that has the main method. Edit the pom.xml to have the following.
below the <dependencies> node.
  </dependencies>
  <build>
  <plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.yourcompany.yourproduct.yourproject.App</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
  </plugins>
  </build> 

4. Issue the following Maven command and copy the jar with dependencies in the target folder.

mvn package
Note: you would have both the jars, copy the one with the following name appended like yourproject-1.0.0-jar-with-dependencies.jar

5. Follow the instruction for the next step of submitting the Storm Topology Job

No comments:

Post a Comment