hdp storm #12

Supports: trusty
Add to new model

Description

Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!

Hortonworks Storm Overview

Hortonworks (HDP 2.1.3) Apache Storm is a free and open source distributed
real-time computation system. Storm makes it easy to reliably process unbounded
streams of data, doing for real-time processing what Hadoop did for batch processing
Storm has many use cases: real-time analytics, on-line machine learning, continuous
computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked
it at over a million tuples processed per second per node. It is scalable,
fault-tolerant, guarantees your data will be processed, and is easy to set up
and operate.
This charm will build a storm cluster consistent of:
1. Nimbus master node with following daemons will configured and loaded

storm-drpc
storm-logviewer
storm-nimbus
storm-ui

  1. Storm worker node(s) with following daemons will configured and loaded:

storm-logviewer
storm-supervisor

Deployment

start a 3 node Hortonworks zookeeper quorum: 

juju deploy hdp-zookeeper hdp-zookeeper
juju add-unit -n 2 hdp-zookeeper

NOTE: Zookeeper must be loaded and active, to verify: 

 $echo ruok | nc {hdp-zookeeper/0 IP address} 2181

imok # I'm ok must be the reply 

$ echo stat | nc {hdp-zookeeper/0 IP address} 2181

Node count: 4 # check for node count

start Apache Storm:

juju deploy hdp-storm nimbus-server     
juju deploy hdp-storm storm-worker
juju add-relation nimbus-server:zookeeper hdp-zookeeper:zookeeper
juju add-relation storm-worker:zookeeper hdp-zookeeper:zookeeper
juju add-relation nimbus-server:nimbus storm-worker:slave

To verify a successful deployment:

http://{nimbus-server ip address}:8080

Real-time usage

Example - Deploying and Managing Apache Storm Topologies:
Following steps demonstrates how to deploy a Storm WordCount application .
WordCount application has two parts- Spout randomly generates data
streams and Bolts processes generated stream. 

 - $juju ssh nimbus-server/0
 - $storm jar /usr/lib/storm/contrib/storm-starter/storm-starter-0.9.1.2.1.3.0-563-jar-with-dependencies.jar  storm.starter.WordCountTopology WordCount

How to monitor deployment:

 - go to http://{nimbus-server ip address}:8080   
 - Under "Topology  summary", click on "WordCount"  
 - Monitor Spouts & Bolts tasks

Scale out usage

Example, adding 5 more worker nodes

 juju add-unit -n 5 storm-worker

To verify a successful scale:

 - http://{nimbus-server ip address}:8080
 - Under "Topology summary", click on "WordCount"
 - Click on "Spout" link in "Spouts (All time)" section
 - Note "Host" list under "Executors (All time)" section
 - Go back to "Topology summary"
 - Click on "Rebalance" in "Topology actions" section
 - Click on "Spout" link in "Spouts (All time)" section
 - Refresh, notice re-balancing of job as more storm-worker threads become available

Contact Information

Amir Sanjar amir.sanjar@canonical.com

Apache & Hortonworks Storm