hdp core batch processing #8

Supports: trusty
Add to new model

A Hortonworks HDP 2.1 Hadoop Cluster

This bundle is a 2 node Hadoop cluster designed to scale out. It contains the
following units:

One Hadoop Master Node ( YARN Resource Manager + HDFS NameNode)
One Hadoop Slave Cluster Node (YARN NodeManager + HDFS DataNode)


from bundle's home directory:
juju quickstart bundles.yaml

Scale Out Usage

In order to increase the amount of slaves, you must add units, to add one unit:

 juju add-unit compute-node

Or you can add multiple units at once:

 juju add-unit -n4 compute-node

Smoke tests after deployment

1. juju ssh yarn-hdfs-master/0  <<= ssh to hadoop master
2. Smoke test HDFS admin functionality- As the HDFS user, create a 
   /user/$CLIENT_USER in
   hadoop file system - Below steps verifies/demos HDFS functionality  
   a) sudo su $HDFS_USER
   b) hdfs dfs -mkdir -p /user/ubuntu
   c) hdfs dfs -chown ubuntu:ubuntu /user/ubuntu
   d) hdfs dfs -chmod -R 755 /user/ubuntu
   e) exit

3. Smoke test YARN and Mapreduce - Run the smoke test as "ubuntu" user, 
    using Terasort and sort 10GB of data.
   a) hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar 
      teragen 10000 /user/ubuntu/teragenout 
   b) hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar
      terasort /user/ubuntu/teragenout /user/ubuntu/terasortout

4. Smoke test HDFS funtionality from ubuntu user space - delete mapreduce
   output from hdfs:
   hdfs dfs -rm -r /user/ubuntu/teragenout

Contact Information

amir sanjar amir.sanjar@canonical.com

Bundle configuration

Embed this bundle

Add this card to your website by copying the code below. Learn more.