hdp hadoop hive mysql #4
Hortonworks HDP 2.1 HIVE, mysql, and Hadoop Cluster
This bundle is a 4 node Hadoop cluster designed to scale out. It contains the following units:
- one Hadoop Master (yarn & hdfs) Node
- one Hadoop compute Node
- one Hive Node
- one MySQL Node
Usage
Deploy the bundle, once you have a cluster running, ssh to the Hadoop Master node:
juju ssh yarn-hdfs-master/0
Smoke test HDFS admin functionality
As the HDFS user, create a /user/$CLIENT_USER on the hadoop file system. The steps below verifies/demonstrates HDFS functionality:
sudo su $HDFS_USER
hdfs dfs -mkdir -p /user/ubuntu
hdfs dfs -chown ubuntu:ubuntu /user/ubuntu
hdfs dfs -chmod -R 755 /user/ubuntu
exit
Smoke test YARN and Mapreduce
Run the test as the $CLIENT_USER, using Terasort and sort 10GB of data:
Hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar teragen 10000 /user/ubuntu/teragenout
Hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar terasort /user/ubuntu/teragenout /user/ubuntu/terasortout
Smoke test HDFS functionality from ubuntu user space
Delete the mapreduce output from hdfs:
hdfs dfs -rm -r /user/ubuntu/teragenout
HIVE + HDFS Usage:
Create an ssh session with the Hive server, switch to the Hive user, and start the Hive console:
juju ssh hdphive/0
sudo su \$HIVE_USER
hive
From the Hive console, create a table:
show databases;
create table test(col1 int, col2 string);
show tables;
exit;
Exit from the Hive user session:
exit
Change to the HDFS user, verify connection to the HDFS cluster, and that a test directory has been created on the remote HDFS cluster:
sudo su \$HDFS_USER
hadoop dfsadmin -report
hdfs dfs -ls /apps/hive/warehouse
Scale Out Usage
This bundle was designed to scale out. In order to increase the amount of slaves, you must add units, to add one unit:
juju add-unit compute-node
Or you can add multiple units at once:
juju add-unit -n4 compute-node
Contact Information
Amir Sanjar amir.sanjar@canonical.com
Upstream Project Name
- Upstream website http://hortonworks.com/
- Upstream bug tracker https://issues.apache.org/jira/browse/HIVE