apache spark #0

Supports: trusty
Add to new model

Description

Apache Spark is a fast and general engine for large-scale data processing.

Learn more at http://spark.apache.org.

Overview

Spark 1.3.x cluster for YARN & HDFS

Apache Spark™ is a fast and general purpose engine for large-scale data processing. Key features:

Speed: Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. Spark has an advanced DAG execution engine that supports cyclic data flow and in-memory computing.

Ease of Use: Write applications quickly in Java, Scala or Python. Spark offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala and Python shells.

General Purpose Engine: Combine SQL, streaming, and complex analytics. Spark powers a stack of high-level tools including Shark for SQL, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these frameworks seamlessly in the same application.

Usage

This charm leverages our pluggable Hadoop model with the hadoop-plugin interface. This means that you will need to deploy a base Apache Hadoop cluster to run Spark. The suggested deployment method is to use the apache-hadoop-spark bundle. This will deploy the Apache Hadoop platform with a single Apache Spark unit that communicates with the cluster by relating to the apache-hadoop-plugin subordinate charm:

juju-quickstart u/bigdata-dev/apache-hadoop-spark

Alternatively, you may manually deploy the recommended environment as follows:

juju deploy apache-hadoop-hdfs-master hdfs-master
juju deploy apache-hadoop-yarn-master yarn-master
juju deploy apache-hadoop-compute-slave compute-slave
juju deploy apache-hadoop-plugin plugin
juju deploy apache-spark spark

juju add-relation yarn-master hdfs-master
juju add-relation compute-slave yarn-master
juju add-relation compute-slave hdfs-master
juju add-relation plugin yarn-master
juju add-relation plugin hdfs-master
juju add-relation spark plugin

Once deployment is complete, you can manually load and run Spark batch or streaming jobs in a variety of ways:

Spark shell

Spark’s shell provides a simple way to learn the API, as well as a powerful tool to analyze data interactively. It is available in either Scala or Python and can be run from the Spark unit as follows:

juju ssh spark/0
spark-shell # for interaction using scala
pyspark     # for interaction using python

Command line

SSH to the Spark unit and manually run a spark-submit job, for example:

juju ssh spark/0
spark-submit --class org.apache.spark.examples.SparkPi \
    --master yarn-client /usr/lib/spark/lib/spark-examples*.jar 10

Apache Zeppelin visual service

Deploy Apache Zeppelin and relate it to the Spark unit:

juju deploy cs:~bigdata-dev/trusty/apache-zeppelin zeppelin
juju add-relation spark zeppelin

Once the relation has been made, access the web interface at http://{spark_unit_ip_address}:9090

IPyNotebook for Spark The IPython Notebook is an interactive computational environment, in which you can combine code execution, rich text, mathematics, plots and rich media.

Deploy IPython Notebook for Spark and relate it to the Spark unit:

juju deploy cs:~bigdata-dev/trusty/apache-spark-notebook notebook
juju add-relation spark notebook

Once the relation has been made, access the web interface at http://{spark_unit_ip_address}:8880

Testing the deployment

Smoke test spark-submit

SSH to the Spark unit and run the SparkPi test as follows:

juju ssh spark/0
~/sparkpi.sh
exit

Verify Job History

Verify the Job History server shows the previous spark-submit test by visiting http://{spark_unit_ip_address}:18080

Contact Information

Help

Configuration

resources_mirror
(string) URL from which to fetch resources (e.g., Hadoop binaries) instead of Launchpad.
spark_execution_mode
(string) Options are empty string "" for local (no YARN), "yarn-client, and "yarn-master". There are two deploy modes that can be used to launch Spark applications on YARN. In yarn-cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application. In yarn-client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.
yarn-client