druid config #65

Supports: xenial bionic
Add to new model

Description

This charm is responsible for providing common configuration options to a Druid Cluster. It is a lightweight charm that provides configuration files to each node in your cluster. For this reason, it is not necessary to scale out this charm as you scale out your cluster. There should only be one unit of this charm per cluster, and all Druid nodes (Druid Overlord, Druid Coordinator, Druid Middlemanager, Druid Historical and Druid Broker) must be directly related to this charm.


Overview

Welcome to the Druid Config charm by Spicule. This charm is responsible for providing common configuration options to a Druid Cluster. It is a lightweight charm that provides configuration files to each node in your cluster. For this reason, it is not necessary to scale out this charm as you scale out your cluster. There should only be one unit of this charm per cluster, and all Druid nodes (Druid Overlord, Druid Coordinator, Druid Middlemanager, Druid Historical and Druid Broker) must be directly related to this charm.

Usage

To deploy this charm from the command line, enter the following:

juju deploy cs:~spiculecharms/druid-config --series xenial

That's it! This charm requires only the smallest instance of whatever Cloud platform you are using, and by applying no hardware constraints, Juju will automatically request the smallest instance available.

Once you have deployed your Druid Config charm, you must add a relation to each of your Druid nodes. This is simple to do from the command line:

juju add-relation druid-config druid-<druid-node-name>

MySQL and Zookeeper

In addition to relating Config to your Druid nodes, you must also relate it to a MySQL charm for metadata storage, as well as a Zookeeper charm for cluster coordination. Deploy these charms and add relations to them, and Config will automatically configure your cluster to point to these servers.

Storage Options

Druid Config is responsible for configuring your Druid Cluster's storage settings. How you set this up depends on your choice of storage solution. Druid currently supports three main storage options:

  • Local storage
  • S3 Buckets
  • HDFS

Local storage is enabled by default on deployment. To utilise local storage, simply spin up your Druid Config charm and relate it to your Druid nodes.

To configure your cluster to utilise an S3 bucket from the command line on deployment, enter the following:

juju deploy cs:~spiculecharms/druid-config --series xenial --config s3-storage="True" --config s3-bucket="<s3-bucket>" --config s3-accessKey="<s3-accessKey>" --config s3-secretKey="<s3-secretKey>"

To configure your cluster to utilise HDFS, simply add a relation to the Hadoop Plugin charm, which you can find in our handy Anssr Hadoop Bundle, and Druid Config will handle the rest for you!

Druid Config is also responsive to changes in these configuration settings, allowing the user to change their storage solution as needed. If you spin up a cluster using local storage at first, you can modify the configuration of Config to use S3 instead, and the Druid Nodes will re-configure and restart automatically. Likewise, if you wish to change to HDFS, simply spin up your Anssr Hadoop Bundle, relate the Hadoop Plugin to Config and your cluster will be re-configured to utilise your Hadoop cluster.

Known Limitations and Issues

Currently, the only known limitations and issues are some extreme use cases which will not apply to most users. For example, Juju will not let users remove the Hadoop-Config relation, so switching between HDFS clusters is not currently possible.

If you find any further issues with this charm, please do let us know! You can find our contact details below.

Contact Information

Thank you for using the Druid Config charm - we hope you find it useful! For more information, here are our contact details:

Spicule: http://www.spicule.co.uk
Anssr: http://anssr.io
Email: info@spicule.co.uk

For assistance with Juju, please refer to the Juju Discourse.

Druid

For more information about Druid, please refer to the Druid website.


Configuration

s3-accessKey
(string) S3 Access Key
s3-bucket
(string) S3 Bucket
s3-secretKey
(string) S3 Secret Key
s3-storage
(boolean) Enable S3 Storage