ceph mon #484

Supports: bionic focal groovy hirsute impish


Ceph is a distributed storage and network file system designed to provide excellent performance, reliability, and scalability.


Ceph is a unified, distributed storage system designed for excellent performance, reliability, and scalability.

The ceph-mon charm deploys Ceph monitor nodes, allowing one to create a monitor cluster. It is used in conjunction with the ceph-osd charm. Together, these charms can scale out the amount of storage available in a Ceph cluster.



This section covers common and/or important configuration options. See file config.yaml for the full list of options, along with their descriptions and default values. See the Juju documentation for details on configuring applications.


The customize-failure-domain option determines how a Ceph CRUSH map is configured.

A value of 'false' (the default) will lead to a map that will replicate data across hosts (implemented as Ceph bucket type 'host'). With a value of 'true' all MAAS-defined zones will be used to generate a map that will replicate data across Ceph availability zones (implemented as bucket type 'rack').

This option is also supported by the ceph-osd charm. Its value must be the same for both charms.


The monitor-count option gives the number of ceph-mon units in the monitor sub-cluster (where one ceph-mon unit represents one MON). The default value is '3' and is generally a good choice, but it is good practice to set this explicitly to avoid a possible race condition during the formation of the sub-cluster. To establish quorum and enable partition tolerance an odd number of ceph-mon units is required.

Important: A monitor count of less than three is not recommended for production environments. Test environments can use a single ceph-mon unit by setting this option to '1'.


The expected-osd-count option states the number of OSDs expected to be deployed in the cluster. This value can influence the number of placement groups (PGs) to use per pool. The PG calculation is based either on the actual number of OSDs or this option's value, whichever is greater. The default value is '0', which tells the charm to only consider the actual number of OSDs. If the actual number of OSDs is less than three then this option must explicitly state that number. Only until a sufficient (or prescribed) number of OSDs has been attained will the charm be able to create Ceph pools.

Note: The inability to create a pool due to an insufficient number of OSDs will cause any consuming application (characterised by a relation involving the ceph-mon:client endpoint) to remain in the 'waiting' state.


The source option states the software sources. A common value is an OpenStack UCA release (e.g. 'cloud:xenial-queens' or 'cloud:bionic-ussuri'). See Ceph and the UCA. The underlying host's existing apt sources will be used if this option is not specified (this behaviour can be explicitly chosen by using the value of 'distro').


A cloud with three MON nodes is a typical design whereas three OSDs are considered the minimum. For example, to deploy a Ceph cluster consisting of three OSDs (one per ceph-osd unit) and three MONs:

juju deploy -n 3 --config ceph-osd.yaml ceph-osd
juju deploy -n 3 --to lxd:0,lxd:1,lxd:2 ceph-mon
juju add-relation ceph-osd:mon ceph-mon:osd

Here, a containerised MON is running alongside each storage node. We've assumed that the machines spawned in the first command are assigned IDs of 0, 1, and 2.

By default, the monitor cluster will not be complete until three ceph-mon units have been deployed. This is to ensure that a quorum is achieved prior to the addition of storage devices.

See the Ceph documentation for notes on monitor cluster deployment strategies.

Note: Refer to the Install OpenStack page in the OpenStack Charms Deployment Guide for instructions on installing a monitor cluster for use with OpenStack.

Network spaces

This charm supports the use of Juju network spaces (Juju v.2.0). This feature optionally allows specific types of the application's network traffic to be bound to subnets that the underlying hardware is connected to.

Note: Spaces must be configured in the backing cloud prior to deployment.

The ceph-mon charm exposes the following Ceph traffic types (bindings):

  • 'public' (front-side)
  • 'cluster' (back-side)

For example, providing that spaces 'data-space' and 'cluster-space' exist, the deploy command above could look like this:

juju deploy -n 3 --config ceph-mon.yaml ceph-mon \
   --bind "public=data-space cluster=cluster-space"

Alternatively, configuration can be provided as part of a bundle:

      charm: cs:ceph-mon
      num_units: 1
        public: data-space
        cluster: cluster-space

Refer to the Ceph Network Reference to learn about the implications of segregating Ceph network traffic.

Note: Existing ceph-mon units configured with the ceph-public-network or ceph-cluster-network options will continue to honour them. Furthermore, these options override any space bindings, if set.


The charm supports Ceph metric monitoring with Prometheus. Add relations to the prometheus application in this way:

juju deploy prometheus2
juju add-relation ceph-mon prometheus2

Note: Prometheus support is available starting with Ceph Luminous (xenial-queens UCA pocket).


This section lists Juju actions supported by the charm. Actions allow specific operations to be performed on a per-unit basis. To display action descriptions run juju actions ceph-mon. If the charm is not deployed then see file actions.yaml.

  • change-osd-weight
  • copy-pool
  • create-cache-tier
  • create-crush-rule
  • create-erasure-profile
  • create-pool
  • crushmap-update
  • delete-erasure-profile
  • delete-pool
  • get-erasure-profile
  • get-health
  • list-erasure-profiles
  • list-inconsistent-objs
  • list-pools
  • pause-health
  • pool-get
  • pool-set
  • pool-statistics
  • purge-osd
  • remove-cache-tier
  • remove-pool-snapshot
  • rename-pool
  • resume-health
  • security-checklist
  • set-noout
  • set-pool-max-bytes
  • show-disk-free
  • snapshot-pool
  • unset-noout

Presenting the list of Ceph pools with details

The following example returns the list of pools with details: id, name, size and min_size. The jq utility has been used to parse the action output in json format.

juju run-action --wait ceph-mon/leader list-pools detail=true \
  --format json | jq '.[].results.pools | fromjson | .[]
  | {pool:.pool, name:.pool_name, size:.size, min_size:.min_size}'

Sample output:

  "pool": 1,
  "name": "test",
  "size": 3,
  "min_size": 2
  "pool": 2,
  "name": "test2",
  "size": 3,
  "min_size": 2


Please report bugs on Launchpad.

For general charm questions refer to the OpenStack Charm Guide.


(string) Which authentication flavour to use. . Valid options are "cephx" and "none". If "none" is specified, keys will still be created and deployed so that it can be enabled later.
(string) The balancer mode used by the Ceph manager. Can only be set for Luminous or later versions, and only when the balancer module is enabled.
(string) The IP address and netmask of the cluster (back-side) network (e.g., . If multiple networks are to be used, a space-delimited list of a.b.c.d/x can be provided.
(string) The IP address and netmask of the public (front-side) network (e.g., . If multiple networks are to be used, a space-delimited list of a.b.c.d/x can be provided.
(string) User provided Ceph configuration. Supports a string representation of a python dictionary where each top-level key represents a section in the ceph.conf template. You may only use sections supported in the template. . WARNING: this is not the recommended way to configure the underlying services that this charm installs and is used at the user's own risk. This option is mainly provided as a stop-gap for users that either want to test the effect of modifying some config or who have found a critical bug in the way the charm has configured their services and need it fixed immediately. We ask that whenever this is used, that the user consider opening a bug on this charm at http://bugs.launchpad.net/charms providing an explanation of why the config was needed so that we may consider it for inclusion as a natively supported config in the charm.
(boolean) Setting this to true will tell Ceph to replicate across Juju's Availability Zone instead of specifically by host.
(int) Default RBD Features to use when creating new images. The value of this configuration option will be shared with consumers of the ``ceph-client`` interface and client charms may choose to add this to the Ceph configuration file on the units they manage. Example: rbd default features = 1 NOTE: If you have clients using the kernel RBD driver you must set this configuration option to a value corrensponding to the features the driver in your kernel supports. The kernel RBD driver tends to be multiple cycles behind the userspace driver available for libvirt/qemu. Nova LXD is among the clients depending on the kernel RBD driver. NOTE: If you want to use the RBD Mirroring feature you must either let this configuration option be the default or make sure the value you set includes the ``exclusive-lock`` and ``journaling`` features.
(boolean) Openstack clouds that use ceph will typically start their life with at least one pool (glance) loaded with a disproportionately high amount of data/objects where other pools may remain empty. This can trigger HEALTH_WARN if mon_pg_warn_max_object_skew is exceeded but that is actually false positive.
(int) The number of OSDs expected to be deployed in the cluster. This value can influence the number of placement groups (PGs) to use for pools. The PG calculation is based either on the actual number of OSDs or this option's value, whichever is greater. The default value is '0', which tells the charm to only consider the actual number of OSDs. If the actual number of OSDs is less than three then this option must explicitly state that number.
(string) The unique identifier (fsid) of the Ceph cluster. . WARNING: this option should only be used when performing an in-place migration of an existing non-charm deployed Ceph cluster to a charm managed deployment.
(string) Apply system hardening. Supports a space-delimited list of modules to run. Supported modules currently include os, ssh, apache and mysql.
(string) Key ID to import to the apt keyring to support use with arbitary source configuration from outside of Launchpad archives or PPA's.
(int) Mon and OSD debug level. Max is 20.
(int) Number of ceph-mon units to wait for before attempting to bootstrap the monitor cluster. For production clusters the default value of 3 ceph-mon units is normally a good choice. . For test and development environments you can enable single-unit deployment by setting this to 1. . NOTE: To establish quorum and enable partition tolerance a odd number of ceph-mon units is required.
(int) Raise HEALTH_ERR status when the filesystem that houses a monitor's data store reports that its available capacity is less than or equal to this percentage.
(int) Raise HEALTH_WARN status when the filesystem that houses a monitor's data store reports that its available capacity is less than or equal to this percentage.
(string) A space-separated list of ceph mon hosts to use. This field is only used to migrate an existing cluster to a juju-managed solution and should otherwise be left unset.
(string) The Ceph secret key used by Ceph monitors. This value will become the mon.key. To generate a suitable value use: . ceph-authtool /dev/stdout --name=mon. --gen-key . If left empty, a secret key will be generated. . NOTE: Changing this configuration after deployment is not supported and new service units will not be able to join the cluster.
(string) Dictionary describing additional checks. Key is a name of a check which will be visible in Nagios. Value is a string (regular expression) which is checked against status messages. . Example: . {'noout_set': 'noout', 'too_few_PGs': 'too few PGs', 'clock': 'clock skew', 'degraded_redundancy': 'Degraded data redundancy'} .
(boolean) Whether additional checks report warning or error when their checks are positive.
(boolean) Whether to report an error when number of known OSDs does not equal to the number of OSDs in or up.
(string) Used by the nrpe-external-master subordinate charm. A string that will be prepended to instance name to set the hostname in nagios. So for instance the hostname would be something like: . juju-myservice-0 . If you're running multiple environments with the same services in them this allows you to differentiate between them.
(float) Threshold for degraded ratio (0.1 = 10%)
(float) Threshold for misplaced ratio (0.1 = 10%)
(boolean) Whether to report Critical instead of Warning when the nodeep-scrub flag is set.
(string) Recovery rate (in objects/s) below which we consider recovery to be stalled.
(string) A comma-separated list of nagios servicegroups. If left empty, the nagios_context will be used as the servicegroup.
(boolean) Causes the charm to not do any of the initial bootstrapping of the Ceph monitor cluster. This is only intended to be used when migrating from the ceph all-in-one charm to a ceph-mon / ceph-osd deployment. Refer to the Charm Deployment guide at https://docs.openstack.org/charm-deployment-guide/latest/ for more information.
(boolean) The charm does not segregate access to pools from different models properly, this means that the correct charm settings can result with client model B having access to the data from model A.
(string) The default configuration for pg-autotune will be to automatically enable the module for new cluster installs on Ceph Nautilus, but to leave it disabled for all cluster upgrades to Nautilus. To enable the pg-autotune feature for upgraded clusters, the pg-autotune option should be set to 'true'. To disable the autotuner for new clusters, the pg-autotune option should be set to 'false'.
(int) The number of placement groups per OSD to target. It is important to properly size the number of placement groups per OSD as too many or too few placement groups per OSD may cause resource constraints and performance degradation. This value comes from the recommendation of the Ceph placement group calculator (http://ceph.com/pgcalc/) and recommended values are: . 100 - If the cluster OSD count is not expected to increase in the foreseeable future. 200 - If the cluster OSD count is expected to increase (up to 2x) in the foreseeable future. 300 - If the cluster OSD count is expected to increase between 2x and 3x in the foreseeable future.
(boolean) If True enables IPv6 support. The charm will expect network interfaces to be configured with an IPv6 address. If set to False (default) IPv4 is expected. . NOTE: these charms do not currently support IPv6 privacy extension. In order for this charm to function correctly, the privacy extension must be disabled and a non-temporary address must be configured/available on your network interface.
(string) Optional configuration to support use of additional sources such as: . - ppa:myteam/ppa - cloud:bionic-ussuri - cloud:xenial-proposed/queens - http://my.archive.com/ubuntu main . The last option should be used in conjunction with the key configuration option.
(string) YAML-formatted associative array of sysctl key/value pairs to be set persistently. By default we set pid_max, max_map_count and threads-max to a high value to avoid problems with large numbers (>20) of OSDs recovering. very large clusters should set those values even higher (e.g. max for kernel.pid_max is 4194303).
{ kernel.pid_max : 2097152, vm.max_map_count : 524288, kernel.threads-max: 2097152 }
(boolean) Configure use of direct IO for OSD journals.
(boolean) If set to True, supporting services will log to syslog.