ceph rbd mirror #9

Supports: xenial bionic cosmic disco
Add to new model

Description

RBD images can be asynchronously mirrored between two Ceph clusters. This
capability uses the RBD journaling image feature to ensure crash-consistent
replication between clusters. The charm automatically creates pools used for
RBD images on the remote cluster and configures mirroring. Pools tagged with
the rbd application are selected.

NOTE: The charm requires Ceph Luminous or later.


Overview

The ceph-rbd-mirror charm supports deployment of the Ceph RBD Mirror daemon
and helps automate remote creation and configuration of mirroring for Ceph
pools used to host RBD images.

Actions for operator driven failover and fallback for the pools used for RBD
images is also provided.

Data center redundancy is a large topic and this work addresses a very
specific piece in the puzzle related to Ceph RBD images.  You need to
combine this with `Ceph RADOS Gateway Multisite replication`_ and other
work to get a complete solution.

.. _Ceph RADOS Gateway Multisite replication: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-rgw-multisite.html

This is supported both for multiple distinct Ceph clusters within a single Juju
model and between different models with help from cross-model relations.

When the charm is related to a local and a remote Ceph cluster it will
automatically create pools eligible for mirroring on the remote cluster and
enable mirroring.

Eligible pools are selected on the basis of Ceph pool tagging and all pools
with the application rbd enabled on them will be selected.

As of the 19.04 charm release charms will automatically have newly created
pools for use with RBD tagged with the ``rbd`` tag.

Only mirroring of whole pools is supported by the charm.

A prerequisite for RBD Mirroring is that every RBD image within each pool is
created with the journaling and exclusive-lock image features enabled.

To support this the ceph-mon charm will announce these image features over
the client relation when it has units connected to its rbd-mirror
endpoint. This will ensure that images created in the deployment get the
appropriate features to support mirroring.

RBD Mirroring is only supported when deployed with Ceph Luminous or later.

The Ceph RBD Mirror feature supports running multiple instances of the daemon.
Having multiple daemons will cause the mirroring load to automatically be
(re-)distributed between the daemons.

This addresses both High Availability and performance concerns. You can
make use of this feature by increasing the number of ceph-rbd-mirror units
in your deployment.

The charm is written for Two-way Replication, which give you the ability to
fail over and fall back to/from a secondary site.

Ceph does have support for mirroring to any number of slave clusters but
this is not implemented nor supported by the charm.

The charm is aware of network spaces and you will be able to tell the RBD
Mirror daemon about network configuration by binding the public and
cluster endpoints.

The RBD Mirror daemon will use the network associated with the cluster
endpoint for mirroring traffic when available.

Deployment

Example bundles with a minimal test configuration can be found
in the ``tests/bundles`` subdirectory of the ``ceph-rbd-mirror`` charm.

Both examples of two Ceph clusters deployed in one model and Ceph clusters
deployed in separate models are available.

To make use of cross model relations you must first set up an offer to export
a application endpoint from a model. In this example we use the model names
site-a and site-b.

juju switch site-a
juju offer ceph-mon:rbd-mirror site-a-rbd-mirror

juju switch site-b
juju offer ceph-mon:rbd-mirror site-b-rbd-mirror

After creating the offers we can import the remote offer to a model and add
a relation between applications just like we normally would do in a
single-model deployment.

juju switch site-a
juju consume admin/site-b.site-b-rbd-mirror
juju add-relation ceph-rbd-mirror:ceph-remote site-b-rbd-mirror

juju switch site-b
juju consume admin/site-a.site-a-rbd-mirror
juju add-relation ceph-rbd-mirror:ceph-remote site-a-rbd-mirror

Usage

Pools

Pools created by other charms through the Ceph broker protocol will
automatically be detected and acted upon. Pools tagged with the rbd
application will be selected for mirroring.

If you manually create a pool, either through actions on the ceph-mon
charm or by talking to Ceph directly, you must inform the ceph-rbd-mirror
charm about them.

This is accomplished by executing the refresh-pools action.

juju run-action -m site-a ceph-mon/leader --wait create-pool name=mypool \
    app-name=rbd
juju run-action -m site-a ceph-rbd-mirror/leader --wait refresh-pools

Failover and Fallback

Controlled failover and fallback

juju run-action -m site-a ceph-rbd-mirror/leader --wait status verbose=True
juju run-action -m site-b ceph-rbd-mirror/leader --wait status verbose=True

juju run-action -m site-a ceph-rbd-mirror/leader --wait demote

juju run-action -m site-a ceph-rbd-mirror/leader --wait status verbose=True
juju run-action -m site-b ceph-rbd-mirror/leader --wait status verbose=True

juju run-action -m site-b ceph-rbd-mirror/leader --wait promote

NOTE When using Ceph Luminous, the mirror status information may not be
accurate. Specifically the entries_behind_master counter may never get to
0 even though the image is fully synchronized.

Recovering from abrupt shutdown

There exist failure scenarios where abrupt shutdown and/or interruptions to
communication may lead to a split-brain situation where the RBD Mirroring
process in both Ceph clusters claim to be the primary.

In such a situation the operator must decide which cluster has the most
recent data and should be elected primary by using the demote and
promote (optionally with force parameter) actions.

After making this decision the secondary cluster must be resynced to track
the promoted master, this is done by running the resync-pools action on
the non-master cluster.

juju run-action -m site-b ceph-rbd-mirror/leader --wait demote
juju run-action -m site-a ceph-rbd-mirror/leader --wait promote force=True

juju run-action -m site-a ceph-rbd-mirror/leader --wait status verbose=True
juju run-action -m site-b ceph-rbd-mirror/leader --wait status verbose=True

juju run-action -m site-b ceph-rbd-mirror/leader --wait resync-pools i-really-mean-it=True

NOTE When using Ceph Luminous, the mirror state information will not be
accurate after recovering from unclean shutdown. Regardless of the output of
the status information you will be able to write to images after a forced
promote.

Bugs

Please report bugs on Launchpad.

For general questions please refer to the OpenStack Charm Guide.


Configuration

source
(string) Repository from which to install Ceph May be one of the following: distro (default) ppa:somecustom/ppa (PPA name must include UCA OpenStack Release name) deb url sources entry|key id or a supported Ubuntu Cloud Archive pocket. Supported Ubuntu Cloud Archive pockets include: cloud:xenial-pike cloud:xenial-queens cloud:bionic-rocky Note that updating this setting to a source that is known to provide a later version of Ceph will trigger a software upgrade.
distro
use-syslog
(boolean) Setting this to True will allow supporting services to log to syslog.