container log archive #1

Supports: xenial bionic focal


We want to keep archive logs for a while, but they're big. Solution: store them in containers (it works for shipping companies!)

This charm provides a way to archive log files from a given directory to container-based storage (currently supports Amazon S3, OpenStack Swift and Google Compute Engine (GCE)).

Common Configuration

It is a subordinate charm, requiring either a "log-archive-relation-changed" hook in the parent to provide the directory to be archived and when to archive it or have those same values passed as charm options. Note: If both methods are used, all the directories will be archived, but the timing from the relation is preferred over a charm option, e.g.:

 relation-set logdirs="base64_dir1 base64_dir2"
 relation-set archive_after=1

Any files older than archive_after days in the specified directories will be archived to container storage. Deletion of local log files is assumed to be handled by the parent charm unless the delete_after_archive config variable is set to true (in which case the charm will remove the log file immediately after it is successfully archived)

The logdir relation variable is a space-separated list of names. To avoid any issues with unusual filenames, each name must be base64-encoded. This is not true when passed as a charm option.

For example, to archive files in both /var/log/logdir1 and /var/log/logdir2:

logdirs="L3Zhci9sb2cvbG9nZGlyMQo= L3Zhci9sb2cvbG9nZGlyMgo="

Logdirs can be either a directory name, where everything in it will be archived or a glob pattern, where just matches will be archived. If you want just one file archived, use a glob pattern that only matches that file (e.g. /var/log/sys[l]og)

Note: Using delete_after_archive on a log file still being written to may lead to filesystem weirdness (why would you archive an active log file anyway?) Also using this option when multiple processes are parsing archived logs can lead to unpredictable behavior. You have been warned.

Provider Configuration

The container_credentials variable should be a base64-encoded JSON string with the authorisation details needed to access the container. For Swift it should look like this:

{ "os_username": "myuser", "os_tenant_name": "myuser_project", "os_password": "secret", "os_auth_url": "http://mykeystoneurl/", "os_region_name": "openstack-region-name" }

For S3:

{ "AWS_ACCESS_KEY_ID": "my-amazon-id", "AWS_SECRET_ACCESS_KEY": "secret", "AWS_REGION": "aws-region-name" }

For GCE use the standard JSON file Google provides:

{ "private_key_id": "mykey", "private_key": "my-cge-key", "client_email": "email-address", "client_id": "gce-client-id", "type": "service_account" }

For Azure:

{ "private_key": "my-private-key" }


It is also possible to add pre-processors that are run against the log files before they are updated to swift. The available pre-processors include gzip, gunzip and a web log anonymizer. To specify pre-processors just specify a space separated list in the pre_processors variable and they will run in order.

preprocessors="gunzip anonymize_web_log gzip"

Some preprocessors take additional options for example:

anonymize_web_log_options="--skip-private -s"

If you would like to send raw logs to one location and pre-processed logs to another the charm can be installed multiple times with different Juju application names. In this case be careful with the delete_after_archive option, it is best avoided.


(string) Options to pass to the anonymize web logs script if it is running as a preprocessor. In addition to anonymizing web logs it can skip ranges or private addresses if specified.
(string) Apt sources.list line for a repository containing necessary packages
(string) GPG key for apt_repository
(string) Default archive_after value in case it isn't set in any relation data
(string) MANDATORY: Login credentials needed for container access (base64 encoded)
(string) MANDATORY: The name of the container/bucket to archive files into Note: For some providers, this has to be a globally unique name
(string) MANDATORY: Which container format to use ('azure', 'gce', 's3' or 'swift')
(string) When to run the log-archiver (crontab(5) format timespec). Use <<rnd>> for a random value
<<rnd>> <<rnd>> * * *
(boolean) If true, logs will be deleted from the local host after successful archiving
(string) Space separated list of directories or glob patterns to archive (in addition to any specified by relations)
(string) Used for nagios monitoring. See nrpe-external-master charm for details
(string) Used to determine how the server is identified in nagios. Choices are 'unit' (ubuntu-mirror-1 format) or 'host' ( format)
(string) If set, this is the Nagios servicegroup for alerts. If unset, an appropriate one will be chosen
(string) A space separated list of preprocessors to run, ie gunzip, gzip or anonymize_web_log. They will be run in order.
(string) The project name (used by Azure and GCE)
(string) The user managing the archiving process