ubuntu mirror ^#3

By hloeung
stable

Supports: bionic xenial trusty focal

Description

Provides an Ubuntu archive mirror and keeps it up-to-date using rsync

Tags:
web_server ›

Overview

This charm provides a complete Ubuntu mirror service, handling fetching the data via rsync from a master server (updates either triggered on-demand using ssh or by cron), and making it available through http, ftp and rsync.

It can work alongside a log-archive subordinate charm to transfer logs to a suitable storage host.

Usage

Assuming you already have a bootstrapped Juju environment, the charm can be deployed using:

juju deploy ubuntu-mirror

Once deployed, you will need to configure it for your local environment. An Ubuntu archive mirror can be created using the following commands:

juju set ubuntu-mirror mirror_ubuntu_name="myhost.mydomain.com"
juju set ubuntu-mirror mirror_ubuntu_sync_time="0 1 * * *"
juju set ubuntu-mirror role_map='{"myhost":["ubuntu"]}'

Replace "myhost" with the hostname of your server and "myhost.mydomain.com" with its fully qualified domain name (FQDN).

The "role_map" variable tells the charm what type of mirror you want to create (currently, this can be any of "ubuntu" (a mirror of archive.ubuntu.com), "releases" (a mirror of releases.ubuntu.com), "cdimage" (a mirror of cdimage.ubuntu.com" or "ports" (a mirror of ports.ubuntu.com) If you have a sufficiently large server, multiple roles can be assigned to one server (role_map='{"myhost":["ubuntu","cdimage"]}' for example)

If you have more than one server, you should list each one (e.g. role_map='{"host1":["cdimage","ubuntu"],"host2":["releases"]}')

The "mirror_ubuntu_name" variable is the name the webserver will use for the Ubuntu mirror. In the above example, it would answer requests for http://myhost.mydomain.com

The "mirror_ubuntu_sync_time" variable tells the charm what time it should update. In the above example, the update would happen every day at 1am.

There are a number of variables that can be set on a per-role basis. They all follow the pattern mirror__, so if we were setting up a cdimage mirror, we would use "mirror_cdimage_name" and "mirror_cdimage_sync_time".

To cater for rsync servers that require authentication, there is a per-role variable, mirror__rsync_auth. If you don't need to authenticate against your upstream mirror, leave this alone, the defaults will be fine. If you do need to authenticate, this variable should be used like this:

juju set ubuntu-mirror mirror_cdimage_rsync_auth='{"myhost":{"user":"eric", "password":"YXBwbGVzCg=="}}'

The password value is just a Base64-encoded string, just in case it needs to contain any special characters (it also helps avoid people casually reading passwords over your shoulder when you run juju get).

As with role_map (above), if you have multiple servers they should all be listed (e.g. mirror_cdimage_rsync_auth='{"host1":{"user":"eric","password":"YXBwbGVzCg=="},"host2":{"user":"frank","password":"b3Jhbmdlcwo="}}')

For simplicity, if all of your servers authenticate with the same username and password, you can use: '{"<>":{"user":"eric","password":"YXBwbGVzCg=="}}'.

Configuration

apache_delete_logs_after: (int) Delete apache logs after n days.; 14
apache_logdir: (string) Where Apache logfiles should be written; /var/log/apache2
apache_maxclients: (int) Maximum number of simultaneous client connections; 4096
apache_maxrequestsperchild: (int) Maximum number of requests a server process serves
apache_maxsparethreads: (int) Maximum number of Apache worker threads which are kept spare; 100
apache_minsparethreads: (int) Minimum number of Apache worker threads which are kept spare; 50
apache_modules: (string) A space separated list of extra apache modules to enable
apache_serverlimit: (int) Maximum number of Apache server processes allowed; 256
apache_startservers: (int) Initial number of Apache server processes to start; 2
apache_status_allowed_from: (string) A list of IP addresses allowed to access the Apache server-status page; []
apache_threadlimit: (int) Upper limit of ThreadsPerChild allowed in a graceful restart; 64
apache_threadsperchild: (int) Constant number of worker threads in each Apache server process; 64
application_name: (string) The name of this charm; ubuntu-mirror
archive_logs_after: (int) Archive the logs after n days. This is only used when a suitable log-archive subordinate charm is provided; 1
archive_logs_apache: (boolean) Enable log archiving for Apache logs; True
archive_logs_ftp: (boolean) Enable log archiving for FTP logs
archive_logs_rsync: (boolean) Enable log archiving for Rsync logs
disk_inode_crit: (string) Disk inode critical threshold; 5%
disk_space_crit: (string) Disk space critical threshold; 20%
disk_space_warn: (string) Disk space warning threshold; 25%
ftp_delete_logs_after: (int) Delete FTP logs after n days.; 28
ftp_logdir: (string) Where ftpd logfiles should be written; /var/log/ftp
ftp_max_instances: (int) Maximum number of FTP connections allowed; 200
ftp_max_per_source: (int) Maximum number of concurrent FTP connections allowed; 10
ftp_welcome_banner: (string) FTP welcome message; FTP server (vsftpd)
mirror_cdimage_aliases: (string) List of Apache aliases for the cdimage mirror; []
mirror_cdimage_apache_early_extra: (string) A base64 string containing apache configuration options to be included early in the config file
mirror_cdimage_apache_late_extra: (string) A base64 string containing apache configuration options to be included late in the config file
mirror_cdimage_command: (string) The command to use to sync the cdimage mirror; mirror-1stage.sh
mirror_cdimage_description: (string) A brief welcome message for the cdimage mirror; Ubuntu CD Images
mirror_cdimage_name: (string) FQDN of the cdimage mirror; cdimage.ubuntu.com
mirror_cdimage_path: (string) Root location of mirrored files for the cdimage mirror; /srv/ftp.root/cdimage
mirror_cdimage_rsync_auth: (string) Optional rsync authentication details for mirror_cdimage_source_url; {}
mirror_cdimage_rsync_log: (boolean) Whether to log rsync requests for the cdimage mirror; True
mirror_cdimage_rsync_module: (string) The name of the rsync module for this mirror role; cdimage
mirror_cdimage_source_url: (string) The URL the cdimage mirror will be fetched from; rsync://cdimage.ubuntu.com/cdimage
mirror_cdimage_sync_time: (string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering; ondemand
mirror_cdimage_trigger: (string) A base64 string containing the ssh trigger public key
mirror_cloud-images_aliases: (string) List of Apache aliases for the cloud-images mirror; []
mirror_cloud-images_apache_early_extra: (string) A base64 string containing apache configuration options to be included early in the config file
mirror_cloud-images_apache_late_extra: (string) A base64 string containing apache configuration options to be included late in the config file
mirror_cloud-images_command: (string) The command to use to sync the cloud-images mirror; mirror-1stage.sh
mirror_cloud-images_description: (string) A brief welcome message for the cloud-images mirror; Ubuntu Cloud Images
mirror_cloud-images_name: (string) FQDN of the cloud-images mirror; cloud-images.ubuntu.com
mirror_cloud-images_path: (string) Root location of mirrored files for the cloud-images mirror; /srv/ftp.root/cloud-images
mirror_cloud-images_rsync_auth: (string) Optional rsync authentication details for mirror_cloud-images_source_url; {}
mirror_cloud-images_rsync_log: (boolean) Whether to log rsync requests for the cloud-images mirror; True
mirror_cloud-images_rsync_module: (string) The name of the rsync module for this mirror role; cloud-images
mirror_cloud-images_source_url: (string) The URL the cloud-images mirror will be fetched from; rsync://strix.canonical.com/cloud-images
mirror_cloud-images_sync_time: (string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering; ondemand
mirror_cloud-images_trigger: (string) A base64 string containing the ssh trigger public key
mirror_old-releases_aliases: (string) List of Apache aliases for the old Ubuntu releases mirror; []
mirror_old-releases_apache_early_extra: (string) A base64 string containing apache configuration options to be included early in the config file
mirror_old-releases_apache_late_extra: (string) A base64 string containing apache configuration options to be included late in the config file
mirror_old-releases_command: (string) The command to use to sync the old Ubuntu releases mirror; mirror-2stage.sh
mirror_old-releases_description: (string) A brief welcome message for the old Ubuntu releases mirror; Ubuntu Old Releases
mirror_old-releases_name: (string) FQDN of the old Ubuntu releases mirror; old-releases.ubuntu.com
mirror_old-releases_path: (string) Root location of mirrored files for the old Ubuntu releases mirror; /srv/ftp.root/old-images
mirror_old-releases_rsync_auth: (string) Optional rsync authentication details for mirror_old-releases_source_url; {}
mirror_old-releases_rsync_log: (boolean) Whether to log rsync requests for the old Ubuntu releases mirror
mirror_old-releases_rsync_module: (string) The name of the rsync module for this mirror role; old-releases
mirror_old-releases_source_url: (string) The URL the old Ubuntu releases mirror will be fetched from; rsync://old-releases.ubuntu.com/old-releases
mirror_old-releases_sync_time: (string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering; ondemand
mirror_old-releases_trigger: (string) A base64 string containing the ssh trigger public key
mirror_ports_aliases: (string) List of Apache aliases for the ports mirror; []
mirror_ports_apache_early_extra: (string) A base64 string containing apache configuration options to be included early in the config file
mirror_ports_apache_late_extra: (string) A base64 string containing apache configuration options to be included late in the config file
mirror_ports_command: (string) The command to use to sync the ports mirror; mirror-2stage.sh
mirror_ports_description: (string) A brief welcome message for the ports mirror; Ubuntu Ports Archive
mirror_ports_name: (string) FQDN of the ports mirror; ports.ubuntu.com
mirror_ports_path: (string) Root location of mirrored files for the ports mirror; /srv/ftp.root/ubuntu-ports
mirror_ports_rsync_auth: (string) Optional rsync authentication details for mirror_ports_source_url; {}
mirror_ports_rsync_log: (boolean) Whether to log rsync requests for the ports mirror
mirror_ports_rsync_module: (string) The name of the rsync module for this mirror role; ubuntu-ports
mirror_ports_source_url: (string) The URL the ports mirror will be fetched from; rsync://ports.ubuntu.com/ubuntu-ports
mirror_ports_sync_time: (string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering; ondemand
mirror_ports_trigger: (string) A base64 string containing the ssh trigger public key
mirror_releases_aliases: (string) List of Apache aliases for the releases mirror; []
mirror_releases_apache_early_extra: (string) A base64 string containing apache configuration options to be included early in the config file
mirror_releases_apache_late_extra: (string) A base64 string containing apache configuration options to be included late in the config file
mirror_releases_command: (string) The command to use to sync the releases mirror; mirror-1stage.sh
mirror_releases_description: (string) A brief welcome message for the releases mirror; Ubuntu Releases CD Images
mirror_releases_name: (string) FQDN of the releases mirror; releases.ubuntu.com
mirror_releases_path: (string) Root location of mirrored files for the releases mirror; /srv/ftp.root/releases
mirror_releases_rsync_auth: (string) Optional rsync authentication details for mirror_releases_source_url; {}
mirror_releases_rsync_log: (boolean) Whether to log rsync requests for the releases mirror; True
mirror_releases_rsync_module: (string) The name of the rsync module for this mirror role; releases
mirror_releases_source_url: (string) The URL the releases mirror will be fetched from; rsync://rsync.releases.ubuntu.com/releases
mirror_releases_sync_time: (string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering; ondemand
mirror_releases_trigger: (string) A base64 string containing the ssh trigger public key
mirror_simple-streams_aliases: (string) List of Apache aliases for the simple-streams mirror; []
mirror_simple-streams_apache_early_extra: (string) A base64 string containing apache configuration options to be included early in the config file
mirror_simple-streams_apache_late_extra: (string) A base64 string containing apache configuration options to be included late in the config file
mirror_simple-streams_command: (string) The command to use to sync the simple-streams mirror; mirror-1stage.sh
mirror_simple-streams_description: (string) A brief welcome message for the simple-streams mirror; Juju Simple Streams Mirror
mirror_simple-streams_name: (string) FQDN of the simple-streams mirror; streams.canonical.com
mirror_simple-streams_path: (string) Root location of mirrored files for the simple-streams mirror; /srv/ftp.root/simple-streams
mirror_simple-streams_rsync_auth: (string) Optional rsync authentication details for mirror_simple-streams_source_url; {}
mirror_simple-streams_rsync_log: (boolean) Whether to log rsync requests for the simple-streams mirror; True
mirror_simple-streams_rsync_module: (string) The name of the rsync module for this mirror role; simple-streams
mirror_simple-streams_source_url: (string) The URL the simple-streams mirror will be fetched from; rsync://strix.canonical.com/simple-streams
mirror_simple-streams_sync_time: (string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering; ondemand
mirror_simple-streams_trigger: (string) A base64 string containing the ssh trigger public key
mirror_ubuntu_aliases: (string) List of Apache aliases for the Ubuntu mirror; []
mirror_ubuntu_apache_early_extra: (string) A base64 string containing apache configuration options to be included early in the config file
mirror_ubuntu_apache_late_extra: (string) A base64 string containing apache configuration options to be included late in the config file
mirror_ubuntu_command: (string) The command to use to sync the Ubuntu mirror; mirror-2stage.sh
mirror_ubuntu_description: (string) A brief welcome message for the Ubuntu mirror; Ubuntu Archive
mirror_ubuntu_name: (string) FQDN of the Ubuntu mirror; archive.ubuntu.com
mirror_ubuntu_path: (string) Root location of mirrored files for the Ubuntu mirror; /srv/ftp.root/ubuntu
mirror_ubuntu_rsync_auth: (string) Optional rsync authentication details for mirror_ubuntu_source_url; {}
mirror_ubuntu_rsync_log: (boolean) Whether to log rsync requests for the Ubuntu mirror
mirror_ubuntu_rsync_module: (string) The name of the rsync module for this mirror role; ubuntu
mirror_ubuntu_source_url: (string) The URL the Ubuntu mirror will be fetched from; rsync://archive.ubuntu.com/ubuntu
mirror_ubuntu_sync_time: (string) When mirror updates should be run. Either a cron(5) format time specification or 'ondemand' for ssh triggering; ondemand
mirror_ubuntu_trigger: (string) A base64 string containing the ssh trigger public key
mirror_user: (string) Mirror system user id; archvsync
nagios_servicegroup: (string) If set, this is the Nagios servicegroup for alerts. If unset, an appropriate one will be chosen
package_status: (string) The status of service-affecting packages will be set to this value in the dpkg database. Useful valid values are "install" and "hold".; install
role_map: (string) JSON document describing which mirror is enabled on which machine. There are two formats. The first format is simple: {"hostname1": ["cdimage", "ubuntu"], "hostname2": ["releases"]} In the second more complex format each hostname's value is a dictionary rather than a list of roles. For example: { "hostname1": { "releases": { "addresses": ["1.2.3.4"], "https": true }, "ubuntu": null }, "hostname2": [ "releases", "ubuntu" ] } Here, hostname1's ubuntu role will result in an HTTP VirtualHost that listens on *:80 and serves the Ubuntu archive. The releases role will result in HTTP and HTTPS VirtualHosts that listen on 1.2.3.4:80 and 1.2.3.4:443 respectively.
rsync_logdir: (string) Rsync log directory; /var/log/rsyncd
rsync_max_connections: (int) Maximum number of rsync connections allowed; 65
rsync_max_connections_per_source: (int) Maximum number of rsync connections allowed per source/IP; 5
rsync_motd: (string) Rsync server welcome message text (base64 encoded); VGhpcyBpcyBhbiBVYnVudHUgbWlycm9yIC0gdHJlYXQgaXQga2luZGx5Cg==
rsync_motdfile: (string) Rsync server welcome message file; /etc/rsyncd/motd
script_dir: (string) Install location for any mirror-related scripts; /srv/ubuntu-mirror/bin